and Tactical Behaviors for Unmanned Ground Vehicles James Albus, - - PowerPoint PPT Presentation

SLIDE 1

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

1

Intelligent Control and Tactical Behaviors for Unmanned Ground Vehicles

James Albus, Senior NIST Fellow (Retired)

Intelligent Systems Division National Institute of Standards and Technology

Senior Robotics Scientist Robotic Technology Inc.

SLIDE 2

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

2

Basic Intelligent System

Perception establishes correspondence between internal world model and external real world Perception Behavior World Model Sensing Action Real World

internal external

Goal Behavior uses world model to generate action to achieve goals Orient Observe Decide Act

OODA loop

SLIDE 3

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

3

4D/RCS Reference Model

OPERATOR INTERFACE

SP WM BG SP WM BG SP WM BG SP WM BG

Points Lines Surfaces

SP WM BG SP WM BG SP WM BG

0.5 second plans Steering, velocity 5 second plans Subtask on object surface Obstacle-free paths

SP WM BG SP WM BG SP WM BG SP WM BG SP WM BG

SERVO PRIMITIVE SUBSYSTEM SURROGATE SECTION SURROGATE PLATOON

SENSORS AND ACTUATORS Plans for next 2 hours Plans for next 24 hours 0.05 second plans Actuator output

SP WM BG SP WM BG SP WM BG SP WM BG SP WM BG SP WM BG SP WM BG SP WM BG

Objects of attention Locomotion Communication Mission Package

VEHICLE

Plans for next 50 seconds Task to be done on objects of attention Plans for next 10 minutes Tasks relative to nearby objects Section Formation Platoon Formation Attention Battalion Formation

SURROGATE BATTALION

6

OODA OODA OODA OODA OODA OODA OODA OODA OODA OODA OODA OODA OODA OODA OODA OODA OODA OODA OODA

Intelligent System Architecture

4D/RCS Reference Model

OODA

SLIDE 4

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

4

4D/RCS Reference Model Architecture for Unmanned Vehicle Systems

• Hierarchical structure
• f goals and commands
• Representation of the world

at many levels

• Planning, replanning,

and reacting at many levels

• Integration of many sensors

stereo CCD & FLIR, LADAR, radar, inertial, acoustic, GPS, internal

Adopted by GDRS for FCS Autonomous Navigation System Adopted by TARDEC for Vetronics Technology Integration

SLIDE 5

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

5

• Combines AI with control theory
• Hierarchical representation of tasks, space, & time
• Combines deliberative with reactive at many levels
• Depends strongly on sensing and perception
• Supports a rich dynamic world model at many levels
• Integrates prior knowledge with current observations
• Models functional architecture of the human brain
• Addresses the full range of human behavior
• Is mature with engineering tools and software libraries

Attributes of 4D/RCS

SLIDE 6

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

6

4D/RCS Reference Model

OPERATOR INTERFACE

SP WM BG SP WM BG SP WM BG SP WM BG

Points Lines Surfaces

SP WM BG SP WM BG SP WM BG

0.5 second plans Steering, velocity 5 second plans Subtask on object surface Obstacle-free paths

SP WM BG SP WM BG SP WM BG SP WM BG SP WM BG

SERVO PRIMITIVE SUBSYSTEM SURROGATE SECTION SURROGATE PLATOON

SENSORS AND ACTUATORS Plans for next 2 hours Plans for next 24 hours 0.05 second plans Actuator output

SP WM BG SP WM BG SP WM BG SP WM BG SP WM BG SP WM BG SP WM BG SP WM BG

Objects of attention Locomotion Communication Mission Package

VEHICLE

Plans for next 50 seconds Task to be done on objects of attention Plans for next 10 minutes Tasks relative to nearby objects Section Formation Platoon Formation Attention Battalion Formation

SURROGATE BATTALION

6

Intelligent System Architecture

SLIDE 7

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

7

A 4D/RCS Computational Node

4D/RCS node

SLIDE 8

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

8

Knowledge is Central

4

SLIDE 9

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

9

Forms of Representation

Iconic

• signals, images, maps (arrays)
• Support communication, geometry, and navigation
• Have range and resolution in space and time

Symbolic

• objects, events, classes (abstract data structures)
• Support mathematics, logic, and linguistics
• Have vocabulary and ontology

Links

• relationships (pointers)
• Support syntax, grammar, and semantics
• Have direction and type

SLIDE 10

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

10

MULTI-RESOLUTION MAPS

0.4 m grid 50 m wide 4 m grid 500 m wide 30 m grid Terrain map

• Data flows up

and down between the different maps

• Path planning
• ccurs at each

level

SLIDE 11

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

11

Sensory Processing

Classification Compare group attributes with class prototype Set pointers that define class membership Focus Attention Direct sensors to region of interest Window and track interesting entities and events Segmentation and Grouping Segment pixels that meet grouping criteria Set pointers that define grouping relationships Computation of Group Attributes e.g., size, shape, texture, motion Recursive estimation of group attributes

* *

SLIDE 12

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

12

Segmentation & Grouping

Spatial pixel patterns => Entities Temporal signal patterns => Events Fundamental Problems: Any segmentation is a hypothesis. Needs confirmation. 2D images are ambiguous in range => infinite # of hypotheses Bottom-up segmentation of optical images is notoriously poor. Need to integrate top-down inputs Segmentation criteria == Gestalt grouping hypotheses Proximity in space or time Similarity in brightness, color, shape, size, texture, etc. Symmetry, Smooth continuation

SLIDE 13

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

13

Classification

Range images are low in resolution Therefore, not enough pixels on target Fundamental Problem Object classification depends on:

• 1. accurate segmentation and grouping
• 2. dimensionality of object attribute vector
• 3. number of pixels on target (> 100)

Optical images are high in resolution, but ambiguous in range. Therefore, segmentation is hard Data fusion helps High-level context helps more

SLIDE 14

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

14

2D Color Image 3D Range Image

LADAR is a Critical Break-Through

Enables transformation into geocoordinates Range and slope are ambiguous Segmentation is difficult

SLIDE 15

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

15

4D/RCS World Model Real-time Map Building

SLIDE 16

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

16

Next Generation LADAR

Intensity Image in the Woods

SLIDE 17

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

17

Geocoordinates Overhead View

SLIDE 18

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

18

Range Image Oblique View

SLIDE 19

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

19

.02 degree angular resolution 2 cm range resolution 5 x 80 degree field of view

High Resolution LADAR

SLIDE 20

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

20

Segmentation based

• n Fusion of

Color & LADAR Images

James Albus Tsai Hong Mike Shneier Gerry Cheok Tommy Chang

National Institute of Standards and Technology

• U. S. Department of Commerce

SLIDE 21

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

21

False color LADAR intensity image

SLIDE 22

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

22

Road Detection Road Edge Detection

Color Segmentation

SLIDE 23

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

23

Road edges from color image

SLIDE 24

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

24

Segment out everything to right and left of road edges

Road Edge Segmentation

SLIDE 25

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

25

Segment out everything > 2 m above road and range > 70 m

Height & Range Segmentation

SLIDE 26

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

26

Segment out road and points < 20 cm above road

Ground Segmentation

SLIDE 27

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

27

Compute Attributes

• f Segmented Cars

Object1 Range = 41 m Closing speed = 2 m/sec Width = 176 cm Height = 128 cm

Object2 Range = 62 m Closing speed = 2 m/sec Width = 162 cm Height = 140 cm

Classify based on height, width, and closing speed

SLIDE 28

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

28

Image Processing

• f

High Resolution Range Images

Human Detection in a Cluttered Environment

at 50 meters Jim Albus, Tsai Hong, Will Shackelford, Tommy Chang, Gary Haas

SLIDE 29

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

29

Three Mannequins

SLIDE 30

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

30

SLIDE 31

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

31

Segmentation based on Connected Components

SLIDE 32

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

32

Humans Detected

SLIDE 33

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

33

Color Image Windowed by Humans Detected in Range Image

SLIDE 34

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

34

SLIDE 35

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

35

SLIDE 36

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

36

Classification

requires pixels on target

Translates into resolution required at distance

SLIDE 37

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

37

Color Video Camera 25 m @ .02 deg/pix ~ human vision ~ 10,000 pixels

• n human target

SLIDE 38

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

38

Current real-time LADAR 25 m @ .2 deg/pix ~ 100 pixels

• n human target

SLIDE 39

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

39

Current real-time LADAR 50 m @ .2 deg/pix ~ 25 pixels

• n human target

SLIDE 40

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

40

Current real-time LADAR 100 m @ .2 deg/pix ~ 6 pixels

• n human target

SLIDE 41

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

41

High resolution LADAR 50 m @ .05 deg/pix Non-real-time ~ 400 pixels

• n human target

SLIDE 42

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

42

High resolution LADAR 100 m @ .02 deg/pix Non-real-time ~ 600 pixels

• n human target

SLIDE 43

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

43

LADAR Resolution Required to Recognize Human Form at Various Distances

Distances 25 m 50 m 100 m Resolutions .2 deg/pix .05 deg/pix .02 deg/pix

SLIDE 44

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

44

4D/RCS Methodology for

Tactical Behaviors

The ability to perform tactical behavior is the reason the Army is interested in robotics

Company level – 30 to 40 vehicles Platoon level – 8 to 10 vehicles Section level – 2 to 4 vehicles Vehicle level – single vehicle

Armor

50 ms plans

• utput every

5 ms

UARV RSTA Communications Weapons Mobility

Vehicle Section Company Platoon Primitive Servo Sensors and Actuators Subsystem

500 ms plans replan every 50 ms 5 s plans replan every 500 ms 1 min plans replan every 5 s 10 min plans replan every 1 min 1 hr plans replan every 5 min 5 hr plans replan every 25 min

Driver Gaze Gaze Focus Pan Tilt Heading Speed Pan Tilt Iris Select Manned C2 DirectFire UAV UGV Scout UAV C2 UGS C2 AntiAir IndirectFire Logistics Artillary Battalion HQ

24 hr plans replan every 2 hr

Battalion

Armor

50 ms plans

• utput every

5 ms

UARV RSTA Communications Weapons Mobility

Vehicle Section Company Platoon Primitive Servo Sensors and Actuators Subsystem

500 ms plans replan every 50 ms 5 s plans replan every 500 ms 1 min plans replan every 5 s 10 min plans replan every 1 min 1 hr plans replan every 5 min 5 hr plans replan every 25 min

Driver Gaze Gaze Focus Pan Tilt Heading Speed Pan Tilt Iris Select Manned C2 DirectFire UAV UGV Scout UAV C2 UGS C2 AntiAir IndirectFire Logistics Artillary Battalion HQ

24 hr plans replan every 2 hr

Battalion

4D/RCS for FCS

Manned/Unmanned collaboration UGV/UAV/UGS collaboration

SLIDE 45

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

45

………

A Light Cavalary Troop receives a command to perform a tactical road march to an assembly area This results in a command to a Scout Platoon to perform a route reconnaissance of the road

An Example Scenario

The scout platoon is composed of three sections, each containing three manned HMMWVs, one unmanned ground vehicle (UGV), and

• ne

unmanned aerial vehicle (UAV.)

SLIDE 46

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

46

c c c

UGV UAV MANNED MANNED MANNED

Scout section is is conducting a route reconnaissance HMMWV rec econnoi

• iter

ering the he right ht fl flank nk com

• mes

es upon upon an an unex expec ecte ted wat ater er

• bstacle

Center HMMWV discovers a bridge The two vehicle commanders report their findings to to the the section leader The sec ection

• n leader

er then en mi might ht comma mand the the man anned ed vehicles es to to tak ake up up

• verwatc

tch positions for for near near-side security ty The sec ecti tion lea eader er al also com

• mmands

ds the the UA UAV to to look

• k fo

for a rou

• ute around the

the wat ater er obs bstac tacle UA UAV send nds hi hi-res esolut ution color

• r imag

ages es dat ata bac ack to to the the sec ecti tion leader er for

• r manual

al viewi wing, g, and/ d/or

• r by

by scanni ning ng the the gr ground nd with a LADAR to to assess the the topography Onc nce a pot

• ten

ential by by-pas ass to to the the mars arsh is is loc

• cat

ated ed, the the UAV is is comma manded to to searc arch the the far far side of

• f the

he mars arsh and and th the regi gion bey eyon

• nd the

the nex ext terr terrain feature for evidence of

• f enemy

enemy forces UG UGV might ht then then be be comm mmanded to to pr proc

• ceed thr

hroug ugh the the by bypas ass and and establish an an overwatc tch position on

• n the

the far far side of

• f the

the next terrain feature The UG UGV pat ath can be be aut utom

• mati

tically gener erate ated fr from the he dat ata retu eturned fr from the the UAV and and approved by by the the section leader before being executed Once the UGV UGV is is set in in position, the UA UAV continues scanning for enemy activity ty further along the the route Manned Manned elem ements nts perfo erform manu nual rec econn nnaissanc nce of

• f the

the mars arsh by by-pa pass, and/or and/or assess the the load carrying capacity ty of

• f the

the bridge.

A Section Scenario

SLIDE 47

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

47

Capture Behavioral Knowledge

4D/RCS Node

NormalReconSituation ConductReconToControlPoint TacticallyAssessGully GullyFeatureDetected TacticallyAssessWaterFeature WaterFeatureDetected DefileConstrictionDetected TacticallyAssessDefile ReinforcingObstacleDetected FallbackToStandoffPosition EnemyPresenceDetected PerformStealthyHalt NonEngagedEnemyDetected FallbackToStandoffPosition DangerAreaDetected PerformStealthyHalt KeyDominatingTerrainDetected TacticallyAssessKeyTerrain RestrictingTerrainDetected TacticallyAssessRestrictingTerrain NonPassableObstacleDetected PerformStealthyHalt TunnelDetected PerformStealthyHalt BridgeDetected PerformStealthyHalt ManMadeObjectsDetected TacticallyAssessManMadeObjects AvenueOfApproachDetected PerformStealthyHalt EngagedEnemyContact FallbackWithFires BuiltUpAreaDetected FallbackToStandoffPosition

ConductReconToControlPoint

Vehicle | PLAN SELECTION TABLE VehicleInoperative PerformOwnVehDisabledProcedures PLAN STATE-TABLE Input Conditions Output Commands S1 WaterFeatureNotImpactMovement S0 NormalMovementPossible NewPlan S1 DetermineIfWaterIsObstacleToMovement PlanObservationGoalPt surv_LookForBypass leth_LookForEnemyOverwatch Vehicle | TacticallyAssessWaterFeature S1 WaterFeatureTraversable S3 PlanWaterTraverseGoalPts SetupWaterFeatureBypassReport comm_SendReport surv_AssessWaterFeatureTraversePath S3 ClearOfWaterFeature S0 NormalMovementPossible S1 NewObservationGoalPtPlanned S1 DetermineIfWaterIsObstacleToMovement PlanObservationGoalPt SetupMovementParams mob_MoveToObservationGoalPt S1 WaterFeatureIsMovementObstacle S2 SetupWaterObstacleReport PlanObstacleOverwatch mob_MoveToObstacleOverwatchPosition surv_AssessForEnemyPresence S3 PermissionToTraverseFeature NewWaterTraverseGoalPtPlanned S3 SetupMovementParams mob_MoveToBypassGoalPt PlanWaterTraverseGoalPts S2 SetAtObstacleOverwatch S0 SetupWaterObstacleOverwatchReport comm_SendReport DOT Driving Manuals and ARMY Field Manuals

+

Domain Experts

Task Decomposition Tree (Route Reconnaissance Example) Hierarchical Organization of Agent Control Modules

BEHAVIOR GENERATION WORLD MODEL KNOWLEDGE SENSORY PROCESSING

Situations World States Objects & Attributes

ColorCameras LADAR Radar Stereo FLIR Nav Segmented Groupings Features and Attributes Objects and Maps Object Groupings and Classifications

Cattails

ConductRoadMarchToAnAssemblyArea(AA) Conduct RouteRecon PrepareFor RoadMarch FollowPlatoon ToAssemblyArea Secure AssemblyArea Organize AssemblyArea Conduct MainRoute Recon DeployTo StartPoint Conduct LeftFlank RouteRecon Locate&Secure ObstacleBypass Conduct Obstacle Recon Conduct RightFlank RouteRecon MoveInto MarchFormation PrepareDetailed MovementPlans MoveTo ControlPoint Conduct DominateTerrain Recon MoveTo Cover/Concealed Position Ford Water Obstacle Secure Area Overwatch Section Perform Ford Recon Locate WaterBypass MoveTo ControlPoint Assess FordTerrain MoveTo Position Scan Path Cross Ford ScanArea ForEnemy MoveTo Water ShiftTo 4WhLo MoveTo Opposite Bank ShiftTo 4WhHi Dry Brakes WaterDepthToSixFeet WaterCoveredLand MajorGroundDeformation TractionSlip LegalToPass MarshDetected SomeWaterVisible Mosses,Evergreens,andShrubs StagnantWater IndeterminantGroundLevel SignificantTractionSlip OrganicMaterialOnWaterSurface BogDetected ErodedEarthEmbankments FlowingWater NonVegetatedWaterInMiddle NarrowWidth,IndeterminantLength StreamDetected ErodedEarthEmbankments FlowingWater NonVegetatedWaterInMiddle SignificantWidth,IndeterminantLength RiverDetected WaterSheenOnGroundSurface LittleToNoVegetation SignificantTractionSlip RuttedWithStandingWater MudDetected SignificantGroundDeformation ExtensiveMarshVegetation LongLeafGrasses - very flat, long green leaves-purple/rose/yellow flowers. Reeds - tall, woody, thin, round, hollow jointed (tan-to-green) stem plants, long narrow green blade leaves, large feathery panicles (elongated clusters of tan/white/purple flowers along main stem). Sedges - triangular tan/green stem plants, papyrus, narrow green to tan grass- like leaves, spikelets of inconspicuous tan-to-yellow-to-white flowers. Water Lilies - very large floating green leaves with white flowers. Saturated Ground Six Feet

• f Water

Plant Height 6-18" 6-48" 1-6' 3-9' Float LargeSurfaceAreaOfStillWater LargeAreaWithoutGrasses,Trees,Shrubs OrganicMaterialMayBeOnWaterSurface Pond/LakeDetected BoundedBySwamp,Marsh MostlyTrees,SomeBushes SlowMovingWaterCoveredLand SignificantTractionSlip ExtensiveWaterSurfaceVisible SwampDetected MajorGroundDeformation 15 cm 2.4 cm .9 to 2.7m 3 cm Scout Platoon B Section Vehicle #5 HM M W V #6 Vehicle #8 A Section Vehicle #7 Vehicle #3 Vehicle #2 C Section Vehicle #10 Vehicle #9 Vehicle #4 PSG ConductRightOfRouteRecon ConductM ainRouteRecon ConductLeftOfRouteRecon Conduct Traveling Overwatch Visually Clear Route Conduct Traveling Overwatch Conduct Recon ToCP Conduct Traveling Overwatch Conduct Route Recon To CP Support Route Recon W aterObsacle Detected ConductW aterObsRecon ConductRouteRecon TacticallyAssess W aterFeature Conduct ReconToCP Surveil (RSTA) Lethality (Gunner) Mobility Pan/Tilt Control W eapon Control Targeting Sensors Pan/Tilt Control FLIR, Laser Radar, Cam era Vehicle #6 Elem ental Move Visually Clear Route Pan/Tilt Control Ladar & Color Cam era Engine Control Steer Control VehPrim / Propulsion Gaze Control TransXfer Control Elem ental Moves Speed Control ParkBrake Control Analyze W aterFeature LookFor Objects M akeVehRdy ForDriving Analyze Terrain W aterFeature Detected W aterObjects Detected M

• veTactically

ToGoalPt Startup Vehicle LookFor Enem yPresence StartEngine Assess Path PrepForStart SetParkBrake PrepForStart PrepForStart that explode into white down, long dark brown cylindrical seed heads Bulrushes - tall tan-to-green stems, with flat green sword shaped leaves, cattails. WaterFeatureDetected SENSORY PROCESSING KNOWLEDGE DATABASE BEHAVIOR GENERATION WORLD MODEL VALUE JUDGMENT (Executing) (ReconToRoute) (Executing) SENSORY INPUT (Platoon “B” Section Agent Control Module) (ConductReconToControlPt) BEHAVIOR GENERATION STATUS STATUS NEXT SUBGOAL

.

COMMANDED TASK (GOAL) DriveOnTwoLaneRd PassVehInFront PassVehInFront DriveOnTwoLaneRd NegotiateLaneConstrict PassVehInFront . STATE-TABLES

STEP 1

TASK ANALYSIS to Create Task Decomposition Tree

STEP 2

MAP to AGENT ARCHITECTURE MAP TASK DECISIONS to STATE-TABLES

STEP 3

DETERMINE ANTECEDENT WORLD STATES

STEP 4

IDENTIFY OBJECTS and THEIR RELEVANT ATTRIBUTES

STEP 5

DETERMINE SENSOR PROCESSING REQUIREMENTS AND RESOLUTIONS

STEP 6

SLIDE 48

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

48

DOT Driving Manuals and ARMY Field Manuals

+

Domain Experts

Task Decomposition Tree (Route Reconnaissance Example)

ConductRoadMarchToAnAssemblyArea(AA) Conduct RouteRecon PrepareFor RoadMarch FollowPlatoon ToAssemblyArea Secure AssemblyArea Organize AssemblyArea Conduct MainRoute Recon DeployTo StartPoint Conduct LeftFlank RouteRecon Locate&Secure ObstacleBypass Conduct Obstacle Recon Conduct RightFlank RouteRecon MoveInto MarchFormation PrepareDetailed MovementPlans MoveTo ControlPoint Conduct DominateTerrain Recon MoveTo Cover/Concealed Position Ford Water Obstacle Secure Area Overwatch Section Perform Ford Recon Locate WaterBypass MoveTo ControlPoint Assess FordTerrain MoveTo Position Scan Path Cross Ford ScanArea ForEnemy MoveTo Water ShiftTo 4WhLo MoveTo Opposite Bank ShiftTo 4WhHi Dry Brakes

(Platoon “B” Section

STEP 1

TASK ANALYSIS to Create Task Decomposition Tree

STEP

MAP to ARCHITE

SLIDE 49

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

49

ConductRoadMarchToAnAssemblyArea(AA) Conduct RouteRecon PrepareFor RoadMarch FollowPlatoon ToAssemblyArea Secure AssemblyArea Organize AssemblyArea Support MarchColumn Execute RoadMarch PlanRoute, ControlPoints SetupMarch ColumnOrg SetupSections, Formantions, MoveTechniques Conduct MainRoute Recon DeployTo StartPoint Conduct LeftFlank RouteRecon Locate&Secure ObstacleBypass Conduct Obstacle Recon PlanRoute, ControlPoints Conduct RightFlank RouteRecon FormRoad MarchOrg MoveInto MarchFormation PrepareDetailed MovementPlans Execute Schedule Halt MoveTo ControlPoint Conduct DominateTerrain Recon Conduct AmbushSite Recon Preform Traveling Overwatch BoundTo Overwatch Evalua Obstru MoveInto Formation MoveTo Cover/Concealed Position Ford Water Obstacle Secure Area Overwatch Section Perform Ford Recon Locate WaterBypass MoveTo ControlPoint Assess FordTerrain MoveTo Position Scan Path Cross Ford ScanArea ForEnemy MoveTo Water ShiftTo 4WhLo MoveTo Opposite Bank ShiftTo 4WhHi Dry Brakes

Task Vocabulary at Each Echelon

SLIDE 50

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

50

ConductReconToControlPoint

Vehicle | PLAN SELECTION TABLE

ple) Hierarchical Organization of Agent Control Modules

nize emblyArea MoveInto MarchFormation

• nduct
• minateTerrain

econ cealed Overwatch Section iftTo hHi Dry Brakes

MostlyTrees,SomeBushes SlowMovingWaterCoveredLand ExtensiveWaterSurfaceVisible

SwampDetected

Scout Platoon B Section Vehicle #5

HM M W V #6

Vehicle #8 A Section Vehicle #7 Vehicle #3 Vehicle #2 C Section Vehicle #10 Vehicle #9 Vehicle #4 PSG

ConductRightOfRouteRecon ConductM ainRouteRecon ConductLeftOfRouteRecon Conduct Traveling Overwatch Visually Clear Route Conduct Traveling Overwatch Conduct Recon ToCP Conduct Traveling Overwatch Conduct Route Recon To CP Support Route Recon W aterObsacle Detected ConductW aterObsRecon

ConductRouteRecon

TacticallyAssess W aterFeature Conduct ReconToCP

Surveil (RSTA) Lethality (Gunner) Mobility

Pan/Tilt Control W eapon Control Targeting Sensors Pan/Tilt Control

FLIR, Laser Radar, Cam era

Vehicle #6

Elem ental Move

Visually Clear Route

Pan/Tilt Control Ladar & Color Cam era

Engine Control

Steer Control

VehPrim / Propulsion

Gaze Control TransXfer Control Elem ental Moves Speed Control

ParkBrake Control Analyze W aterFeature LookFor Objects M akeVehRdy ForDriving Analyze Terrain W aterFeature Detected W aterObjects Detected M

• veTactically

ToGoalPt Startup Vehicle LookFor Enem yPresence StartEngine Assess Path PrepForStart SetParkBrake PrepForStart PrepForStart

SENSORY PROCESSING KNOWLEDGE DATABASE BEHAVIOR GENERATION WORLD MODEL VALUE JUDGMENT (Executing) (ReconToRout (Executing)

SENSORY INPUT

(Platoon “B” Section Agent Control Module) (ConductRec BE GEN

ST STA

STEP 2

MAP to AGENT ARCHITECTURE MAP TASK to STATE

STE

SLIDE 51

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

51

NormalReconSituation ConductReconToControlPoint TacticallyAssessGully GullyFeatureDetected

TacticallyAssessWaterFeature

WaterFeatureDetected

ConductReconToControlPoint

Vehicle | PLAN SELECTION TABLE

ple) Hierarchical Organization of Agent Control Modules

nize emblyArea MoveInto MarchFormation

• nduct
• minateTerrain

econ cealed Overwatch Section iftTo hHi Dry Brakes

StagnantWater OrganicMaterialOnWaterSurface

very flat, long green se/yellow flowers. tan/green stem plants, green to tan grass-

MostlyTrees,SomeBushes SlowMovingWaterCoveredLand SignificantTractionSlip ExtensiveWaterSurfaceVisible

SwampDetected

MajorGroundDeformation Scout Platoon B Section Vehicle #5

HM M W V #6

Vehicle #8 A Section Vehicle #7 Vehicle #3 Vehicle #2 C Section Vehicle #10 Vehicle #9 Vehicle #4 PSG

ConductRightOfRouteRecon ConductM ainRouteRecon ConductLeftOfRouteRecon Conduct Traveling Overwatch Visually Clear Route Conduct Traveling Overwatch Conduct Recon ToCP Conduct Traveling Overwatch Conduct Route Recon To CP Support Route Recon W aterObsacle Detected ConductW aterObsRecon

ConductRouteRecon

TacticallyAssess W aterFeature Conduct ReconToCP

Surveil (RSTA) Lethality (Gunner) Mobility Pan/Tilt Control W eapon Control

Targeting Sensors Pan/Tilt Control FLIR, Laser Radar, Cam era

Vehicle #6

Elem ental Move Visually Clear Route Pan/Tilt Control Ladar & Color Cam era Engine Control Steer Control VehPrim / Propulsion Gaze Control TransXfer Control Elem ental Moves Speed Control ParkBrake Control Analyze W aterFeature LookFor Objects M akeVehRdy ForDriving Analyze Terrain W aterFeature Detected W aterObjects Detected M

• veTactically

ToGoalPt Startup Vehicle LookFor Enem yPresence StartEngine Assess Path PrepForStart SetParkBrake PrepForStart PrepForStart

WaterFeatureDetected

SENSORY PROCESSING KNOWLEDGE DATABASE BEHAVIOR GENERATION WORLD MODEL VALUE JUDGMENT (Executing) (ReconToRoute) (Executing)

SENSORY INPUT

(Platoon “B” Section Agent Control Module) (ConductReconToControlPt) BEHAVIOR GENERATION

STATUS STATUS NEXT SUBGOAL

.

COMMANDED TASK (GOAL) DriveOnTwoLaneRd PassVehInFront PassVehInFront DriveOnTwoLaneRd NegotiateLaneConstrict PassVehInFront. STATE-TABLES

STEP 2

MAP to AGENT ARCHITECTURE MAP TASK DECISION to STATE-TABLES

STEP 3

SLIDE 52

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

52

Scout Platoon B Section Vehicle #5

HMMWV #6

Vehicle #8 A Section Vehicle #7 Vehicle #3 Vehicle #2 C Section Vehicle #10 Vehicle #9 Vehicle #4 PSG

ConductRightOfRouteRecon ConductMainRouteRecon ConductLeftOfRouteRecon Conduct Traveling Overwatch Visually Clear Route Conduct Traveling Overwatch Conduct Recon ToCP Conduct Traveling Overwatch Conduct Route Recon To CP Support Route Recon WaterObsacle Detected ConductWaterObsRecon

ConductRouteRecon

TacticallyAssess WaterFeature Conduct ReconToCP

Surveil (RSTA) Lethality (Gunner) Mobility

Pan/Tilt Control Weapon Control Targeting Sensors Pan/Tilt Control

FLIR, Laser Radar, Camera

Vehicle #6

Elemental Move

Visually Clear Route

Pan/Tilt Control Ladar & Color Camera

Engine Control

Steer Control

VehPrim/ Propulsion

Gaze Control TransXfer Control Elemental Moves Speed Control

ParkBrake Control

Analyze WaterFeature LookFor Objects MakeVehRdy ForDriving Analyze Terrain WaterFeature Detected WaterObjects Detected MoveTactically ToGoalPt Startup Vehicle LookFor EnemyPresence StartEngine Assess Path PrepForStart SetParkBrake PrepForStart PrepForStart

Agent Architecture

SLIDE 53

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

53

NormalReconSituation ConductReconToControlPoint TacticallyAssessGully GullyFeatureDetected

TacticallyAssessWaterFeature

WaterFeatureDetected DefileConstrictionDetected TacticallyAssessDefile ReinforcingObstacleDetected FallbackToStandoffPosition EnemyPresenceDetected PerformStealthyHalt NonEngagedEnemyDetected FallbackToStandoffPosition DangerAreaDetected PerformStealthyHalt KeyDominatingTerrainDetected TacticallyAssessKeyTerrain RestrictingTerrainDetected TacticallyAssessRestrictingTerrain NonPassableObstacleDetected PerformStealthyHalt TunnelDetected PerformStealthyHalt BridgeDetected PerformStealthyHalt ManMadeObjectsDetected TacticallyAssessManMadeObjects AvenueOfApproachDetected PerformStealthyHalt EngagedEnemyContact FallbackWithFires BuiltUpAreaDetected FallbackToStandoffPosition

ConductReconToControlPoint

Vehicle | PLAN SELECTION TABLE

VehicleInoperative PerformOwnVehDisabledProcedures

PLAN STATE-TABLE Input Conditions Output Commands

S1 WaterFeatureNotImpactMovement S0 NormalMovementPossible NewPlan S1 DetermineIfWaterIsObstacleToMovement PlanObservationGoalPt surv_LookForBypass leth_LookForEnemyOverwatch

Vehicle | TacticallyAssessWaterFeature

S1 WaterFeatureTraversable S3 PlanWaterTraverseGoalPts SetupWaterFeatureBypassReport comm_SendReport surv_AssessWaterFeatureTraversePath S3 ClearOfWaterFeature S0 NormalMovementPossible S1 NewObservationGoalPtPlanned S1 DetermineIfWaterIsObstacleToMovement PlanObservationGoalPt SetupMovementParams mob_MoveToObservationGoalPt S1 WaterFeatureIsMovementObstacle S2 SetupWaterObstacleReport PlanObstacleOverwatch mob_MoveToObstacleOverwatchPosition surv_AssessForEnemyPresence S3 PermissionToTraverseFeature NewWaterTraverseGoalPtPlanned S3 SetupMovementParams mob_MoveToBypassGoalPt PlanWaterTraverseGoalPts S2 SetAtObstacleOverwatch S0 SetupWaterObstacleOverwatchReport comm_SendReport

Hierarchical Organization of Agent Control Modules

Situations d States

SixFeet Land eformation

LegalToPass

MarshDetected

ible

ns,andShrubs undLevel nSlip nWaterSurface

BogDetected

ankments terInMiddle terminantLength

StreamDetected

ankments terInMiddle IndeterminantLength

RiverDetected

• undSurface

tion nSlip ingWater

MudDetected

dDeformation

shVegetation

OfStillWater tGrasses,Trees,Shrubs ayBeOnWaterSurface

Pond/LakeDetected

p,Marsh Bushes CoveredLand nSlip rfaceVisible

SwampDetected

rmation

WaterFeatureDetected

(Platoon “B” Section

MAP TASK DECISIONS to STATE-TABLES

STEP 3

DETERMINE ANTECEDENT WORLD STATES

STEP 4

SLIDE 54

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

54

NormalRe GullyFeatu WaterFeat DefileCons Reinforcin EnemyPre NonEngag DangerAre KeyDomin Restricting NonPassa TunnelDet BridgeDet ManMadeO AvenueOf EngagedE BuiltUpAre

Con

Veh

VehicleIno

WORLD MODEL KNOWLEDGE SENSORY PROCESSING

Situations World States Objects & Attributes

ColorCameras LADAR Radar Stereo FLIR Nav Segmented Groupings Features and Attributes Objects and Maps Object Groupings and Classifications

Cattails

WaterDepthToSixFeet WaterCoveredLand MajorGroundDeformation TractionSlip

LegalToPass

MarshDetected

SomeWaterVisible

Mosses,Evergreens,andShrubs StagnantWater IndeterminantGroundLevel SignificantTractionSlip OrganicMaterialOnWaterSurface

BogDetected

ErodedEarthEmbankments FlowingWater NonVegetatedWaterInMiddle NarrowWidth,IndeterminantLength

StreamDetected

ErodedEarthEmbankments FlowingWater NonVegetatedWaterInMiddle SignificantWidth,IndeterminantLength

RiverDetected

WaterSheenOnGroundSurface LittleToNoVegetation SignificantTractionSlip RuttedWithStandingWater

MudDetected

SignificantGroundDeformation

ExtensiveMarshVegetation

LongLeafGrasses - very flat, long green leaves-purple/rose/yellow flowers. Reeds - tall, woody, thin, round, hollow jointed (tan-to-green) stem plants, long narrow green blade leaves, large feathery panicles (elongated clusters of tan/white/purple flowers along main stem). Sedges - triangular tan/green stem plants, papyrus, narrow green to tan grass- like leaves, spikelets of inconspicuous tan-to-yellow-to-white flowers. Water Lilies - very large floating green leaves with white flowers. Saturated Ground Six Feet

• f Water

Plant Height 6-18" 6-48" 1-6' 3-9' Float

LargeSurfaceAreaOfStillWater LargeAreaWithoutGrasses,Trees,Shrubs OrganicMaterialMayBeOnWaterSurface

Pond/LakeDetected

BoundedBySwamp,Marsh MostlyTrees,SomeBushes SlowMovingWaterCoveredLand SignificantTractionSlip ExtensiveWaterSurfaceVisible

SwampDetected

MajorGroundDeformation

15 cm 2.4 cm .9 to 2.7m 3 cm that explode into white down, long dark brown cylindrical seed heads Bulrushes - tall tan-to-green stems, with flat green sword shaped leaves, cattails.

WaterFe

(Platoon “B” Section

DETERMINE ANTECEDENT WORLD STATES

STEP 4

IDENTIFY OBJECTS and THEIR RELEVANT ATTRIBUTES

STEP 5

DETERMINE SENSOR PROCESSING REQUIREMENTS AND RESOLUTIONS

STEP 6

Determine Transition Conditions Identify Objects and Attributes Determine Sensor Requirements

SLIDE 55

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

55

Capture Behavioral Knowledge

4D/RCS Node

NormalReconSituation ConductReconToControlPoint TacticallyAssessGully GullyFeatureDetected TacticallyAssessWaterFeature WaterFeatureDetected DefileConstrictionDetected TacticallyAssessDefile ReinforcingObstacleDetected FallbackToStandoffPosition EnemyPresenceDetected PerformStealthyHalt NonEngagedEnemyDetected FallbackToStandoffPosition DangerAreaDetected PerformStealthyHalt KeyDominatingTerrainDetected TacticallyAssessKeyTerrain RestrictingTerrainDetected TacticallyAssessRestrictingTerrain NonPassableObstacleDetected PerformStealthyHalt TunnelDetected PerformStealthyHalt BridgeDetected PerformStealthyHalt ManMadeObjectsDetected TacticallyAssessManMadeObjects AvenueOfApproachDetected PerformStealthyHalt EngagedEnemyContact FallbackWithFires BuiltUpAreaDetected FallbackToStandoffPosition

ConductReconToControlPoint

Vehicle | PLAN SELECTION TABLE VehicleInoperative PerformOwnVehDisabledProcedures PLAN STATE-TABLE Input Conditions Output Commands S1 WaterFeatureNotImpactMovement S0 NormalMovementPossible NewPlan S1 DetermineIfWaterIsObstacleToMovement PlanObservationGoalPt surv_LookForBypass leth_LookForEnemyOverwatch Vehicle | TacticallyAssessWaterFeature S1 WaterFeatureTraversable S3 PlanWaterTraverseGoalPts SetupWaterFeatureBypassReport comm_SendReport surv_AssessWaterFeatureTraversePath S3 ClearOfWaterFeature S0 NormalMovementPossible S1 NewObservationGoalPtPlanned S1 DetermineIfWaterIsObstacleToMovement PlanObservationGoalPt SetupMovementParams mob_MoveToObservationGoalPt S1 WaterFeatureIsMovementObstacle S2 SetupWaterObstacleReport PlanObstacleOverwatch mob_MoveToObstacleOverwatchPosition surv_AssessForEnemyPresence S3 PermissionToTraverseFeature NewWaterTraverseGoalPtPlanned S3 SetupMovementParams mob_MoveToBypassGoalPt PlanWaterTraverseGoalPts S2 SetAtObstacleOverwatch S0 SetupWaterObstacleOverwatchReport comm_SendReport DOT Driving Manuals and ARMY Field Manuals

+

Domain Experts

Task Decomposition Tree (Route Reconnaissance Example) Hierarchical Organization of Agent Control Modules

BEHAVIOR GENERATION WORLD MODEL KNOWLEDGE SENSORY PROCESSING

Situations World States Objects & Attributes

ColorCameras LADAR Radar Stereo FLIR Nav Segmented Groupings Features and Attributes Objects and Maps Object Groupings and Classifications

Cattails

ConductRoadMarchToAnAssemblyArea(AA) Conduct RouteRecon PrepareFor RoadMarch FollowPlatoon ToAssemblyArea Secure AssemblyArea Organize AssemblyArea Conduct MainRoute Recon DeployTo StartPoint Conduct LeftFlank RouteRecon Locate&Secure ObstacleBypass Conduct Obstacle Recon Conduct RightFlank RouteRecon MoveInto MarchFormation PrepareDetailed MovementPlans MoveTo ControlPoint Conduct DominateTerrain Recon MoveTo Cover/Concealed Position Ford Water Obstacle Secure Area Overwatch Section Perform Ford Recon Locate WaterBypass MoveTo ControlPoint Assess FordTerrain MoveTo Position Scan Path Cross Ford ScanArea ForEnemy MoveTo Water ShiftTo 4WhLo MoveTo Opposite Bank ShiftTo 4WhHi Dry Brakes WaterDepthToSixFeet WaterCoveredLand MajorGroundDeformation TractionSlip LegalToPass MarshDetected SomeWaterVisible Mosses,Evergreens,andShrubs StagnantWater IndeterminantGroundLevel SignificantTractionSlip OrganicMaterialOnWaterSurface BogDetected ErodedEarthEmbankments FlowingWater NonVegetatedWaterInMiddle NarrowWidth,IndeterminantLength StreamDetected ErodedEarthEmbankments FlowingWater NonVegetatedWaterInMiddle SignificantWidth,IndeterminantLength RiverDetected WaterSheenOnGroundSurface LittleToNoVegetation SignificantTractionSlip RuttedWithStandingWater MudDetected SignificantGroundDeformation ExtensiveMarshVegetation LongLeafGrasses - very flat, long green leaves-purple/rose/yellow flowers. Reeds - tall, woody, thin, round, hollow jointed (tan-to-green) stem plants, long narrow green blade leaves, large feathery panicles (elongated clusters of tan/white/purple flowers along main stem). Sedges - triangular tan/green stem plants, papyrus, narrow green to tan grass- like leaves, spikelets of inconspicuous tan-to-yellow-to-white flowers. Water Lilies - very large floating green leaves with white flowers. Saturated Ground Six Feet

• f Water

Plant Height 6-18" 6-48" 1-6' 3-9' Float LargeSurfaceAreaOfStillWater LargeAreaWithoutGrasses,Trees,Shrubs OrganicMaterialMayBeOnWaterSurface Pond/LakeDetected BoundedBySwamp,Marsh MostlyTrees,SomeBushes SlowMovingWaterCoveredLand SignificantTractionSlip ExtensiveWaterSurfaceVisible SwampDetected MajorGroundDeformation 15 cm 2.4 cm .9 to 2.7m 3 cm Scout Platoon B Section Vehicle #5 HM M W V #6 Vehicle #8 A Section Vehicle #7 Vehicle #3 Vehicle #2 C Section Vehicle #10 Vehicle #9 Vehicle #4 PSG ConductRightOfRouteRecon ConductM ainRouteRecon ConductLeftOfRouteRecon Conduct Traveling Overwatch Visually Clear Route Conduct Traveling Overwatch Conduct Recon ToCP Conduct Traveling Overwatch Conduct Route Recon To CP Support Route Recon W aterObsacle Detected ConductW aterObsRecon ConductRouteRecon TacticallyAssess W aterFeature Conduct ReconToCP Surveil (RSTA) Lethality (Gunner) Mobility Pan/Tilt Control W eapon Control Targeting Sensors Pan/Tilt Control FLIR, Laser Radar, Cam era Vehicle #6 Elem ental Move Visually Clear Route Pan/Tilt Control Ladar & Color Cam era Engine Control Steer Control VehPrim / Propulsion Gaze Control TransXfer Control Elem ental Moves Speed Control ParkBrake Control Analyze W aterFeature LookFor Objects M akeVehRdy ForDriving Analyze Terrain W aterFeature Detected W aterObjects Detected M

• veTactically

ToGoalPt Startup Vehicle LookFor Enem yPresence StartEngine Assess Path PrepForStart SetParkBrake PrepForStart PrepForStart that explode into white down, long dark brown cylindrical seed heads Bulrushes - tall tan-to-green stems, with flat green sword shaped leaves, cattails. WaterFeatureDetected SENSORY PROCESSING KNOWLEDGE DATABASE BEHAVIOR GENERATION WORLD MODEL VALUE JUDGMENT (Executing) (ReconToRoute) (Executing) SENSORY INPUT (Platoon “B” Section Agent Control Module) (ConductReconToControlPt) BEHAVIOR GENERATION STATUS STATUS NEXT SUBGOAL

.

COMMANDED TASK (GOAL) DriveOnTwoLaneRd PassVehInFront PassVehInFront DriveOnTwoLaneRd NegotiateLaneConstrict PassVehInFront . STATE-TABLES

STEP 1

TASK ANALYSIS to Create Task Decomposition Tree

STEP 2

MAP to AGENT ARCHITECTURE MAP TASK DECISIONS to STATE-TABLES

STEP 3

DETERMINE ANTECEDENT WORLD STATES

STEP 4

IDENTIFY OBJECTS and THEIR RELEVANT ATTRIBUTES

STEP 5

DETERMINE SENSOR PROCESSING REQUIREMENTS AND RESOLUTIONS

STEP 6

SLIDE 56

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

56

GoOn_TurnRightOnto_ FollowRoute TurnRightAtIntersection TurnRightUsingExitRamp TurnRightUsingFork GoOn_TurnLeftOnto_ FollowRoute TurnLeftAtIntersection TurnLeftUsingExitRamp TurnLeftUsingFork TurnLeftUsingRightWithU Turn TurnLeftUsingUWithRight Turn GoOn_ StopAt_ FollowVehicle FollowRoute TurnLeftAtIntersection TurnLeftUsingExitRamp TurnLeftUsingFork TurnLeftUsingRightWithUTurn TurnLeftUsingUWithRightTurn TurnRightAtIntersection TurnRightUsingExitRamp TurnRightUsingFork TurnIntoDrive GoOn_Becomes_ RespondToEmerVeh RespondToOwnVehEmer RespondToSchoolBus RespondToTrafficPerson PullOntoRoad Make_U_Turn BackUp FollowRoad PassVehInFront DriveOnTwoLaneRd DriveOnMultiLaneRd PullOntoRoad ChangeLaneToGoFaster ChangeToGoalLane AccomodatePassingVehicle RespondToFollowingVeh NegotiateLaneConstriction NegotiateMovingConstriction RespondToPedestrian RespondToBicyclist RespondToVehEnteringLane DriveOnNarrowRoad RespondToOncomingPassingVeh CrossThru_Intersect CrossThru_StopSign CrossThru_YieldSign CrossThru_SignalLight CrossThru_UncontrolledInter CrossThru_TrafficPerson MergeInto_TravelLane AccomodateMerge Negotiate_RRCrossing Negotiate_TollBooth Negotiate_PedestrianCross Negotiate_GateKeeper TurnLeftAtInterTo_ TurnLeft_StopSign TurnLeft_YieldSign TurnLeft_SignalLight TurnLeft_UncontrolledInter TurnLeft_IntoDrive TurnLeft_FromDrive TurnLeft_IntoParkingSpace TurnLeft_TrafficPerson TurnRightAtInterTo_ TurnRight_StopSign TurnRight_YieldSign TurnRight_SignalLight TurnRight_UncontrolledInter TurnRight_IntoDrive TurnRight_FromDrive TurnRight_IntoParkingSpace TurnRight_TrafficPerson Make_U_Turn Do_U_TurnAtIntersection Do_U_TurnThruAccess TurnAroundUsingDrive TurnAroundInRoad TurnAround_TrafficPerson Backup__ BackupVehicle BackupIntoParallelPark BackupOutOfParkSpace BackLeftTo__ BackLeft_IntoLane BackLeft_IntoDrive BackLeft_IntoParkingSpace BackRightTo__ BackRight_IntoLane BackRight_IntoDrive BackRight_IntoParkingSpace

Destination Manager RouteSegment Manager

Uses maps, traffic, weather, and construction reports to select active mapquest-like

• utput command

DriveBehavior Manager

Detects and recognizes relevant vehicles and

• bjects, and determines

how they affect basic driving behaviors Receives goal lane with list

• f relevant vehicles and
• bjects along immediate

route and generates goal path to avoid collisions

GoalPath Trajectory

Receives commanded goal path and calculates real-time dynamically feasible trajectories that adapt to side skid and traction slip

Elemental Maneuver Subsystem

Reads signs, detects road intersections and decides

• n real-time road

changing maneuvers

On-road Driving

InitSubsystems StartupVehicle ShutDownVehicle TurnOffSubsystems Follow_StLine Follow_CirArcCW Follow_CirArcCCW Stop/Halt SetupForwardDirTraj SetupReverseDirTraj InitSubsystems StartupVehicle ShutDownVehicle TurnOffSubsystems FollowLane PassOnLeft PassOnRight TurnRightTo__ TurnLeftTo__ StopAt PullOff_OnLeftShoulder PullOff_OnRightShoulder GotoGap_LeftLane GotoGap_RightLane Premerge_LeftLane Premerge_RightLane ChangeTo_LeftLane ChangeTo_RightLane StopAtIntersection AbortPass CreepForward PeekForPass Backup BackOut_ToGoLeft BackOut_ToGoRight BackInto_FromLeft BackInto_FromRight DoUTurn_AtInter DoUTurn_MidRoad Do3Pt_UTurn CreepBackward AllowVehToEnter_FromLeft AllowVehToEnter_FromRight YieldToPassingVeh ReactToPassingVehAbort PullOntoRd_FromLeftSh PullOntoRd_FromRightSh InitializeSytem MakeVehOperational ShutDownVehicle TurnOffSystem Goto_Destination FollowVehicle InitializeSystems StartupVehicle TurnOffSystems ShutDownVehicle InitSubsystems StartUpVehicle ShutDownVehicle TurnOffSubsystems Fork_Right Fork_Left Merge_Right Merge_Left GoTo_RightExitRamp GoTo_LeftExitRamp BackOut_GoLeft BackOut_GoRight RespondTo_OwnVehEmer Accommodate_SchoolBus Accommodate_EmerVeh

Activities Agent Hierarchy with Commands

(Section) (Vehicle) (Mobility)

Servo Dynamics Mobility Vehicle Section

SLIDE 57

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

57

On-road Driving Analysis

WORLD MODEL

Service Drive Road Edge Service Drive Road Surface South Drive

• Service Drive

LaneSegments South Drive LaneSegments South Drive Road Edge

3.(bg1) GoOn SouthDrive - TurnRightOnto ServiceDrive 3.2(bg2) TurnRightOnto ServiceDrive Prim/ Propul BG3 Surveil SP1 10xColor GazeSP2 GDRS, SICK,CC Mobility BG2 Vehicle BG1

• W1. Apriori-line map

BuildsLaneMap 3.1(sp1) LookForIntersections ServiceDrive 3.2.1(sp2) LocateRoadFeatures 3.2.2(sp2) LocateRoadFeatures 3.2.1(bg3) FollowLane 3.2.2(bg3) TurnRight 3.2.1.1(bg4) GoAtAngle,GoAtSpeed 3.2.2.1(bg4) GoAtAngle,GoAtSpeed

• W2. LaneMap

BuildsLaneSegMap BuildsObjOfInterest BuildsActiveObj

• W3. DrivingLaneSegMap

ActiveObj

Vehicle has detected Service Drive intersection and is now within the turning distance tolerance. This causes BG2 to send a TurnRight command to BG3 which will cause it to build goal paths along the lane segments forming the right turn onto Service Drive. This turn will be made at speed since the vehicle has the right-of-way at this

• intersection. This has been determined by the lack of detecting any

traffic control devices for own lane of travel and the fact that Service Drive “tees” into South Drive which usually means that vehicles on South Drive will have the right-of-way. BG2 will continuously update changes in the LaneSegments as measured by SP2 detecting road edges and surfaces. The curving road edges leading into Service Drive continue to be of interest and are, therefore, still in the BG3 ActiveObjectsTable and the SP2 ObjectsOfAttention list. BG3 will continuously adapt the goal path to the LaneSegment changes from BG2 by controlling the real-time trajectory vector to Steer and Speed Control in BG4.

NARRATIVE

SLIDE 58

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

58

RCS Methodology

This is a tedious process.

There are many tasks in the command library at each level There are many parameters for each task There are many objects that must be recognized There are many situations that must be understood

But, the numbers are not infinite. They are, in fact,

quite modest. (One of the advantages of hierarchies.)

SLIDE 59

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

59

Autonomous On-Road Driving Vehicle Echelon and Down

~ 200 tasks ~ 100 parameters ~ 1000 transition conditions ~ 10,000 objects or events Other skills may require similar numbers Estimated numbers:

SLIDE 60

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

60

Summary

4D/RCS Reference Model Architecture has a proven success record for intelligent control 4D/RCS Methodology provides a systematic approach to software engineering for tactical behaviors

SLIDE 61

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

61

4D/RCS Documentation

4D/RCS Version 2.0

• NIST Report, 2002

Numerous journal articles, reports, and conference papers Extensive software library http://www.isd.mel.nist.gov/projects/rcslib

RCS Handbook

– Wiley, 2001

Engineering of Mind

• Wiley, 2001

SLIDE 62

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

62

Most Recent Publication 2007

SLIDE 63

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

63

Conclusions

• 2. Human level performance in autonomous on-road

and off-road driving will be feasible by 2020

• 1. Useful autonomous on-road and off-road driving

will be feasible by 2010

• 3. Future Combat System will provide the rational

and funding to build intelligent vehicle systems

SLIDE 64

NIST • Manufacturing Engineering Laboratory • Intelligent Systems Division

64

Questions?