[PPT] - Chapter 3 Deliberation with Refinement Automated Planning and PowerPoint Presentation

SLIDE 1

1 Nau – Lecture slides for Automated Planning and Acting

Automated Planning and Acting

Malik Ghallab, Dana Nau and Paolo Traverso

Last update: April 1, 2020

http://www.laas.fr/planning

Licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License

Chapter 3 Deliberation with Refinement Methods

Dana S. Nau University of Maryland

SLIDE 2

2 Nau – Lecture slides for Automated Planning and Acting

ungrasp grasp knob turn knob maintain move back pull monitor identify type

f

door pull monitor move close to knob

pen door

… … get out close door respond to user requests

… …

bring o7 to room2 go to hallway deliver

7

… … … … … move to door fetch

7

navigate to room2 navigate to room1

Motivation

Hierarchically organized

deliberation ▸ At high levels, abstract actions ▸ At lower levels, more detail

Refine abstract actions

into ways of carrying out those actions ▸ How? Planning Acting

SLIDE 3

3 Nau – Lecture slides for Automated Planning and Acting

Hinged door that opens to the left, toward robot ungrasp grasp knob turn knob maintain move back pull monitor identify type

f

door pull monitor move close to knob

pen door

… … get out close door … move to door

Opening a Door

Many different methods,

depending on what kind

f door

▸ Sliding or hinged?

identify type

f

door

SLIDE 4

4 Nau – Lecture slides for Automated Planning and Acting

Opening a Door

Many different methods,

depending on what kind

f door

▸ Sliding or hinged? ▸ Hinge on left or right?

Hinged door that opens to the left, toward robot ungrasp grasp knob turn knob maintain move back pull monitor identify type

f

door pull monitor move close to knob

pen door

… … get out close door … move to door identify type

f

door

SLIDE 5

5 Nau – Lecture slides for Automated Planning and Acting

Opening a Door

Many different methods,

depending on what kind

f door

▸ Sliding or hinged? ▸ Hinge on left or right? ▸ Open toward or away?

Hinged door that opens to the left, toward robot ungrasp grasp knob turn knob maintain move back pull monitor identify type

f

door pull monitor move close to knob

pen door

… … get out close door … move to door identify type

f

door

SLIDE 6

6 Nau – Lecture slides for Automated Planning and Acting

Opening a Door

Many different methods,

depending on what kind

f door

▸ Sliding or hinged? ▸ Hinge on left or right? ▸ Open toward or away? ▸ Knob, lever, push bar, …

Hinged door that opens to the left, toward robot ungrasp grasp knob turn knob maintain move back pull monitor identify type

f

door pull monitor move close to knob

pen door

… … get out close door … move to door identify type

f

door

SLIDE 7

7 Nau – Lecture slides for Automated Planning and Acting

Opening a Door

Many different methods,

depending on what kind

f door

▸ Sliding or hinged? ▸ Hinge on left or right? ▸ Open toward or away? ▸ Knob, lever, push bar, pull handle, push plate, …

Hinged door that opens to the left, toward robot ungrasp grasp knob turn knob maintain move back pull monitor identify type

f

door pull monitor move close to knob

pen door

… … get out close door … move to door identify type

f

door

SLIDE 8

8 Nau – Lecture slides for Automated Planning and Acting

Opening a Door

Many different methods,

depending on what kind

f door

▸ Sliding or hinged? ▸ Hinge on left or right? ▸ Open toward or away? ▸ Knob, lever, push bar, pull handle, push plate, something else?

Hinged door that opens to the left, toward robot ungrasp grasp knob turn knob maintain move back pull monitor identify type

f

door pull monitor move close to knob

pen door

… … get out close door … move to door identify type

f

door

SLIDE 9

9 Nau – Lecture slides for Automated Planning and Acting

Outline

3.1 Representation

a. State variables, commands, refinement methods
b. Example

3.2 Acting

a. Rae (Refinement Acting Engine)
b. Example
c. Extensions

3.3 Planning

a. Motivation and basic ideas
b. Deterministic action models
c. SeRPE (Sequential Refinement Planning Engine)

3.4 Using Planning in Acting

a. Techniques
b. Caveats

SLIDE 10

10 Nau – Lecture slides for Automated Planning and Acting

State-Variable Representation

Quick review:

Objects: Robots = {r1}, Containers = {c1, c2, c3, …},

Locations = {loc0, loc1, loc2, …}

State variables: syntactic terms

to which we can assign values ▸ loc(r) ∈ Locations ▸ load(r) ∈ Containers ⋃ {nil} ▸ pos(c) ∈ Locations ⋃ Robots ⋃ {unknown} ▸ view(r,l) ∈ {T, F} – whether robot r has looked at location l

r can only see what’s at its current location
State: assign a value to each state variable

▸ {loc(r1) = loc0, pos(c1) = loc2, pos(c3) = loc4, pos(c2) = unknown, …}

Details: Automated Planning and Acting, Sections 2.1 and 3.1.1 … loc2

loc0

loc1

r1

c1

loc4

loc3

c2

SLIDE 11

11 Nau – Lecture slides for Automated Planning and Acting

State-Variable Representation

Extensions:

Range(x)

▸ can be finite, infinite, continuous, discontinuous, vectors, matrices, other data structures

Assignment statement x ← expr

▸ expression that returns a ground value in Range(x) and has no side-effects on the current state

Tests (e.g., preconditions)

▸ Simple: x = v, x ≠ v, x > v, x < v ▸ Compound: conjunction, disjunction,

r negation of simple tests

… loc2

loc0

loc1

r1

c1

loc4

loc3

c2

SLIDE 12

12 Nau – Lecture slides for Automated Planning and Acting

Acting

tasks Execution Platform Environment events commands

! M

Commands

Command: primitive function that

the execution platform can perform ▸ take(r,o,l): robot r takes object o at location l ▸ put(r,o,l): r puts o at location l ▸ perceive(r,l): robot r perceives what objects are at l

r can only perceive what’s at its

current location ▸ …

SLIDE 13

13 Nau – Lecture slides for Automated Planning and Acting

Hinged door that opens to the left, toward robot ungrasp grasp knob turn knob maintain move back pull monitor identify type

f

door pull monitor move close to knob

pen door

… … get out close door … move to door

pen door

Tasks and Methods

Task: an activity for the

actor to perform

For each task, a set of

refinement methods ▸ Operational models:

tell how to perform the task
don’t predict what it will do
assignment statements
control constructs: if-then-else, while, … .
tasks
can extend to include events, goals
commands to the execution platform

method-name(arg1, …, argk) task: task-identifier pre: test body: a program

SLIDE 14

14 Nau – Lecture slides for Automated Planning and Acting

Hinged door that opens to the left, toward robot ungrasp grasp knob turn knob maintain move back pull monitor identify type

f

door pull monitor move close to knob

pen door

… … get out close door … move to door

pen door

Opening a Door

m-opendoor(r,d,l,h) task: opendoor(r,d) pre: loc(r) = l ∧ adjacent(l,d) ∧ handle(d,h) body: while ¬reachable(r,h) do move-close(r,h) monitor-status(r,d) if door-status(d) = closed then unlatch(r,d) throw-wide(r,d) end-monitor-status(r,d) m1-unlatch(r,d,l,o) task: unlatch(r,d) pre: loc(r,l) ∧ toward-side(l,d) ∧ side(d,left) ∧ type(d,rotate) ∧ handle(d,o) body: grasp(r,o) turn(r,o,alpha1) pull(r,val1) if door-status(d) = cracked then ungrasp(r,o) else fail

= What kind:

Ø

Hinged on left, opens toward us, lever handle m1-throw-wide(r,d,l,o) task: throw-wide(r,d) pre: loc(r,l) ∧ toward-side(l,d) ∧ side(d,left) ∧ type(d,rotate) ∧ handle(d,o) ∧ door-status(d) = cracked body: grasp(r,o) pull(r,val1) move-by(r,val2)

SLIDE 15

15 Nau – Lecture slides for Automated Planning and Acting

Outline

3.1 Representation

a. State variables, commands, refinement methods
b. Example

3.2 Acting

a. Rae (Refinement Acting Engine)
b. Example
c. Extensions

3.3 Planning

a. Motivation and basic ideas
b. Deterministic action models
c. SeRPE (Sequential Refinement Planning Engine)

3.4 Using Planning in Acting

a. Techniques
b. Caveats

SLIDE 16

16 Nau – Lecture slides for Automated Planning and Acting

Acting

tasks Execution Platform Environment events commands

! M

Rae (Refinement Acting Engine)

Based on OpenPRS

▸ Programming language,

pen-source robotics software

▸ Deployed in many applications

Input:

external tasks, events, current state

Output: commands to execution platform
Perform multiple tasks/events in parallel

▸ Purely reactive, no lookahead

For each task/event, a refinement stack

▸ current path in Rae’s search tree for the task/event

Agenda = {all current refinement stacks}

SLIDE 17

17 Nau – Lecture slides for Automated Planning and Acting

Rae (Refinement Acting Engine)

Stack element (τ,m,i,tried) τ: task m: method i: instruction pointer tried: methods already tried

Agenda = {current refinement stacks}

▸ One for each external task or event

Summary of Rae: loop: for every new external task or event τ do choose a method instance m for τ create a refinement stack and add it to Agenda for each stack in Agenda Progress it, and remove it if it’s finished

SLIDE 18

18 Nau – Lecture slides for Automated Planning and Acting

Progress (subroutine)

is m’s current step a command? command status? return success running failed choose a candidate m′ push (τʹ, m′,…) onto stack no more steps in m? pop(σ) no candidates for τʹ? yes yes τʹ ← next step of m type(τʹ) task command send a to the execution platform assignment update state retry τ using an untried candidate yes retry τ using an untried candidate suc ceeded started m? yes no no

SLIDE 19

19 Nau – Lecture slides for Automated Planning and Acting

is m’s current step a command? command status? return success running failed choose a candidate m′ push (τʹ, m′,…) onto stack no more steps in m? pop(σ) no candidates for τʹ? yes yes τʹ ← next step of m type(τʹ) task command send a to the execution platform assignment update state retry τ using an untried candidate yes retry τ using an untried candidate suc ceeded started m? yes no no

Poll. Is it possible to

have situations where both of these have the answer “no”?

SLIDE 20

20 Nau – Lecture slides for Automated Planning and Acting

loc2

loc0

loc1

c1

loc4

loc3

c2

r1

Example

Objects

▸ Robots = {r1} ▸ Containers = {c1, c2, …} ▸ Locations = {loc1, loc2, …}

State variables

▸ loc(r) ∈ Locations ▸ load(r) ∈ Containers ⋃ {nil} ▸ pos(c) ∈ Locations ⋃ Robots ⋃ {unknown} ▸ view(l) ∈ {T, F}

Whether the robot has looked at location l or not
If view(l) = T then for every container c at location l, pos(c) = l
Commands to the execution platform:

▸ take(r,o,l): r takes object o at location l ▸ put(r,o,l): r puts o at location l ▸ perceive(r,l): robot r perceives what objects are at location l ▸ move-to(r,l): robot r moves to location l

SLIDE 21

21 Nau – Lecture slides for Automated Planning and Acting

take(r1,c2,loc2) search(r1,c2) m-fetch(r1,c2) m-search(r1,c2) move-to(r1,loc1) m-search(r1,c2) perceive(loc1) move-to(r1,loc2) perceive(loc2) … search(r1,c2) fetch(r1,c2) … …

Example

m-search(r,c) task: search(r,c) pre: pos(c) = unknown body: if ∃l (view(r,l) = F) then move-to(r,l) perceive(l) if pos(c) = l then take(r,c,l) else search(r,c) else fail m-fetch(r,c) task: fetch(r,c) pre: body: if pos(c) = unknown then search(r,c) else if loc(r) = pos(c) then take(r,c,pos(c)) else do move-to(r,pos(c)) take(r,c,pos(c)) ? τ = fetch(r1,c2) Candidates = {m-fetch(r1,c2)} τ: fetch(r1,c2) m: m-fetch(r1,c2) i: nil tried: ∅ Refinement stack Add to Agenda: loc2

loc0

loc1

c1

loc4

loc3

c2

r1

SLIDE 22

22 Nau – Lecture slides for Automated Planning and Acting

m-fetch(r1,c2) fetch(r1,c2)

Search tree

Example

m-search(r,c) task: search(r,c) pre: pos(c) = unknown body: if ∃l (view(r,l) = F) then move-to(r,l) perceive(l) if pos(c) = l then take(r,c,l) else search(r,c) else fail m-fetch(r,c) task: fetch(r,c) pre: body: if pos(c) = unknown then search(r,c) else if loc(r) = pos(c) then take(r,c,pos(c)) else do move-to(r,pos(c)) take(r,c,pos(c)) τ: fetch(r1,c2) m: m-fetch(r1,c2) i: nil tried: ∅ Refinement stack Progress(stack):

is m’s current step a command? command status? return success running failed choose a candidate m′ push (τʹ, m′,…) onto stack no more steps in m? pop(σ) no candidates for τʹ? yes yes τʹ ← next step of m type(τʹ) task command send a to the execution platform assignment update state retry τ using an untried candidate yes retry τ using an untried candidate suc ceeded started m? yes no no

loc2

loc0

loc1

c1

loc4

loc3

c2

r1

SLIDE 23

23 Nau – Lecture slides for Automated Planning and Acting

m-fetch(r1,c2) fetch(r1,c2) search(r1,c2)

Example

m-search(r,c) task: search(r,c) pre: pos(c) = unknown body: if ∃l (view(r,l) = F) then move-to(r,l) perceive(l) if pos(c) = l then take(r,c,l) else search(r,c) else fail m-fetch(r,c) task: fetch(r,c) pre: body: if pos(c) = unknown then search(r,c) else if loc(r) = pos(c) then take(r,c,pos(c)) else do move-to(r,pos(c)) take(r,c,pos(c)) τ: fetch(r1,c2) m: m-fetch(r1,c2) i: (instruction pointer) tried: ∅ Refinement stack Candidates = {m-search(r,c)} Progress(stack):

is m’s current step a command? command status? return success running failed choose a candidate m′ push (τʹ, m′,…) onto stack no more steps in m? pop(σ) no candidates for τʹ? yes yes τʹ ← next step of m type(τʹ) task command send a to the execution platform assignment update state retry τ using an untried candidate yes retry τ using an untried candidate suc ceeded started m? yes no no

Search tree loc2

loc0

loc1

c1

loc4

loc3

c2

r1

SLIDE 24

24 Nau – Lecture slides for Automated Planning and Acting

m-fetch(r1,c2) m-search(r1,c2) fetch(r1,c2) search(r1,c2)

Example

m-search(r,c) task: search(r,c) pre: pos(c) = unknown body: if ∃l (view(r,l) = F) then move-to(r,l) perceive(l) if pos(c) = l then take(r,c,l) else search(r,c) else fail m-fetch(r,c) task: fetch(r,c) pre: body: if pos(c) = unknown then search(r,c) else if loc(r) = pos(c) then take(r,c,pos(c)) else do move-to(r,pos(c)) take(r,c,pos(c)) τ: fetch(r1,c2) m: m-fetch(r1,c2) i: (instruction pointer) tried: ∅ τ: search(r1,c2) m: m-search(r1,c2) i: nil tried: ∅ Refinement stack Progress(stack):

is m’s current step a command? command status? return success running failed choose a candidate m′ push (τʹ, m′,…) onto stack no more steps in m? pop(σ) no candidates for τʹ? yes yes τʹ ← next step of m type(τʹ) task command send a to the execution platform assignment update state retry τ using an untried candidate yes retry τ using an untried candidate suc ceeded started m? yes no no

Search tree loc2

loc0

loc1

c1

loc4

loc3

c2

r1

SLIDE 25

25 Nau – Lecture slides for Automated Planning and Acting

m-fetch(r1,c2) m-search(r1,c2) fetch(r1,c2) search(r1,c2) move-to(r1,loc1) code execution

Example

m-search(r,c) task: search(r,c) pre: pos(c) = unknown body: if ∃l (view(r,l) = F) then move-to(r,l) perceive(l) if pos(c) = l then take(r,c,l) else search(r,c) else fail m-fetch(r,c) task: fetch(r,c) pre: body: if pos(c) = unknown then search(r,c) else if loc(r) = pos(c) then take(r,c,pos(c)) else do move-to(r,pos(c)) take(r,c,pos(c)) τ: fetch(r1,c2) m: m-fetch(r1,c2) i: (instruction pointer) tried: ∅ τ: search(r1,c2) m: m-search(r1,c2) i: (instruction pointer) tried: ∅ Refinement stack executing … Progress(stack):

is m’s current step a command? command status? return success running failed choose a candidate m′ push (τʹ, m′,…) onto stack no more steps in m? pop(σ) no candidates for τʹ? yes yes τʹ ← next step of m type(τʹ) task command send a to the execution platform assignment update state retry τ using an untried candidate yes retry τ using an untried candidate suc ceeded started m? yes no no

Search tree … loc2

loc0

loc1

c1

loc4

loc3

c2

r1

SLIDE 26

26 Nau – Lecture slides for Automated Planning and Acting

Example

m-search(r,c) task: search(r,c) pre: pos(c) = unknown body: if ∃l (view(r,l) = F) then move-to(r,l) perceive(l) if pos(c) = l then take(r,c,l) else search(r,c) else fail m-fetch(r,c) task: fetch(r,c) pre: body: if pos(c) = unknown then search(r,c) else if loc(r) = pos(c) then take(r,c,pos(c)) else do move-to(r,pos(c)) take(r,c,pos(c)) τ: fetch(r1,c2) m: m-fetch(r1,c2) i: (instruction pointer) tried: ∅ τ: search(r1,c2) m: m-search(r1,c2) i: (instruction pointer) tried: ∅ succeeded executing … Progress(stack):

is m’s current step a command? command status? return success running failed choose a candidate m′ push (τʹ, m′,…) onto stack no more steps in m? pop(σ) no candidates for τʹ? yes yes τʹ ← next step of m type(τʹ) task command send a to the execution platform assignment update state retry τ using an untried candidate yes retry τ using an untried candidate suc ceeded started m? yes no no

succeeded Search tree

m-fetch(r1,c2) m-search(r1,c2) fetch(r1,c2) search(r1,c2) move-to(r1,loc1) perceive(loc1) code execution

loc2

loc0

loc1

c1

loc4

loc3

c2

r1

SLIDE 27

27 Nau – Lecture slides for Automated Planning and Acting

m-fetch(r1,c2) m-search(r1,c2) fetch(r1,c2) search(r1,c2) move-to(r1,loc1) perceive(loc1) code execution

Example

m-search(r,c) task: search(r,c) pre: pos(c) = unknown body: if ∃l (view(r,l) = F) then move-to(r,l) perceive(l) if pos(c) = l then take(r,c,l) else search(r,c) else fail m-fetch(r,c) task: fetch(r,c) pre: body: if pos(c) = unknown then search(r,c) else if loc(r) = pos(c) then take(r,c,pos(c)) else do move-to(r,pos(c)) take(r,c,pos(c)) τ: fetch(r1,c2) m: m-fetch(r1,c2) i: (instruction pointer) tried: ∅ τ: search(r1,c2) m: m-search(r1,c2) i: (instruction pointer) tried: ∅ Refinement stack succeeded sensor failure Progress(stack):

is m’s current step a command? command status? return success running failed choose a candidate m′ push (τʹ, m′,…) onto stack no more steps in m? pop(σ) no candidates for τʹ? yes yes τʹ ← next step of m type(τʹ) task command send a to the execution platform assignment update state retry τ using an untried candidate yes retry τ using an untried candidate suc ceeded started m? yes no no

succeeded Search tree loc2

loc0

loc1

c1

loc4

loc3

c2

r1

SLIDE 28

28 Nau – Lecture slides for Automated Planning and Acting

m-fetch(r1,c2) m-search(r1,c2) fetch(r1,c2) search(r1,c2) move-to(r1,loc1) perceive(loc1) code execution

Example

m-search(r,c) task: search(r,c) pre: pos(c) = unknown body: if ∃l (view(r,l) = F) then move-to(r,l) perceive(l) if pos(c) = l then take(r,c,l) else search(r,c) else fail m-fetch(r,c) task: fetch(r,c) pre: body: if pos(c) = unknown then search(r,c) else if loc(r) = pos(c) then take(r,c,pos(c)) else do move-to(r,pos(c)) take(r,c,pos(c)) τ: fetch(r1,c2) m: m-fetch(r1,c2) i: (instruction pointer) tried: ∅ Refinement stack failed succeeded sensor failure Progress(stack):

is m’s current step a command? command status? return success running failed choose a candidate m′ push (τʹ, m′,…) onto stack no more steps in m? pop(σ) no candidates for τʹ? yes yes τʹ ← next step of m type(τʹ) task command send a to the execution platform assignment update state retry τ using an untried candidate yes retry τ using an untried candidate suc ceeded started m? yes no no

τ: search(r1,c2) m: m-search(r1,c2) i: (instruction pointer) tried: ∅ ? succeeded retry Search tree loc2

loc0

loc1

c1

loc4

loc3

c2

r1

SLIDE 29

29 Nau – Lecture slides for Automated Planning and Acting

Retry (subroutine)

fail no stack empty?

Retry

yes (τ,m,i,tried) ← pop(stack) add m to tried untried candidates for τ? no yes choose one push onto stack

SLIDE 30

30 Nau – Lecture slides for Automated Planning and Acting

m-fetch(r1,c2) m-search(r1,c2) fetch(r1,c2) search(r1,c2) move-to(r1,loc1) perceive(loc1) code execution

Example

m-search(r,c) task: search(r,c) pre: pos(c) = unknown body: if ∃l (view(r,l) = F) then move-to(r,l) perceive(l) if pos(c) = l then take(r,c,l) else search(r,c) else fail m-fetch(r,c) task: fetch(r,c) pre: body: if pos(c) = unknown then search(r,c) else if loc(r) = pos(c) then take(r,c,pos(c)) else do move-to(r,pos(c)) take(r,c,pos(c)) ? τ: fetch(r1,c2) m: m-fetch(r1,c2) i: (instruction pointer) tried: ∅ Refinement stack failed succeeded succeeded sensor failure retry Retry(stack): τ: search(r1,c2) m: m-search(r1,c2) i: (instruction pointer) tried: ∅ fail no stack empty?

Retry

yes (τ,m,i,tried) ← pop(stack) add m to tried untried candidates for τ? no yes choose one push onto stack

pop

Poll: Is this the same as backtracking?

1. yes

2. no Search tree loc2

loc0

loc1

c1

loc4

loc3

c2

r1

SLIDE 31

31 Nau – Lecture slides for Automated Planning and Acting

m-fetch(r1,c2) m-search(r1,c2) fetch(r1,c2) search(r1,c2) move-to(r1,loc1) perceive(loc1) code execution

is m’s current step a command? command status? return success running failed choose a candidate m′ push (τʹ, m′,…) onto stack no more steps in m? pop(σ) no candidates for τʹ? yes yes τʹ ← next step of m type(τʹ) task command send a to the execution platform assignment update state retry τ using an untried candidate yes retry τ using an untried candidate suc ceeded started m? yes no no

∅

Example

m-search(r,c) task: search(r,c) pre: pos(c) = unknown body: if ∃l (view(r,l) = F) then move-to(r,l) perceive(l) if pos(c) = l then take(r,c,l) else search(r,c) else fail m-fetch(r,c) task: fetch(r,c) pre: body: if pos(c) = unknown then search(r,c) else if loc(r) = pos(c) then take(r,c,pos(c)) else do move-to(r,pos(c)) take(r,c,pos(c)) ? Refinement stack failed retry succeeded sensor failure failed failed Progress(stack): τ: fetch(r1,c2) m: m-fetch(r1,c2) i: (instruction pointer) tried: ∅ Search tree loc2

loc0

loc1

c1

loc4

loc3

c2

r1

SLIDE 32

32 Nau – Lecture slides for Automated Planning and Acting

Extensions to Rae

Events:

Example: an emergency

▸ If r isn’t already handing another emergency, then

stop what it’s doing
go handle the emergency

method-name(arg1, …, argk) event: event-identifier pre: test body: program m-emergency(r,l,i) event: emergency(l,i) pre: emergency-handling(r) = F body: emergency-handling(r) ← T if load(r) ≠ nil then put(r,load(r)) move-to(l) address-emergency(l,i)

SLIDE 33

33 Nau – Lecture slides for Automated Planning and Acting

Extensions to Rae

Goals:
Write as a special kind of task

▸ achieve(condition)

Like other tasks, but includes monitoring

▸ Modify Progress

if condition becomes true before finishing body(m),

stop early

if condition isn’t true after finishing body(m),

fail and try another method method-name(arg1, …, argk) task: achieve(condition) pre: test body: program Example: m-fetch(r,c) task: fetch(r,c) pre: body: if pos(c) = unknown then search(r,c) achieve(pos(c)≠unknown) else if loc(r) = pos(c) then take(r,c,pos(c)) else do move-to(r,pos(c)) take(r,c,pos(c))

SLIDE 34

34 Nau – Lecture slides for Automated Planning and Acting

Extensions to Rae

Concurrent subtasks:

▸ refinement stack for each one

Controlling the progress of tasks:

▸ e.g., suspend a task for a while ▸ If there are multiple stacks, which ones get higher priority?

Application-specific heuristics
For a task τ, which candidate to try first?

▸ Refinement planning

body of a method: … {concurrent: τ1, τ2, …, τn} … Agenda ={stack1, stack2, …, stackn} Candidates = Instances(τ,M, ξ) τ: … m: … i: (instruction pointer) tried: … Refinement stack τ: τ1 m: … i: … tried: … τ: τn m: … i: … tried: … …

SLIDE 35

35 Nau – Lecture slides for Automated Planning and Acting

Outline

3.1 Representation

a. State variables, commands, refinement methods
b. Example

3.2 Acting

a. Rae (Refinement Acting Engine)
b. Example
c. Extensions

3.3 Planning

a. Motivation and basic ideas
b. Deterministic action models
c. SeRPE (Sequential Refinement Planning Engine)

3.4 Using Planning in Acting

a. Techniques
b. Caveats

SLIDE 36

36 Nau – Lecture slides for Automated Planning and Acting

Motivation

Which candidate to choose?
Bad choices may lead to

▸ more costly solutions ▸ failure, need to recover ▸ unrecoverable failures

Idea: look ahead using

Rae’s refinement methods

SLIDE 37

37 Nau – Lecture slides for Automated Planning and Acting

Hinged door that opens to the left, toward robot

ungrasp grasp knob turn knob maintain move back pull monitor identify type

f

door pull monitor move close to knob

pen door

… … get out close door … move to door

pen door

How to Do the Planning?

m-opendoor(r,d,l,h) task: opendoor(r,d) pre: loc(r) = l ∧ adjacent(l,d) ∧ handle(d,h) body: while ¬reachable(r,h) do move-close(r,h) monitor-status(r,d) if door-status(d) = closed then unlatch(r,d) throw-wide(r,d) end-monitor-status(r,d)

SeRPE planner (described in the book)

▸ Step-by-step execution of Rae’s refinement methods, except:

Doesn’t execute the commands

▸ Classical action models to predict outcomes

Backtracking search over all possible choices
SeRPE generalizes a planner, SHOP, that uses HTN planning

▸ SHOP method body: list of tasks and classical actions ▸ depth-first search through space of refinements

SeRPE method body: the same program Rae uses

▸ SeRPE uses it to generate tasks and actions ▸ depth-first search through space of refinements

SLIDE 38

38 Nau – Lecture slides for Automated Planning and Acting

Hinged door that opens to the left, toward robot

ungrasp grasp knob turn knob maintain move back pull monitor identify type

f

door pull monitor move close to knob

pen door

… … get out close door … move to door

pen door

How to Do the Planning?

m-opendoor(r,d,l,h) task: opendoor(r,d) pre: loc(r) = l ∧ adjacent(l,d) ∧ handle(d,h) body: while ¬reachable(r,h) do move-close(r,h) monitor-status(r,d) if door-status(d) = closed then unlatch(r,d) throw-wide(r,d) end-monitor-status(r,d)

Backtracking in SHOP:

▸ method body: ⟨τ1, …, τi–1, τi, …, τn⟩ ▸ revert to si–1 and τi–1

Backtracking in SeRPE:

▸ Backtrack over statements in a computer program ▸ Need a language interpreter capable of handling this ▸ A group of students worked on this for several months, didn’t get it to work properly

Additional concern:

▸ Classical action models too limited for some of the environments where we’d like to use SeRPE

Result: stopped work on implementing SeRPE

current

SLIDE 39

39 Nau – Lecture slides for Automated Planning and Acting

Classical Action Models

= Commands:

Ø take(r,o,l):

robot r takes object o at location l

Ø put(r,o,l):

r puts o at location l

Ø perceive(r,l):

robot r perceives what objects are at l

can only perceive what’s

at its current location

= Action models:

take(r,o,l) pre: cargo(r) = nil, loc(r) = l, loc(o) = l eff: cargo(r) ← o, loc(o) ← r put(r,o,l) pre: loc(r) = l, loc(o) = r eff: cargo(r) ← nil, loc(o) ← l perceive(r,l): ? Poll: can we write a reasonable classical action model for perceive?

loc2

loc0

loc1

r1

c1

loc4

loc3

c2

SLIDE 40

40 Nau – Lecture slides for Automated Planning and Acting

a c b grasp(c) a c b a b c

Why use classical action models?

Given a state s, a classical action a always predicts

the same outcome ▸ Doesn’t deal with nondeterministic outcomes, partial observability, exogenous events

Can tolerate prediction errors if they’re infrequent

and don’t have severe consequences ▸ Much simpler planning algorithms ▸ Actor can fix the errors online

But not if the errors are more frequent and

the consequences are worse ▸ Need to reason about multiple outcomes

SLIDE 41

41 Nau – Lecture slides for Automated Planning and Acting

Hinged door that opens to the left, toward robot

ungrasp grasp knob turn knob maintain move back pull monitor identify type

f

door pull monitor move close to knob

pen door

… … get out close door … move to door

pen door

How to Do the Planning?

m-opendoor(r,d,l,h) task: opendoor(r,d) pre: loc(r) = l ∧ adjacent(l,d) ∧ handle(d,h) body: while ¬reachable(r,h) do move-close(r,h) monitor-status(r,d) if door-status(d) = closed then unlatch(r,d) throw-wide(r,d) end-monitor-status(r,d)

SeRPE planner (described in the book)

▸ Step-by-step execution of Rae’s refinement methods, except:

Doesn’t execute the commands

▸ Classical action models to predict outcomes Instead: simulated execution of commands

SLIDE 42

42 Nau – Lecture slides for Automated Planning and Acting

Hinged door that opens to the left, toward robot

ungrasp grasp knob turn knob maintain move back pull monitor identify type

f

door pull monitor move close to knob

pen door

… … get out close door … move to door

pen door

Is simulated execution feasible?

m-opendoor(r,d,l,h) task: opendoor(r,d) pre: loc(r) = l ∧ adjacent(l,d) ∧ handle(d,h) body: while ¬reachable(r,h) do move-close(r,h) monitor-status(r,d) if door-status(d) = closed then unlatch(r,d) throw-wide(r,d) end-monitor-status(r,d)

AI planning uses descriptive models to address two problems

▸ Search space size

AI planners search a huge search space, need fast

predictions ▸ Effort needed to create detailed operational models

Especially if you aren’t a domain expert
How to avoid those problems?

▸ Search space size

The operational models focus the search
Computers much more powerful than 20 years ago

⇒ easier to compute detailed simulations quickly ▸ Creating operational models

Perhaps use the actor’s control software?

SLIDE 43

43 Nau – Lecture slides for Automated Planning and Acting

is m’s current step a command? command status? return success running failed choose a candidate m′ push (τʹ, m′,…) onto stack no more steps in m? pop(σ) no candidates for τʹ? yes yes τʹ ← next step of m type(τʹ) task command send a to the execution platform assignment update state retry τ using an untried candidate yes retry τ using an untried candidate suc ceeded started m? yes no no

RAEplan(task τ): Candidates = {applicable method instances} nondeterministically choose m ∈ Candidates create refinement stack σ for τ and m loop while Progress(σ) ≠ failure if σ is completed then return m return failure

RAEplan

Simulate RAE on a single task

Progress(σ): start simulation

f a

SLIDE 44

44 Nau – Lecture slides for Automated Planning and Acting

is m’s current step a command? command status? return success running failed choose a candidate m′ push (τʹ, m′,…) onto stack no more steps in m? pop(σ) no candidates for τʹ? yes yes τʹ ← next step of m type(τʹ) task command send a to the execution platform assignment update state retry τ using an untried candidate yes retry τ using an untried candidate suc ceeded started m? yes no no

Progress(σ): start simulation

f a

RAEplan(task τ): Candidates = {applicable method instances} nondeterministically choose m ∈ Candidates create refinement stack σ for τ and m loop while Progress(σ) ≠ failure if σ is completed then return m return failure

RAEplan

Simulate RAE on a single task

Which candidate to choose next? How to deal with uncertainty about

utcome?

SLIDE 45

45 Nau – Lecture slides for Automated Planning and Acting

is m’s current step a command? command status? return success running failed choose a candidate m′ push (τʹ, m′,…) onto stack no more steps in m? pop(σ) no candidates for τʹ? yes yes τʹ ← next step of m type(τʹ) task command send a to the execution platform assignment update state retry τ using an untried candidate yes retry τ using an untried candidate suc ceeded started m? yes no no

Progress(σ): start simulation

f a

RAEplan(task τ): Candidates = {applicable method instances} nondeterministically choose m ∈ Candidates create refinement stack σ for τ and m loop while Progress(σ) ≠ failure if σ is completed then return m return failure

RAEplan

Simulate RAE on a single task

= Idea 1: depth-first backtracking Ø Backtrack through code execution

every method body, every command simulation

Ø Every possible point where there’s

more than one possible candidate
more than one possible outcome

= Backtracking is hard in SeRPE Ø Even harder here = Idea 1: depth-first backtracking

SLIDE 46

46 Nau – Lecture slides for Automated Planning and Acting

is m’s current step a command? command status? return success running failed choose a candidate m′ push (τʹ, m′,…) onto stack no more steps in m? pop(σ) no candidates for τʹ? yes yes τʹ ← next step of m type(τʹ) task command send a to the execution platform assignment update state retry τ using an untried candidate yes retry τ using an untried candidate suc ceeded started m? yes no no

Progress(σ): start simulation

f a

RAEplan(task τ): Candidates = {applicable method instances} nondeterministically choose m ∈ Candidates create refinement stack σ for τ and m loop while Progress(σ) ≠ failure if σ is completed then return m return failure

RAEplan

Simulate RAE on a single task

= Idea 2: Ø At every point where we need to choose a

candidate, try all of them in parallel

= Idea 2: multithreading or multiprocessing

Poll: is this a reasonable approach?

SLIDE 47

47 Nau – Lecture slides for Automated Planning and Acting

is m’s current step a command? command status? return success running failed choose a candidate m′ push (τʹ, m′,…) onto stack no more steps in m? pop(σ) no candidates for τʹ? yes yes τʹ ← next step of m type(τʹ) task command send a to the execution platform assignment update state retry τ using an untried candidate yes retry τ using an untried candidate suc ceeded started m? yes no no

Progress(σ): start simulation

f a

RAEplan(task τ): Candidates = {applicable method instances} nondeterministically choose m ∈ Candidates create refinement stack σ for τ and m loop while Progress(σ) ≠ failure if σ is completed then return m return failure

RAEplan

Simulate RAE on a single task

Ø Multiple runs

random choices and outcomes in each run

Ø Return the m ∈ Candidates that has the highest

expected utility

= Idea 3: Monte Carlo rollouts

Patra, Traverso, Ghallab, and Nau. Acting and planning using operational models. AAAI, 2019 Patra, Mason, Kumar, Traverso, Ghallab, and Nau. Integrating Acting, Planning, and Learning in Hierarchical Operational Models. ICAPS, 2020

SLIDE 48

48 Nau – Lecture slides for Automated Planning and Acting

Simulating an action

Simplest case:

▸ Descriptive model, e.g., probabilistic action template a(x1, …, xk) pre: … (p1) eff1: e11, e12, … probabilities p1, …, pm … ∑ pi =1 (pm) effm: em1, em2, … ▸ Choose randomly, each effi has probability pi ▸ Use effi to update the current state

More general:

▸ Arbitrary computation, e.g., physics-based simulation ▸ Run the code to get predicted effects

is m’s current step a command? command status? return success running failed choose a candidate m′ push (τʹ, m′,…) onto stack no more steps in m? pop(σ) no candidates for τʹ? yes yes τʹ ← next step of m type(τʹ) task command send a to the execution platform assignment update state retry τ using an untried candidate yes retry τ using an untried candidate suc ceeded started m? yes no no

Progress(σ): start simulation

f a

SLIDE 49

49 Nau – Lecture slides for Automated Planning and Acting

method method state state method method task method command task method

… …

code execution state state command

…

state state

…

command task task

… … …

code execution command

…

state state

…

method method

… … … … …

. . . possible

utcomes

possible choices . . . action action state state state state state state action action . . . possible choices . . . possible

utcomes

… … . . . state

Monte Carlo Rollouts

Probabilistic actions (Chapter 5):

At each state,

choose action

At each action,

random outcome In RAEplan:

= At each task Ø Choose method instance Ø Body generates tasks

and commands

= At each command Ø run simulation Ø get simulated outcome

SLIDE 50

50 Nau – Lecture slides for Automated Planning and Acting

Summary of Experimental Results

Four different domains, different combinations of characteristics
Evaluation criteria:

▸ Efficiency, successes vs failures, how many retries

Result: planning helps

▸ Rae operates better with RAEplan than without ▸ Rae operates better with more planning than with less planning

SLIDE 51

51 Nau – Lecture slides for Automated Planning and Acting

Outline

3.1 Representation

a. State variables, commands, refinement methods
b. Example

3.2 Acting

a. Rae (Refinement Acting Engine)
b. Example
c. Extensions

3.3 Planning

a. Motivation and basic ideas
b. Deterministic action models
c. SeRPE (Sequential Refinement Planning Engine)

3.4 Using Planning in Acting

a. Techniques
b. Caveats

SLIDE 52

52 Nau – Lecture slides for Automated Planning and Acting

3.4 Acting and Refinement Planning

Hierarchical acting with refinement planning

▸ RAE can call a planner each time it wants to choose a method

Non-hierarchical actor with refinement planning
Refine-Lookahead
Refine-Lazy-Lookahead
Refine-Concurrent-Lookahead

▸ Nearly the same as

Run-Lookahead
Run-Lazy-Lookahead
Run-Concurrent-Lookahead

▸ But instead of a classical planner, uses a refinement planner, SeRPE-Lookahead

Needs to return success, failure, or a plan

Or a modified version of RAEplan that returns a plan instead of a method instance

SLIDE 53

53 Nau – Lecture slides for Automated Planning and Acting

Run-Lookahead(Σ, g) s ← abstraction of observed state ξ while s ⊭ g do π ← Lookahead(Σ, s, g) if π = failure then return failure a ← pop-first-action(π); perform(a) s ← abstraction of observed state ξ Refine-Lookahead(M,A,τ) s ← abstraction of observed state ξ loop π ← SeRPE-Lookahead(M,A,s,τ) if π = success or π = failure then return π a ← pop-first-action(π); perform(a) s ← abstraction of observed state ξ

Refine-Lookahead

Planning stage Acting stage

SLIDE 54

54 Nau – Lecture slides for Automated Planning and Acting

Run-Lazy-Lookahead(Σ, g) s ← abstraction of observed state ξ while s ⊭ g do π ← Lookahead(Σ, s, g) if π = failure then return failure until π = ⟨ ⟩ or s ⊨ g or Simulate(Σ, s, g, π) = failure do a ← pop-first-action(π); perform(a) s ← abstraction of observed state ξ

Refine-Lazy-Lookahead

Refine-Lazy-Lookahead(M,A,τ) s ← abstraction of observed state ξ loop π ← SeRPE-Lookahead(M,A,s,τ) if π = success or π = failure then return π until π = ⟨ ⟩ or Simulate(Σ, s, τ, π) = failure do a ← pop-first-action(π); perform(a) s ← abstraction of observed state ξ

Planning Stage Acting Stage

SLIDE 55

55 Nau – Lecture slides for Automated Planning and Acting

basic idea of Run-Concurrent-Lookahead: ▸ global s, π ▸ in one thread, this loop: ▸ s ← abstraction of observed state ▸ π ← Lookahead(Σ,s,g) ▸ in another thread, this loop: ▸ a ← pop-first-element(π) ▸ perform a ▸ return if observed state ⊨ g

Questions about implementation details

Refine-Concurrent-Lookahead

Refine-Concurrent-Lookahead(M,A,τ) ▸ same idea as Run-Concurrent-Lookahead ▸ return if π = success or π = failure Similar questions about implementation details

Planning stage Acting stage

SLIDE 56

56 Nau – Lecture slides for Automated Planning and Acting

m ← RAEplan-Lookahead(Candidates, τ, s) if m = failure then output(“failed to address” τ) else Agenda ← Agenda ∪ {⟨(τ, m, nil, ∅)⟩}

RAE and RAEplan

m ← RAEplan-Lookahead(Candidates, τ, ξ) if m = failure then output(“failed to address” τ) else m′ ← RAEplan-Lookahead(Candidates, τ, s) if m′ ≠ failure then

SLIDE 57

57 Nau – Lecture slides for Automated Planning and Acting

How to do Lookahead

Receding horizon

▸ Cut off search before accomplishing τ

e.g., depth dmax or when we run out of time

▸ At leaf nodes, use heuristic function

Subgoaling

▸ Instead of planning for g, plan for a subgoal g′ ▸ When actor is finished with gi, call planner on next subgoal gi+1

Sampling

▸ RAEplan’s Monte Carlo rollouts already do this

Planning Acting

SLIDE 58

58 Nau – Lecture slides for Automated Planning and Acting

Example

Killzone 2

▸ video game

SHOP-style HTN methods

▸ Method body is a sequence of tasks and actions

Used to plan enemy actions at the squad level

▸ Don’t want the best possible plan ▸ Need believable actions, very quickly

Use subgoaling, e.g., “get to shelter”

▸ Solution plan is maybe 4–6 actions long ▸ Planning is very shallow

≈ just pick an applicable method
Perhaps more like RAE than RAEplan?
Replan several times per second as the world changes

SLIDE 59

59 Nau – Lecture slides for Automated Planning and Acting

Caveats

Start in state s0, want to accomplish task τ

▸ Refinement method m:

task: τ
pre: s0
body: a1, a2, a3
Actor uses Refine-Lookahead

▸ Lookahead returns π = ⟨a1, a2, a3⟩ ▸ Actor performs a1, calls Lookahead again ▸ No applicable method for τ in s1, Lookahead returns failure

Fixes

(1) When writing refinement methods, make them general enough to work in different states (2) In some cases Lookahead might be able to fall back on classical planning until it finds something that matches a method (3) Keep snapshot of the search tree at s1, resume next time it’s called

m a1 a2 s0 s1 a3 s2

SLIDE 60

60 Nau – Lecture slides for Automated Planning and Acting

m a1 a2 s0 s1 a3 s2

Caveats

Start in state s0, want to accomplish task τ

▸ Refinement method m:

task: τ
pre: s0
body: a1, a2, a3
Actor uses Refine-Lazy-Lookahead

▸ Lookahead returns π = ⟨a1, a2, a3⟩ ▸ Unexpected event during a1, outcome isn’t s1; actor calls Lookahead again ▸ No applicable method for τ in s4, Lookahead returns failure

Can use (1), (2) on the previous slide, and (3) with this modification:

▸ Keep snapshot of Lookahead’s search tree after each action

Restart it immediately after a1, using s4 as current state
Also: make recovery methods for unexpected states
e.g., fix flat tire, get back on the road

a1 s0 s4

Poll: will this also work for Refine-Lookahead?

SLIDE 61

61 Nau – Lecture slides for Automated Planning and Acting

Summary

3.1 Operational models

▸ ξ versus s, tasks, events, ▸ Commands to the execution platform ▸ Extensions to state-variable representation ▸ Refinement method: name, task/event, preconditions, body ▸ Examples: fetch container, emergency, opening a door

3.2 Refinement Acting Engine (RAE)

▸ Purely reactive: select a method and apply it ▸ Rae: input stream, Candidates, Instances, Agenda, refinement stacks ▸ Progress: command status, nextstep, type of step ▸ Retry: Candidates \ tried, comparison to backtracking ▸ Refinement trees ▸ Concurrent tasks: for each, a refinement stack ▸ Goal: achieve(condition), uses monitoring ▸ Controlling progress, heuristics

SLIDE 62

62 Nau – Lecture slides for Automated Planning and Acting

Summary

3.3 Refinement planning

▸ plan by simulating RAE on a single external task/event/goal ▸ SeRPE uses deterministic actions ▸ RAEplan simulates the actor’s commands, does Monte Carlo rollouts

3.4 Acting and planning

▸ Refine-Lookahead, Refine-Lazy-Lookahead, Refine-Concurrent-Lookahead

Like Run-Lookahead, Run-Lazy-Lookahead, Run-Concurrent-Lookahead

▸ Caveats

Current state may not be what we expect
Possible ways to handle that