[PPT] - Angelic Hierarchical Planning: Optimal and Online Algorithms PowerPoint Presentation

SLIDE 1

ICAPS ‘08

Angelic Hierarchical Planning:

Optimal and Online Algorithms

Bhaskara Marthi

MIT/Willow Garage

bhaskara@csail.mit.edu

Stuart Russell

UC Berkeley

russell@cs.berkeley.edu

1

Jason Wolfe

UC Berkeley

jawolfe@cs.berkeley.edu

SLIDE 2

High-Level Actions (HLAs)

Here, a high-level action (HLA) =

a set of allowed immediate refinements:

each is a sequence of actions
may have associated preconditions
Almost all actions we think about are high-level
Plan a trip
Vacuum the house
Go to work

[Go(work)] [GetIn(car), Drive(work), GetOut(car)] [Walk(stop), Bus(work)]

2

if ¬Raining

SLIDE 3

Abstract Lookahead

k-step lookahead >> 1-step lookahead
e.g., chess

Kc3 Rxa1 Qa4 c2 c2 Kxa1

3

SLIDE 4

Abstract Lookahead

k-step lookahead >> 1-step lookahead
e.g., chess

Kc3 Rxa1 Qa4 c2 c2 Kxa1

3

Perth Darwin Melbourne Sydney

k-step lookahead no use if steps too small
e.g., first k turns in TSP of Australia

SLIDE 5

Abstract Lookahead

k-step lookahead >> 1-step lookahead
e.g., chess

3

L R L R L R

k-step lookahead no use if steps too small
e.g., first k turns in TSP of Australia

SLIDE 6

Abstract Lookahead

k-step lookahead >> 1-step lookahead
e.g., chess

3

L R L R L R

k-step lookahead no use if steps too small
e.g., first k turns in TSP of Australia
this is one small part of a human life,

≈ 20,000,000,000,000 primitive actions

SLIDE 7

Abstract Lookahead

k-step lookahead >> 1-step lookahead
e.g., chess
Abstract plans with HLAs are shorter

3

L R L R L R

k-step lookahead no use if steps too small
e.g., first k turns in TSP of Australia
this is one small part of a human life,

≈ 20,000,000,000,000 primitive actions

SLIDE 8

Abstract Lookahead

k-step lookahead >> 1-step lookahead
e.g., chess
Abstract plans with HLAs are shorter
Much shorter plans => exponential savings

3

is provably optimal

L R L R L R

k-step lookahead no use if steps too small
e.g., first k turns in TSP of Australia
this is one small part of a human life,

≈ 20,000,000,000,000 primitive actions

SLIDE 9

Abstract Lookahead

k-step lookahead >> 1-step lookahead
e.g., chess
Abstract plans with HLAs are shorter
Much shorter plans => exponential savings
Can look ahead much further

3

looks like a good start

L R L R L R

Melbourne

k-step lookahead no use if steps too small
e.g., first k turns in TSP of Australia
this is one small part of a human life,

≈ 20,000,000,000,000 primitive actions

SLIDE 10

Abstract Lookahead

k-step lookahead >> 1-step lookahead
e.g., chess
Abstract plans with HLAs are shorter
Much shorter plans => exponential savings
Can look ahead much further
Requires models for HLAs
i.e., transition and cost fns

3

looks like a good start

L R L R L R

Melbourne

k-step lookahead no use if steps too small
e.g., first k turns in TSP of Australia
this is one small part of a human life,

≈ 20,000,000,000,000 primitive actions

SLIDE 11

Abstract Lookahead

k-step lookahead >> 1-step lookahead
e.g., chess
Abstract plans with HLAs are shorter
Much shorter plans => exponential savings
Can look ahead much further
Requires models for HLAs
i.e., transition and cost fns
No suitable models in literature

3

looks like a good start

L R L R L R

Melbourne

k-step lookahead no use if steps too small
e.g., first k turns in TSP of Australia
this is one small part of a human life,

≈ 20,000,000,000,000 primitive actions

SLIDE 12

Abstract Lookahead

k-step lookahead >> 1-step lookahead
e.g., chess
Abstract plans with HLAs are shorter
Much shorter plans => exponential savings
Can look ahead much further
Requires models for HLAs
i.e., transition and cost fns
No suitable models in literature
We extend our angelic semantics

3

looks like a good start

L R L R L R

Melbourne

k-step lookahead no use if steps too small
e.g., first k turns in TSP of Australia
this is one small part of a human life,

≈ 20,000,000,000,000 primitive actions

SLIDE 13

s0

Angelic Semantics for HLAs [MRW ’07]

Models HLAs in deterministic domains

State space

4

SLIDE 14

s0

h1

Angelic Semantics for HLAs [MRW ’07]

Models HLAs in deterministic domains
Central idea is reachable set of an HLA from some state

State space

4

SLIDE 15

s0

h1

Angelic Semantics for HLAs [MRW ’07]

Models HLAs in deterministic domains
Central idea is reachable set of an HLA from some state
When extended to sequences of actions, ...

State space

4

SLIDE 16

s0

h2 h1

Angelic Semantics for HLAs [MRW ’07]

Models HLAs in deterministic domains
Central idea is reachable set of an HLA from some state
When extended to sequences of actions, ...

State space

4

SLIDE 17

s0

h2 h1 h2

Angelic Semantics for HLAs [MRW ’07]

Models HLAs in deterministic domains
Central idea is reachable set of an HLA from some state
When extended to sequences of actions, ...

State space

4

SLIDE 18

s0

h2 h1

Angelic Semantics for HLAs [MRW ’07]

Models HLAs in deterministic domains
Central idea is reachable set of an HLA from some state
When extended to sequences of actions, ...

State space

4

SLIDE 19

s0

[h1, h2]

Angelic Semantics for HLAs [MRW ’07]

Models HLAs in deterministic domains
Central idea is reachable set of an HLA from some state
When extended to sequences of actions, ...

State space

4

SLIDE 20

s0

[h1, h2]

Angelic Semantics for HLAs [MRW ’07]

Models HLAs in deterministic domains
Central idea is reachable set of an HLA from some state
When extended to sequences of actions, ...
… allows proving that a plan can or cannot possibly reach the goal

State space

4

SLIDE 21

s0

[h1, h2]

Angelic Semantics for HLAs [MRW ’07]

Models HLAs in deterministic domains
Central idea is reachable set of an HLA from some state
When extended to sequences of actions, ...
… allows proving that a plan can or cannot possibly reach the goal

State space

4

G

SLIDE 22

s0

[h1, h2] [h1, h2] is a solution

Angelic Semantics for HLAs [MRW ’07]

Models HLAs in deterministic domains
Central idea is reachable set of an HLA from some state
When extended to sequences of actions, ...
… allows proving that a plan can or cannot possibly reach the goal

State space

4

G

SLIDE 23

s0

[h1, h2] [h1, h2] is a solution

Angelic Semantics for HLAs [MRW ’07]

Models HLAs in deterministic domains
Central idea is reachable set of an HLA from some state
When extended to sequences of actions, ...
… allows proving that a plan can or cannot possibly reach the goal
May seem related to nondeterminism ...

State space

4

G

SLIDE 24

s0

[h1, h2] [h1, h2] is a solution

Angelic Semantics for HLAs [MRW ’07]

Models HLAs in deterministic domains
Central idea is reachable set of an HLA from some state
When extended to sequences of actions, ...
… allows proving that a plan can or cannot possibly reach the goal
May seem related to nondeterminism ...
but uncertainty is angelic: resolved by the agent, not an adversary

State space

4

G

SLIDE 25

s0

[h1, h2] [h1, h2] is a solution

Angelic Semantics for HLAs [MRW ’07]

Models HLAs in deterministic domains
Central idea is reachable set of an HLA from some state
When extended to sequences of actions, ...
… allows proving that a plan can or cannot possibly reach the goal
May seem related to nondeterminism ...
but uncertainty is angelic: resolved by the agent, not an adversary

[a4, a1, a3, a2]

State space

4

G

SLIDE 26

Angelic Semantics cont.

Approximate descriptions provide lower & upper bounds
n reachable sets
Descriptions are true: follow logically from hierarchy

5

SLIDE 27

Angelic Semantics cont.

Approximate descriptions provide lower & upper bounds
n reachable sets
Descriptions are true: follow logically from hierarchy
Sound & complete planning algorithm uses descriptions to
Commit to provably successful abstract plans:

Downward Refinement Property (DRP) automatically satisfied

potentially exponential speedup
Prune provably unsuccessful abstract plans (USP satisfied)

5

SLIDE 28

Contributions

Extend angelic semantics with action costs
Developed novel algorithms that do lookahead with HLAs
Angelic Hierarchical A* (AHA*)
Angelic Hierarchical Learning Real-Time A* (AHLRTA*)
Both require three inputs:
planning problem
action hierarchy (set of HLAs)
approximate models for HLAs

6

Melbourne

SLIDE 29

Deterministic Planning Problems

Here, a planning problem =

7

SLIDE 30

S

Deterministic Planning Problems

Here, a planning problem =
State space S

7

SLIDE 31

S

s0 G

Deterministic Planning Problems

Here, a planning problem =
State space S
Initial state s0, terminal set G

7

SLIDE 32

S

s0 G

Deterministic Planning Problems

Here, a planning problem =
State space S
Initial state s0, terminal set G
Primitive action set

7

SLIDE 33

S

s0 G

Deterministic Planning Problems

Here, a planning problem =
State space S
Initial state s0, terminal set G
Primitive action set
Transition function: S × A → S
Cost function : S × A → R ∪ {∞}

7

Transitions & costs for action a1

5

4 ∞ 8 3 ∞

SLIDE 34

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row

8

T1 T2 T3 T4

A B C

SLIDE 35

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1

8

T1 T2 T3 T4

A B C

SLIDE 36

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

T1 T2 T3 T4

A B C

SLIDE 37

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

T1 T2 T3 T4

A B C

SLIDE 38

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

T1 T2 T3 T4

A B C

SLIDE 39

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

T1 T2 T3 T4

A B C

SLIDE 40

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

T1 T2 T3 T4

A B C

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

SLIDE 41

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

T1 T2 T3 T4

A B C

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

SLIDE 42

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

T1 T2 T3 T4

A B C

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

SLIDE 43

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

T1 T2 T3 T4

A B C

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

SLIDE 44

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

T1 T2 T3 T4

A B C

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

SLIDE 45

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

T1 T2 T3 T4

A B C

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

SLIDE 46

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

T1 T2 T3 T4

A B C

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

SLIDE 47

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

T1 T2 T3 T4

A B C

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

SLIDE 48

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

T1 T2 T3 T4

A B C

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

SLIDE 49

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

T1 T2 T3 T4

A B C

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

SLIDE 50

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

T1 T2 T3 T4

A B C

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

SLIDE 51

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

T1 T2 T3 T4

A B C

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

SLIDE 52

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

T1 T2 T3 T4

A B C

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

SLIDE 53

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

T1 T2 T3 T4

A B C

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

SLIDE 54

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

T1 T2 T3 T4

A B C

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

SLIDE 55

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

T1 T2 T3 T4

A B C

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

SLIDE 56

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

T1 T2 T3 T4

A B C

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

SLIDE 57

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

T1 T2 T3 T4

A B C

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

SLIDE 58

Running Example: Warehouse World Domain

Elaborated Blocks World with

discrete spatial constraints

Gripper must stay in bounds
Can’t pass through blocks
Can only turn at top row
All actions have cost 1
Goal: have C on T4
Can’t just move directly
Final plan has 22 steps

8

T1 T2 T3 T4

A B C

L, D, GetR, U, Turn, D, PutL, R, R, D, GetL, L, PutL, U, L, GetL, U, Turn, R, D, D, PutR

SLIDE 59

Running Example: Warehouse World HLAs

9

L D GetR U Turn D PutL

SLIDE 60

Running Example: Warehouse World HLAs

9

L D GetR U Turn D PutL

Nav(2,3) Nav(3,3) Nav(2,3)

SLIDE 61

Running Example: Warehouse World HLAs

9

L D GetR U Turn D PutL

Nav(2,3) Nav(3,3) Nav(2,3)

NavT(2,3) NavT(2,3)

SLIDE 62

Running Example: Warehouse World HLAs

9

L D GetR U Turn D PutL

Nav(2,3) Nav(3,3) Nav(2,3)

NavT(2,3) NavT(2,3)

Move(C,A)

SLIDE 63

Running Example: Warehouse World HLAs

9

L D GetR U Turn D PutL

Nav(2,3) Nav(3,3) Nav(2,3)

NavT(2,3) NavT(2,3)

Move(C,A)

Act

...

SLIDE 64

Plans of interest are primitive

refinements of special HLA Act

[Act]

10

Running Example: Warehouse World HLAs

SLIDE 65

Plans of interest are primitive

refinements of special HLA Act

Each HLA has a set of immediate

refinements into action sequences

[Act]

10

Running Example: Warehouse World HLAs

[Move(B,C), Act]

...

iff at G

[ ]

SLIDE 66

Plans of interest are primitive

refinements of special HLA Act

Each HLA has a set of immediate

refinements into action sequences

[Act]

10

Running Example: Warehouse World HLAs

[Move(B,C), Act]

...

iff at G

[ ] [NavT(left of B), GetR, NavT(left of target), PutR] ...

...

SLIDE 67

Plans of interest are primitive

refinements of special HLA Act

Each HLA has a set of immediate

refinements into action sequences

[Act]

10

Running Example: Warehouse World HLAs

... ...

[Move(B,C), Act]

...

iff at G

[ ] [NavT(left of B), GetR, NavT(left of target), PutR] ...

...

SLIDE 68

Abstract Lookahead Trees (ALTs)

ALTs generalize lookahead trees for flat algs (e.g., A*)

11

SLIDE 69

Abstract Lookahead Trees (ALTs)

ALTs generalize lookahead trees for flat algs (e.g., A*)
Represent a set of potential plans

11

Act

SLIDE 70

Abstract Lookahead Trees (ALTs)

ALTs generalize lookahead trees for flat algs (e.g., A*)
Represent a set of potential plans
Basic operation: refine a plan (replace with all refs. at some HLA)

11

Act

SLIDE 71

Abstract Lookahead Trees (ALTs)

ALTs generalize lookahead trees for flat algs (e.g., A*)
Represent a set of potential plans
Basic operation: refine a plan (replace with all refs. at some HLA)

11

Act

SLIDE 72

Abstract Lookahead Trees (ALTs)

ALTs generalize lookahead trees for flat algs (e.g., A*)
Represent a set of potential plans
Basic operation: refine a plan (replace with all refs. at some HLA)

11

Act Move(C,A) Act

SLIDE 73

Abstract Lookahead Trees (ALTs)

ALTs generalize lookahead trees for flat algs (e.g., A*)
Represent a set of potential plans
Basic operation: refine a plan (replace with all refs. at some HLA)

11

Act Move(C,A) Act Act Move(A,C)

SLIDE 74

Abstract Lookahead Trees (ALTs)

ALTs generalize lookahead trees for flat algs (e.g., A*)
Represent a set of potential plans
Basic operation: refine a plan (replace with all refs. at some HLA)

11

Act Move(C,A) Act Act Move(A,C)

SLIDE 75

Abstract Lookahead Trees (ALTs)

ALTs generalize lookahead trees for flat algs (e.g., A*)
Represent a set of potential plans
Basic operation: refine a plan (replace with all refs. at some HLA)

11

Act Move(C,A) Act Act Move(A,C)

SLIDE 76

Abstract Lookahead Trees (ALTs)

ALTs generalize lookahead trees for flat algs (e.g., A*)
Represent a set of potential plans
Basic operation: refine a plan (replace with all refs. at some HLA)

11

Act Move(C,A) Act Act Move(A,C) Move(B,C) Act

SLIDE 77

Abstract Lookahead Trees (ALTs)

ALTs generalize lookahead trees for flat algs (e.g., A*)
Represent a set of potential plans
Basic operation: refine a plan (replace with all refs. at some HLA)

11

Act Move(C,A) Act Act Move(A,C) Move(B,C) Act Act Move(C,B)

SLIDE 78

Abstract Lookahead Trees (ALTs)

ALTs generalize lookahead trees for flat algs (e.g., A*)
Represent a set of potential plans
Basic operation: refine a plan (replace with all refs. at some HLA)

11

Act Move(C,A) Act Act Move(A,C) Move(B,C) Act Act Move(C,B)

SLIDE 79

Abstract Lookahead Trees (ALTs)

ALTs generalize lookahead trees for flat algs (e.g., A*)
Represent a set of potential plans
Basic operation: refine a plan (replace with all refs. at some HLA)

11

Act Move(C,A) Act Act Move(A,C) Move(B,C) Act Act Move(C,B)

SLIDE 80

Abstract Lookahead Trees (ALTs)

ALTs generalize lookahead trees for flat algs (e.g., A*)
Represent a set of potential plans
Basic operation: refine a plan (replace with all refs. at some HLA)

11

Act Move(C,A) Act Act Move(A,C) Move(B,C) Act Act Move(C,B) Nav(xC-1,yC) . . . Act M

v

e ( B , C )

SLIDE 81

Abstract Lookahead Trees (ALTs)

ALTs generalize lookahead trees for flat algs (e.g., A*)
Represent a set of potential plans
Basic operation: refine a plan (replace with all refs. at some HLA)
Nodes have optimistic & pessimistic valuations

11

Act Move(C,A) Act Act Move(A,C) Move(B,C) Act Act Move(C,B) Nav(xC-1,yC) . . . Act M

v

e ( B , C )

8/∞ 0/0 2/4 4/8 5/7 7/9 9/∞ 3/∞ 3/3 1/1 6/6 8/8

SLIDE 82

Modeling HLAs

An HLA is fully characterized by planning problem + hierarchy

12

NavT(0,1)

SLIDE 83

Modeling HLAs

An HLA is fully characterized by planning problem + hierarchy
But without abstraction, lose benefits of hierarchy

12

NavT(0,1)

6 8 7 5 10 ... ...

SLIDE 84

Modeling HLAs

An HLA is fully characterized by planning problem + hierarchy
But without abstraction, lose benefits of hierarchy
Extension of idea from “Angelic Semantics for HLAs” [MRW ‘07]:
Valuation of HLA h from state s:
For each s’, min cost of any primitive refinement of h that takes s to s’

12

NavT(0,1)

6

5
6

5

SLIDE 85

Modeling HLAs

An HLA is fully characterized by planning problem + hierarchy
But without abstraction, lose benefits of hierarchy
Extension of idea from “Angelic Semantics for HLAs” [MRW ‘07]:
Valuation of HLA h from state s:
For each s’, min cost of any primitive refinement of h that takes s to s’
Exact description of h = valuation of h from each s

12

NavT(0,1)

6

5
6
5
6
5
...

SLIDE 86

Modeling HLAs

An HLA is fully characterized by planning problem + hierarchy
But without abstraction, lose benefits of hierarchy
Extension of idea from “Angelic Semantics for HLAs” [MRW ‘07]:
Valuation of HLA h from state s:
For each s’, min cost of any primitive refinement of h that takes s to s’
Exact description of h = valuation of h from each s
But this description has no compact, efficient representation in general

12

NavT(0,1)

6

5
6
5
6
5
...

SLIDE 87

6 4 3 1 5

Optimistic and Pessimistic Valuations
Instead, use approximate valuations

Exact

13

SLIDE 88

6 4 3 1 5

Optimistic and Pessimistic Valuations
Instead, use approximate valuations
We choose a simple form: reachable set + cost bound on set

Exact

13

SLIDE 89

6 4 3 1 5

Optimistic and Pessimistic Valuations
Instead, use approximate valuations
We choose a simple form: reachable set + cost bound on set
Optimistic valuations never overestimate best achievable cost

Exact

13

Optimistic

1

SLIDE 90

6 4 3 1 5

Optimistic and Pessimistic Valuations
Instead, use approximate valuations
We choose a simple form: reachable set + cost bound on set
Optimistic valuations never overestimate best achievable cost
Pessimistic valuations never underestimate best achievable cost

Exact

13

Optimistic

1

Pessimistic

4 /

SLIDE 91

Representing Descriptions: NCSTRIPS

14

SLIDE 92

Representing Descriptions: NCSTRIPS

Descriptions specify propositions (possibly) added/deleted by HLA

14

SLIDE 93

Representing Descriptions: NCSTRIPS

Descriptions specify propositions (possibly) added/deleted by HLA

14

NavT(xt,yt) (Pre: At(xs,ys))

SLIDE 94

Representing Descriptions: NCSTRIPS

Descriptions specify propositions (possibly) added/deleted by HLA
Also include a cost bound

14

NavT(xt,yt) (Pre: At(xs,ys)) Opt: -At(xs,ys), +At(xt,yt), -FaceR, +FaceR cost ≥ |xs - xt| + |ys- yt|

s t

~ ~

SLIDE 95

Representing Descriptions: NCSTRIPS

Descriptions specify propositions (possibly) added/deleted by HLA
Also include a cost bound
Can condition on features of initial state

14

NavT(xt,yt) (Pre: At(xs,ys)) Opt: -At(xs,ys), +At(xt,yt), -FaceR, +FaceR cost ≥ |xs - xt| + |ys- yt| Pess: IF Free(xt,yt) ∧ ∀x Free(x,ymax) :

At(xs,ys), +At(xt,yt), -FaceR, +FaceR

cost ≤ |xs - xt| + 2 ymax - yt - ys + 1

s t

~ ~

s t

~ ~

SLIDE 96

Representing Descriptions: NCSTRIPS

Descriptions specify propositions (possibly) added/deleted by HLA
Also include a cost bound
Can condition on features of initial state

14

NavT(xt,yt) (Pre: At(xs,ys)) Opt: -At(xs,ys), +At(xt,yt), -FaceR, +FaceR cost ≥ |xs - xt| + |ys- yt| Pess: IF Free(xt,yt) ∧ ∀x Free(x,ymax) :

At(xs,ys), +At(xt,yt), -FaceR, +FaceR

cost ≤ |xs - xt| + 2 ymax - yt - ys + 1 ELSE: nil

s t x s t

~ ~

s t

~ ~

SLIDE 97

Representing Descriptions: NCSTRIPS

Descriptions specify propositions (possibly) added/deleted by HLA
Also include a cost bound
Can condition on features of initial state
An simple algorithm progresses a valuation (DNF + #)

through an NCSTRIPS description to produce next valuation

14

NavT(xt,yt) (Pre: At(xs,ys)) Opt: -At(xs,ys), +At(xt,yt), -FaceR, +FaceR cost ≥ |xs - xt| + |ys- yt| Pess: IF Free(xt,yt) ∧ ∀x Free(x,ymax) :

At(xs,ys), +At(xt,yt), -FaceR, +FaceR

cost ≤ |xs - xt| + 2 ymax - yt - ys + 1 ELSE: nil

s t x s t

~ ~

s t

~ ~

SLIDE 98

Angelic Hierarchical A* (AHA*)

Construct an ALT with the single plan [Act]
Loop
Select a plan with minimal optimistic cost to G
If primitive, return it
Otherwise, refine one of its HLAs
Prune dominated refinements

15

SLIDE 99

AHA*: Intuitive Picture

s0 G Act highest-level primitive

16

SLIDE 100

AHA*: Intuitive Picture

s0 G Act highest-level primitive

17

SLIDE 101

AHA*: Intuitive Picture

s0 G Act highest-level primitive

18

SLIDE 102

AHA*: Intuitive Picture

s0 G Act highest-level primitive

19

SLIDE 103

AHA*: Intuitive Picture

G Act s0 highest-level primitive

20

SLIDE 104

AHA*: Intuitive Picture

G Act s0 highest-level primitive

21

SLIDE 105

AHA*: Intuitive Picture

G s0 highest-level primitive

22

SLIDE 106

AHA*: Intuitive Picture

G s0 highest-level primitive

23

SLIDE 107

AHA*: Intuitive Picture

G s0 highest-level primitive

23

SLIDE 108

Analysis of AHA*

AHA* is hierarchically optimal (HO)
Optimistic valuation → admissible heuristic
Pruning never rules out all HO plans
Better descriptions lead to lower runtime
optimistic → directed search
pessimistic → pruning (refine HO plans w/o backtracking)
Reduces to A* given “flat” hierarchy: Act → [Prim, Act]

24

runtimes in seconds on five warehouse world instances of increasing solution length

warehouse world Solution Length A* AHA* 7 0.9 0.6 16 10 4.7 25 40 11 37 550 30 44 > 10000 68

SLIDE 109

Online Search

Situated agents must cope with passage of time
offline planning rarely feasible
common alternative: real-time search
Korf’s Learning Real-Time A* (LRTA*):
Combines limited lookahead + learning
Always reaches goal, converges to optimal
Angelic Hierarchical LRTA* (AHLRTA*)
Performs hierarchical lookahead
Shares LRTA*’s guarantees
Reduces to LRTA* given

“flat” hierarchy

25

G s0

SLIDE 110

Online Results

10 100 1000 10000 1000 2000 3000 4000 5000 refinements per env. step

warehouse world

LRTA* AHLRTA*

26

total solution cost

avg. over 3

instances

1 AHLRTA* refinement ≈ 5 LRTA* refinements

SLIDE 111

Summary

Model-based hierarchical planning is theoretically interesting, shows promising empirical performance

27