Infinite-Horizon Proactive Dynamic DCOPs Khoi Hoang Ferdinando - - PowerPoint PPT Presentation

▶
infinite horizon proactive dynamic dcops
SMART_READER_LITE
LIVE PREVIEW

Infinite-Horizon Proactive Dynamic DCOPs Khoi Hoang Ferdinando - - PowerPoint PPT Presentation

Oct 28, 2023 101 likes •1k views

Infinite-Horizon Proactive Dynamic DCOPs Khoi Hoang Ferdinando Fioretto Ping Hou William Yeoh Roie Zivan Makoto Yokoo New Mexico State University All About Discovery! nmsu.edu New Mexico State University Outline Distributed Constraint

slide-1
SLIDE 1

New Mexico State University

All About Discovery!

nmsu.edu

Infinite-Horizon Proactive Dynamic DCOPs

New Mexico State University

Khoi Hoang Ping Hou William Yeoh Ferdinando Fioretto Roie Zivan Makoto Yokoo

slide-2
SLIDE 2

New Mexico State University

All About Discovery!

nmsu.edu

Outline

ØDistributed Constraint Optimization Problems ØDynamic DCOPs ØProactive Dynamic DCOPs ØInfinite-Horizon Proactive Dynamic DCOPs* ØOverview and Details

slide-3
SLIDE 3

New Mexico State University

All About Discovery!

nmsu.edu

Outline

ØDistributed Constraint Optimization Problems ØDynamic DCOPs ØProactive Dynamic DCOPs ØInfinite-Horizon Proactive Dynamic DCOPs* ØOverview and Details

slide-4
SLIDE 4

New Mexico State University

All About Discovery!

nmsu.edu

Outline

ØDistributed Constraint Optimization Problems ØDynamic DCOPs ØProactive Dynamic DCOPs ØInfinite-Horizon Proactive Dynamic DCOPs* ØOverview and Details

slide-5
SLIDE 5

New Mexico State University

All About Discovery!

nmsu.edu

Outline

ØDistributed Constraint Optimization Problems ØDynamic DCOPs ØProactive Dynamic DCOPs ØInfinite-Horizon Proactive Dynamic DCOPs* ØOverview and Details

slide-6
SLIDE 6

New Mexico State University

All About Discovery!

nmsu.edu

Outline

ØDistributed Constraint Optimization Problems ØDynamic DCOPs ØProactive Dynamic DCOPs ØInfinite-Horizon Proactive Dynamic DCOPs* ØOverview and Details

slide-7
SLIDE 7

New Mexico State University

All About Discovery!

nmsu.edu

A B C

Distributed Constraint Optimization Problems

[1] Modi et al., ADOPT: Asynchronous Distributed Constraint Optimization with Quality Guarantees, 2005

slide-8
SLIDE 8

New Mexico State University

All About Discovery!

nmsu.edu

xA xB fAB(xA,xB) 5 1 10 … … …

A B C

Distributed Constraint Optimization Problems

[1] Modi et al., ADOPT: Asynchronous Distributed Constraint Optimization with Quality Guarantees, 2005

fAB

slide-9
SLIDE 9

New Mexico State University

All About Discovery!

nmsu.edu

xA xB fAB(xA,xB) 5 1 10 … … …

A B C

xC xA fCA(xC,xA) 7 1 4 … … … xB xC fBC(xB,xC) 3 1 12 … … …

Distributed Constraint Optimization Problems

[1] Modi et al., ADOPT: Asynchronous Distributed Constraint Optimization with Quality Guarantees, 2005

fAB fCA fBC

slide-10
SLIDE 10

New Mexico State University

All About Discovery!

nmsu.edu

A B C

Distributed Constraint Optimization Problems

[1] Modi et al., ADOPT: Asynchronous Distributed Constraint Optimization with Quality Guarantees, 2005

Maximize fAB + fBC + fCA xA = ? xB = ? xC = ?

slide-11
SLIDE 11

New Mexico State University

All About Discovery!

nmsu.edu

Distributed Constraint Optimization Problems

  • Meeting scheduling problems
  • Smart devices scheduling
  • Resource allocation
  • Sensor network
slide-12
SLIDE 12

New Mexico State University

All About Discovery!

nmsu.edu

Limitations

  • DCOPs

– Static problem – Not consider possible changes

slide-13
SLIDE 13

New Mexico State University

All About Discovery!

nmsu.edu

Limitations

  • DCOPs

– Static problem – Not consider possible changes

  • Dynamic DCOPs

– Reacting to changes of the problem

P0

[2] R. Lass et al., Dynamic distributed constraint reasoning, 2005 [3] A. Petcu and B. Faltings, Superstabilizing, fault-containing multiagent combinatorial optimization, 2005

slide-14
SLIDE 14

New Mexico State University

All About Discovery!

nmsu.edu

Dynamic DCOPs

  • DCOPs

– Static problem – Not consider possible changes

  • Dynamic DCOPs

– Reacting to changes of the problem

P0

React

[2] R. Lass et al., Dynamic distributed constraint reasoning, 2005 [3] A. Petcu and B. Faltings, Superstabilizing, fault-containing multiagent combinatorial optimization, 2005

slide-15
SLIDE 15

New Mexico State University

All About Discovery!

nmsu.edu

Dynamic DCOPs

  • DCOPs

– Static problem – Not consider possible changes

  • Dynamic DCOPs

– Reacting to changes of the problem

P0 X0

[2] R. Lass et al., Dynamic distributed constraint reasoning, 2005 [3] A. Petcu and B. Faltings, Superstabilizing, fault-containing multiagent combinatorial optimization, 2005

slide-16
SLIDE 16

New Mexico State University

All About Discovery!

nmsu.edu

Dynamic DCOPs

  • DCOPs

– Static problem – Not consider possible changes

  • Dynamic DCOPs

– Reacting to changes of the problem

P0 P1 X0

[2] R. Lass et al., Dynamic distributed constraint reasoning, 2005 [3] A. Petcu and B. Faltings, Superstabilizing, fault-containing multiagent combinatorial optimization, 2005

slide-17
SLIDE 17

New Mexico State University

All About Discovery!

nmsu.edu

Dynamic DCOPs

  • DCOPs

– Static problem – Not consider possible changes

  • Dynamic DCOPs

– Reacting to changes of the problem

P0 P1 X0

React

[2] R. Lass et al., Dynamic distributed constraint reasoning, 2005 [3] A. Petcu and B. Faltings, Superstabilizing, fault-containing multiagent combinatorial optimization, 2005

slide-18
SLIDE 18

New Mexico State University

All About Discovery!

nmsu.edu

Dynamic DCOPs

  • DCOPs

– Static problem – Not consider possible changes

  • Dynamic DCOPs

– Reacting to changes of the problem

P0 P1 X0 X1

[2] R. Lass et al., Dynamic distributed constraint reasoning, 2005 [3] A. Petcu and B. Faltings, Superstabilizing, fault-containing multiagent combinatorial optimization, 2005

slide-19
SLIDE 19

New Mexico State University

All About Discovery!

nmsu.edu

Dynamic DCOPs

  • DCOPs

– Static problem – Not consider possible changes

  • Dynamic DCOPs

– Reacting to changes of the problem

P0 P1 X0 X1 P2

[2] R. Lass et al., Dynamic distributed constraint reasoning, 2005 [3] A. Petcu and B. Faltings, Superstabilizing, fault-containing multiagent combinatorial optimization, 2005

slide-20
SLIDE 20

New Mexico State University

All About Discovery!

nmsu.edu

Dynamic DCOPs

  • DCOPs

– Static problem – Not consider possible changes

  • Dynamic DCOPs

– Reacting to changes of the problem

P0 P1 X0 X1 P2

React

[2] R. Lass et al., Dynamic distributed constraint reasoning, 2005 [3] A. Petcu and B. Faltings, Superstabilizing, fault-containing multiagent combinatorial optimization, 2005

slide-21
SLIDE 21

New Mexico State University

All About Discovery!

nmsu.edu

Dynamic DCOPs

  • DCOPs

– Static problem – Not consider possible changes

  • Dynamic DCOPs

– Reacting to changes of the problem

P0 P1 X0 X2 P2 X1

[2] R. Lass et al., Dynamic distributed constraint reasoning, 2005 [3] A. Petcu and B. Faltings, Superstabilizing, fault-containing multiagent combinatorial optimization, 2005

slide-22
SLIDE 22

New Mexico State University

All About Discovery!

nmsu.edu

Limitations

  • Dynamic DCOPs

– Not take advantage of possible changes – Good for current, bad for future (myopic solutions)

[2] R. Lass et al., Dynamic distributed constraint reasoning, 2005 [3] A. Petcu and B. Faltings, Superstabilizing, fault-containing multiagent combinatorial optimization, 2005

slide-23
SLIDE 23

New Mexico State University

All About Discovery!

nmsu.edu

Limitations

  • Dynamic DCOPs

– Not take advantage of possible changes – Good for current, bad for future (myopic solutions)

  • How about if we know

– How often the problems change – Knowledge about possible changes

slide-24
SLIDE 24

New Mexico State University

All About Discovery!

nmsu.edu

Proactive Dynamic DCOPs

  • Knowledge about changes of random events

– Initial distribution and transition function

  • Solve all the problems beforehand up to

horizon h

  • Keep the solution at time step h

[4] Hoang et al., Proactive Dynamic Distributed Constraint Optimization, 2016

P0 P1 Ph

slide-25
SLIDE 25

New Mexico State University

All About Discovery!

nmsu.edu

Proactive Dynamic DCOPs

  • Knowledge about changes of random events

– Initial distribution and transition function

  • Solve all the problems beforehand up to

horizon h

  • Keep the solution at time step h

[4] Hoang et al., Proactive Dynamic Distributed Constraint Optimization, 2016

P0 P1 Ph X0 Xh X1

Proactive

slide-26
SLIDE 26

New Mexico State University

All About Discovery!

nmsu.edu

Limitations

Is this solution optimal from h onwards???

[4] Hoang et al., Proactive Dynamic Distributed Constraint Optimization, 2016

P0 P1 Ph X0 Xh X1

Proactive

slide-27
SLIDE 27

New Mexico State University

All About Discovery!

nmsu.edu

Key contributions

  • Infinite-Horizon Proactive Dynamic DCOPs

– Optimal solution from h onwards – Based on converged distribution at h* – Proactive vs. Reactive dynamic DCOP algorithms (first time!!!)

P0 P1 Ph Ph*

Infinite-Horizon Proactive

slide-28
SLIDE 28

New Mexico State University

All About Discovery!

nmsu.edu

Key contributions

  • Infinite-Horizon Proactive Dynamic DCOPs

– Optimal solution from h onwards – Based on converged distribution at h* – Proactive vs. Reactive dynamic DCOP algorithms (first time!!!)

P0 P1 Ph Xh* Ph*

Infinite-Horizon Proactive

slide-29
SLIDE 29

New Mexico State University

All About Discovery!

nmsu.edu

Key contributions

  • Infinite-Horizon Proactive Dynamic DCOPs

– Optimal solution from h onwards – Based on converged distribution at h* – Proactive vs. Reactive dynamic DCOP algorithms (first time!!!)

P0 P1 Ph X0 X1 Ph* Xh*

Infinite-Horizon Proactive

Xh*

slide-30
SLIDE 30

New Mexico State University

All About Discovery!

nmsu.edu

Content

ØDistributed Constraint Optimization Problem

  • Proactive Dynamic DCOPs
  • Infinite-Horizon Proactive Dynamic DCOPs
  • Algorithms
  • Experiments
  • Conclusions
slide-31
SLIDE 31

New Mexico State University

All About Discovery!

nmsu.edu

xA xB fAB(xA,xB) 5 1 10 … … …

A B C

xC xA fCA(xC,xA) 7 1 4 … … … xB xC fBC(xB,xC) 3 1 12 … … …

Distributed Constraint Optimization Problems

fAB fCA fBC

slide-32
SLIDE 32

New Mexico State University

All About Discovery!

nmsu.edu

xA xB fAB(xA,xB) 5 1 10 … … …

A B C

xC xA fCA(xC,xA) 7 1 4 … … … xB xC fBC(xB,xC) 3 1 12 … … …

Distributed Constraint Optimization Problems

fAB fCA fBC Maximize fAB + fBC + fCA

slide-33
SLIDE 33

New Mexico State University

All About Discovery!

nmsu.edu

Distributed Meeting Scheduling Problem

[5] Maheswaran et al., Taking DCOP to the real world: efficient complete solutions for distributed event scheduling, 2004.

slide-34
SLIDE 34

New Mexico State University

All About Discovery!

nmsu.edu

Distributed Meeting Scheduling Problem

Person A Utility 8:00 2 9:00 5 … … 16:00 10

[5] Maheswaran et al., Taking DCOP to the real world: efficient complete solutions for distributed event scheduling, 2004.

slide-35
SLIDE 35

New Mexico State University

All About Discovery!

nmsu.edu

Distributed Meeting Scheduling Problem

Person A Utility 8:00 2 9:00 5 … … 16:00 10 Person A Person B Utility 8:00 8:00 8:00 9:00

  • infinity

… … … 16:00 16:00

[5] Maheswaran et al., Taking DCOP to the real world: efficient complete solutions for distributed event scheduling, 2004.

Maximize fA + fB + fC + fAB + fBC + fCA

slide-36
SLIDE 36

New Mexico State University

All About Discovery!

nmsu.edu

Distributed Constraint Optimization Problem (DCOP)

DCOP is a tuple <A, X, D, F>

  • A = {a1, a2,…,an}
  • X = {x1, x2,…,xm}
  • D = {D1, D2,…,Dm}
  • F = {f1, f2,…,fl}
  • F(σ) = Σ fi
  • σmax = argmax F(σ)

Person A Utility 8:00 2 9:00 5 … … 16:00 10

[1] Modi et al., ADOPT: Asynchronous Distributed Constraint Optimization with Quality Guarantees, 2005

slide-37
SLIDE 37

New Mexico State University

All About Discovery!

nmsu.edu

Distributed Constraint Optimization Problem (DCOP)

DCOP is a tuple <A, X, D, F>

  • A = {a1, a2,…,an}
  • X = {x1, x2,…,xm}
  • D = {D1, D2,…,Dm}
  • F = {f1, f2,…,fl}
  • F(σ) = Σ fi
  • σmax = argmax F(σ)

Person A Utility 8:00 2 9:00 5 … … 16:00 10

[1] Modi et al., ADOPT: Asynchronous Distributed Constraint Optimization with Quality Guarantees, 2005

slide-38
SLIDE 38

New Mexico State University

All About Discovery!

nmsu.edu

Distributed Constraint Optimization Problem (DCOP)

DCOP is a tuple <A, X, D, F>

  • A = {a1, a2,…,an}
  • X = {x1, x2,…,xm}
  • D = {D1, D2,…,Dm}
  • F = {f1, f2,…,fl}
  • F(σ) = Σ fi
  • σmax = argmax F(σ)

Person A Utility 8:00 2 9:00 5 … … 16:00 10

[1] Modi et al., ADOPT: Asynchronous Distributed Constraint Optimization with Quality Guarantees, 2005

slide-39
SLIDE 39

New Mexico State University

All About Discovery!

nmsu.edu

Distributed Constraint Optimization Problem (DCOP)

DCOP is a tuple <A, X, D, F>

  • A = {a1, a2,…,an}
  • X = {x1, x2,…,xm}
  • D = {D1, D2,…,Dm}
  • F = {f1, f2,…,fl}
  • F(σ) = Σ fi
  • σmax = argmax F(σ)

Person A Utility 8:00 2 9:00 5 … … 16:00 10

[1] Modi et al., ADOPT: Asynchronous Distributed Constraint Optimization with Quality Guarantees, 2005

slide-40
SLIDE 40

New Mexico State University

All About Discovery!

nmsu.edu

Distributed Constraint Optimization Problem (DCOP)

DCOP is a tuple <A, X, D, F>

  • A = {a1, a2,…,an}
  • X = {x1, x2,…,xm}
  • D = {D1, D2,…,Dm}
  • F = {f1, f2,…,fl}
  • F(σ) = Σ fi
  • σmax = argmax F(σ)

Person A Utility 8:00 2 9:00 5 … … 16:00 10

[1] Modi et al., ADOPT: Asynchronous Distributed Constraint Optimization with Quality Guarantees, 2005

slide-41
SLIDE 41

New Mexico State University

All About Discovery!

nmsu.edu

Content

  • Distributed Constraint Optimization Problem

ØProactive Dynamic DCOPs

  • Infinite-Horizon Proactive Dynamic DCOP
  • Algorithms
  • Experiments
  • Conclusions

New Mexico State University

slide-42
SLIDE 42

New Mexico State University

All About Discovery!

nmsu.edu

  • Random variables

– Initial distribution – Transition function

Proactive Dynamic DCOPs

Week 0 Raining 8:00 Week 1 Raining 10:00

[4] Hoang et al., Proactive Dynamic Distributed Constraint Optimization, 2016

slide-43
SLIDE 43

New Mexico State University

All About Discovery!

nmsu.edu

Proactive Dynamic DCOPs

  • Constraints with random variables
  • Proactive: solve the whole problems beforehand
  • Keep the solution at h onwards

P0 P1 Ph X0 Xh X1

Proactive

[4] Hoang et al., Proactive Dynamic Distributed Constraint Optimization, 2016

slide-44
SLIDE 44

New Mexico State University

All About Discovery!

nmsu.edu

Proactive Dynamic DCOPs

  • Y = {y1, y2,…,ym}

– Ω: event space – p0: initial distribution – T: transition function

  • c: switching cost
  • h: horizon

x = 0 x = 1 x = 0

P0 P1 Ph X0 Xh X1

Proactive

[4] Hoang et al., Proactive Dynamic Distributed Constraint Optimization, 2016

slide-45
SLIDE 45

New Mexico State University

All About Discovery!

nmsu.edu

Content

  • Distributed Constraint Optimization Problem
  • Proactive Dynamic DCOPs

ØInfinite-Horizon Proactive Dynamic DCOP

  • Algorithms
  • Experiments
  • Conclusions

New Mexico State University

slide-46
SLIDE 46

New Mexico State University

All About Discovery!

nmsu.edu

Limitations

Is this solution optimal from h onwards???

P0 P1 Ph X0 Xh X1

Proactive

slide-47
SLIDE 47

New Mexico State University

All About Discovery!

nmsu.edu

Infinite-Horizon Proactive Dynamic DCOPs

  • Each random variable => Markov chain
  • Optimal solutions from h onwards

ØMarkov chain convergence

New Mexico State University

slide-48
SLIDE 48

New Mexico State University

All About Discovery!

nmsu.edu

Infinite-Horizon Proactive Dynamic DCOPs

  • Under some specific conditions:

– Markov chains converge at h* – Solve the problem at h with converged distribution

New Mexico State University

P0 P1 Ph X0 Xh* X1 Ph* Xh*

Infinite-Horizon Proactive

slide-49
SLIDE 49

New Mexico State University

All About Discovery!

nmsu.edu

Content

  • Distributed Constraint Optimization Problem
  • Proactive Dynamic DCOPs
  • Infinite-Horizon Proactive Dynamic DCOP

ØAlgorithms

  • Experiments
  • Conclusions

New Mexico State University

slide-50
SLIDE 50

New Mexico State University

All About Discovery!

nmsu.edu

Algorithms

  • Preprocessing

– Eliminate random variables – Calculate expected utility – Regular DCOPs at every time step

  • FORWARD
  • BACKWARD

New Mexico State University

slide-51
SLIDE 51

New Mexico State University

All About Discovery!

nmsu.edu

Preprocessing (cont.)

  • Constraints with random variables
  • Regular DCOPs at every time step

x ts = k a*prob(y=0) + b*prob(y=1) 1 c*prob(y=0) + d*prob(y=1) x y a 1 b 1 c 1 1 d

New Mexico State University

slide-52
SLIDE 52

New Mexico State University

All About Discovery!

nmsu.edu

FORWARD

  • Solve problem at h with converged

distribution

  • Solve from P0 forward
  • Online – Offline algorithm

New Mexico State University

P0 P1 Ph Ph*

slide-53
SLIDE 53

New Mexico State University

All About Discovery!

nmsu.edu

FORWARD

  • Solve problem at h with converged

distribution

  • Solve from P0 forward
  • Online – Offline algorithm

New Mexico State University

P0 P1 Ph Xh* Ph*

slide-54
SLIDE 54

New Mexico State University

All About Discovery!

nmsu.edu

FORWARD

  • Solve problem at h with converged

distribution

  • Solve from P0 forward
  • Online – Offline algorithm

New Mexico State University

P0 P1 Ph Xh* Ph* Xh*

slide-55
SLIDE 55

New Mexico State University

All About Discovery!

nmsu.edu

FORWARD

  • Solve problem at h with converged

distribution

  • Solve from P0 forward
  • Online – Offline algorithm

New Mexico State University

P0 P1 Ph X0 Xh* Ph* Xh*

FORWARD

slide-56
SLIDE 56

New Mexico State University

All About Discovery!

nmsu.edu

FORWARD

  • Solve problem at h with converged

distribution

  • Solve from P0 forward
  • Online – Offline algorithm

New Mexico State University

P0 P1 Ph X0 Xh* Ph* Xh*

FORWARD

X1

slide-57
SLIDE 57

New Mexico State University

All About Discovery!

nmsu.edu

BACKWARD

  • Solve problem at h with converged

distribution

  • Solve from Ph backward
  • Offline algorithm

New Mexico State University

P0 Ph-1 Ph Ph*

slide-58
SLIDE 58

New Mexico State University

All About Discovery!

nmsu.edu

BACKWARD

  • Solve problem at h with converged

distribution

  • Solve from Ph backward
  • Offline algorithm

New Mexico State University

P0 Ph-1 Ph Ph* Xh*

slide-59
SLIDE 59

New Mexico State University

All About Discovery!

nmsu.edu

BACKWARD

  • Solve problem at h with converged

distribution

  • Solve from Ph backward
  • Offline algorithm

New Mexico State University

P0 Ph-1 Ph Ph* Xh* Xh*

slide-60
SLIDE 60

New Mexico State University

All About Discovery!

nmsu.edu

BACKWARD

  • Solve problem at h with converged

distribution

  • Solve from Ph backward
  • Offline algorithm

New Mexico State University

P0 Ph-1 Ph Ph* Xh* Xh*

BACKWARD

slide-61
SLIDE 61

New Mexico State University

All About Discovery!

nmsu.edu

BACKWARD

  • Solve problem at h with converged

distribution

  • Solve from Ph backward
  • Offline algorithm

New Mexico State University

P0 Ph-1 Ph Ph* Xh* Xh*

BACKWARD

Xh-1

slide-62
SLIDE 62

New Mexico State University

All About Discovery!

nmsu.edu

Content

  • Distributed Constraint Optimization Problem
  • Proactive Dynamic DCOPs
  • Infinite-Horizon Proactive Dynamic DCOP
  • Algorithms

ØExperiments

  • Conclusions

New Mexico State University

slide-63
SLIDE 63

New Mexico State University

All About Discovery!

nmsu.edu

Experimental setup

  • Random network
  • Number of variables

– Decision = Random: 8

  • Horizon:

5

  • Constraint density:

0.5

  • Real distributed system, actual runtime

New Mexico State University

slide-64
SLIDE 64

New Mexico State University

All About Discovery!

nmsu.edu

Experiment with Offline Algorithms

S-DPOP:

  • Not consider switching cost

LS-SDPOP:

  • Suffer from bad initial solutions

BACKWARD, FORWARD:

  • Similar runtimes to S-DPOP
  • Better solutions

200 400 600 800 1000 0.850 0.900 0.950 1.000 Runtimes (ms)

Normalized Rewards

  • ●
  • ●
  • ●
  • ●
  • FORWARD

BACKWARD S−DPOP LS−SDPOP

Switching cost = 100,000

New Mexico State University

slide-65
SLIDE 65

New Mexico State University

All About Discovery!

nmsu.edu

Experiment with Offline Algorithms

LS-SDPOP:

  • Slower, better solutions

Other algorithms:

  • Start to differ

200 400 600 800 1000 0.997 0.998 0.999 1.000 Runtimes (ms)

Normalized Rewards

  • ●
  • FORWARD

BACKWARD S−DPOP LS−SDPOP

Switching cost = 1,000

New Mexico State University

slide-66
SLIDE 66

New Mexico State University

All About Discovery!

nmsu.edu

Experiment with Offline Algorithms

  • Small switching cost
  • Agent sticks to initial solutions
  • Similar quality, runtime

200 400 600 800 1000 0.997 0.998 0.999 1.000 Runtimes (ms)

Normalized Rewards

  • ●
  • ●
  • FORWARD

BACKWARD S−DPOP LS−SDPOP

Switching cost = 10

New Mexico State University

slide-67
SLIDE 67

New Mexico State University

All About Discovery!

nmsu.edu

Experiment with Online Algorithms

New Mexico State University

time step 1 2 time 0 ms 500 ms 1000 ms ONLINE FORWARD ONLINE REACTIVE HYBRID

slide-68
SLIDE 68

New Mexico State University

All About Discovery!

nmsu.edu

Experiment with Online Algorithms

  • Reactive

P0 Solving

New Mexico State University

slide-69
SLIDE 69

New Mexico State University

All About Discovery!

nmsu.edu

Experiment with Online Algorithms

  • Reactive

P0 P1 Adopting

New Mexico State University

slide-70
SLIDE 70

New Mexico State University

All About Discovery!

nmsu.edu

Experiment with Online Algorithms

  • Reactive

P0 P1 Adopting

New Mexico State University

slide-71
SLIDE 71

New Mexico State University

All About Discovery!

nmsu.edu

Experiment with Online Algorithms

  • Reactive

P0 P1 Solving

New Mexico State University

slide-72
SLIDE 72

New Mexico State University

All About Discovery!

nmsu.edu

Experiment with Online Algorithms

  • Reactive

P0 P1 P2 Adopting

New Mexico State University

slide-73
SLIDE 73

New Mexico State University

All About Discovery!

nmsu.edu

Experiment with Online Algorithms

  • Online FORWARD

P0 P1 P2 Adopting Solving

New Mexico State University

slide-74
SLIDE 74

New Mexico State University

All About Discovery!

nmsu.edu

Experiment with Online Algorithms

  • Online HYBRID

P0 P1 P2 Adopting Solving

New Mexico State University

slide-75
SLIDE 75

New Mexico State University

All About Discovery!

nmsu.edu

Experiment with Online Algorithms

Reactive:

  • Small switching cost
  • Large time duration

FORWARD:

  • Large switching cost
  • Small time duration

12000 Difference in Effective Rewards 8400

  • 2400

2000 1200 4800 1500 1000 Switching Costs 500 1000 2000 Time Durations 3000 4000

New Mexico State University

slide-76
SLIDE 76

New Mexico State University

All About Discovery!

nmsu.edu

Experiment with Online Algorithms

Similar results HYBRID > FORWARD

  • 2400

2000 1200 4800 8400 1500 1000 Switching Costs 500 1000 2000 Time Durations 4000 3000 12000 Difference in Effective Rewards

New Mexico State University

slide-77
SLIDE 77

New Mexico State University

All About Discovery!

nmsu.edu

Conclusions

  • Infinite-Horizon Proactive Dynamic DCOP:

– Optimal solution at time step h onwards – Random variables as Markov chains – Markov chain convergence

  • Experiments:

– Comparison between proactive and reactive dynamic DCOP algorithms (first time!)

New Mexico State University

slide-78
SLIDE 78

New Mexico State University

All About Discovery!

nmsu.edu

Thank you

New Mexico State University

slide-79
SLIDE 79

New Mexico State University

All About Discovery!

nmsu.edu

slide-80
SLIDE 80

New Mexico State University

All About Discovery!

nmsu.edu

Markov chains

  • Markov property: Memoryless
slide-81
SLIDE 81

New Mexico State University

All About Discovery!

nmsu.edu

Markov chain properties

  • A state j is said accessible from state i (i -> j)
  • State i and state j communicate (i <-> j)
  • A class of states: communicate each other
slide-82
SLIDE 82

New Mexico State University

All About Discovery!

nmsu.edu

Markov chain properties

  • Period of a state i:
  • Aperiodic: period = 1
slide-83
SLIDE 83

New Mexico State University

All About Discovery!

nmsu.edu

Markov chain properties

  • Recurrent: (i -> j) => (j -> i)
  • Transient: otherwise
  • Ergodic: Both aperiodic and recurrent
slide-84
SLIDE 84

New Mexico State University

All About Discovery!

nmsu.edu

Markov chain properties

  • Recurrent: (i -> j) => (j -> i)
  • Transient: otherwise
  • Ergodic: Both recurrent and aperiodic
slide-85
SLIDE 85

New Mexico State University

All About Discovery!

nmsu.edu

Markov chain properties

  • Unichain: A chain that contains

– Single recurrent class – Probably some transient states

slide-86
SLIDE 86

New Mexico State University

All About Discovery!

nmsu.edu

Markov chain properties

  • Unichain: A chain that contains

– Single recurrent class – Ergodic unichain: aperiodic – Probably some transient states

slide-87
SLIDE 87

New Mexico State University

All About Discovery!

nmsu.edu

Markov chains

  • Convergence:
  • Conditions on convergence:

1.Positive transition matrix 2.All states: one single class and ergodic 3.The chain is an ergodic unichain

slide-88
SLIDE 88

New Mexico State University

All About Discovery!

nmsu.edu

FORWARD

  • Solve the last time step with stationary

distribution

  • Solve from time step 0
  • At time step h-1:
  • Either online or offline algorithms
slide-89
SLIDE 89

New Mexico State University

All About Discovery!

nmsu.edu

BACKWARD

  • Solve the last time step with stationary

distribution

  • Solve from time step h-1 backwards
  • Offline algorithms