Optimal decentralized control of coupled subsystems with control - - PowerPoint PPT Presentation

Optimal decentralized control

• f coupled subsystems

with control sharing

Aditya Mahajan

McGill University

IEEE Conference on Decision and Control, 2011

A Mahajan (McGill) Control sharing info struc 1

Notation

Random variables: 𝑌, realizations: , state spaces: 𝒴. 𝑏􀐊

􀐕 means that variable 𝑏 belongs to subsystem at time .

𝑏􀍣:􀐕 = 𝑏􀍣, 𝑏􀍤, …, 𝑏􀐕 𝐛 = 𝑏􀍣, 𝑏􀍤, …, 𝑏􀐏.

A Mahajan (McGill) Control sharing info struc 2

System Model

􀍣

􀐕

􀍤

􀐕

⋯ 􀐏

􀐕

􀍣

􀐕

􀍤

􀐕

⋯ 􀐏

􀐕

􀍣

􀐕

􀍤

􀐕

􀐏

􀐕

𝐯􀐕􀍭􀍣 𝐯􀐕􀍭􀍣 𝐯􀐕􀍭􀍣

Control-coupled subsystems Controller with control sharing

􀐊

􀐕􀍬􀍣 = 􀐊 􀐕 􀐊 􀐕, 𝐯􀐕, 􀐊 􀐕

􀐊

􀐕 = 􀐊 􀐕􀐊 􀍣:􀐕, 𝐯􀍣:􀐕􀍭􀍣

Objective

min

all policies 𝐡 𝔽 [ 􀏻

∑

􀐕􀍮􀍣

􀐕𝐲􀐕, 𝐯􀐕]

A Mahajan (McGill) Control sharing info struc 3

Some applications

Feedback communication systems (physical layer)

Point-to-point real-time source coding, multi-terminal source coding with feedback, some classes of multiple access channel with feedback

Queueing networks (media access layer)

Multi-access broadcast, some classes of decentralized scheduling and routing.

Cellular networks

Paging and registration in cellular networks

A Mahajan (McGill) Control sharing info struc 4

Conceptual difficulties

The system has non-classical information structure Data at each controller is increasing with time

􀐊

􀐕 = 􀐊 􀐕􀐊 􀍣:􀐕, 𝐯􀍣:􀐕􀍭􀍣

Is part of this data redundant? Can part of this data be compressed to a sufficient statistic?

Multi-stage decision making

How does current control action affect future estimation? What information does controller communicate to controller via its control action?

A Mahajan (McGill) Control sharing info struc 5

Literature Overview

Control sharing info-structure (Bismut, 1972, Sandell and Athans, 1974)

Considered the LQG version of the problem Exploit the fact that the action space is continuous and compact to embed the observations in control Reduces to one-step delayed sharing pattern

Other non-classical info-structures with sharing

Delayed state sharing: Aicadri, Davoli, and Minciardi, 1987 Delayed (observation) sharing: Witsenhausen 1971, Varaiya and Walrand, 1979, Nayyar, Mahajan, and Teneketzis, 2011 Periodic sharing: Ooi, Verbout, Ludwig, Wornell, 1997 Belief sharing: Yüksel, 2009 Partial history sharing: Mahajan, Nayyar, Teneketzis, 2008

A Mahajan (McGill) Control sharing info struc 6

Outline of the results

First structural result (based on person-by-person opt.)

􀐊

􀍣:􀐕􀍭􀍣 is redundant for optimal performance.

wlo, 􀐊

􀐕 = 􀐊 􀐕􀐊 􀐕, 𝐯􀍣:􀐕􀍭􀍣

Second structural result (based on common info approach

• f MNT 2008)

Define Π􀐊

􀐕 = ℙ𝑌􀐊 􀐕 = | 𝐕􀍣:􀐕􀍭􀍣 and 𝚸􀐕 = Π􀍣 􀐕 , …, Π􀐏 􀐕 .

𝝆􀐕 is a sufficient statistic of 𝐯􀍣:􀐕􀍭􀍣 for optimal performance. wlo, 􀐊

􀐕 = 􀐊 􀐕􀐊 􀐕, 𝝆􀐕

Dynamic programming decomposition

A Mahajan (McGill) Control sharing info struc 7

Structural result based on person-by-person

• ptimality

Main lemma

The states processes are conditionally independent given the past control actions. ℙ𝐘􀍣:􀐕 = 𝐲􀍣:􀐕 | 𝐕􀍣:􀐕 =

􀐏

∏

􀐊􀍮􀍣

ℙ𝑌􀐊

􀍣:􀐕 = 􀐊 􀍣:􀐕 | 𝐕􀍣:􀐕

Implications

Fix 􀍭􀐊 and consider optimal design of 􀐊. Let 𝑆􀐊

􀐕 = 𝑌􀐊 􀐕, 𝐕􀍣:􀐕􀍭􀍣. Then

{𝑆􀐊

􀐕, = , …} is a controlled MDP with control action 􀐊 􀐕.

ℙ􀐊

􀐕􀍬􀍣 | 􀐊 􀍣:􀐕, 􀐊 􀍣:􀐕 = ℙ􀐊 􀐕􀍬􀍣 | 􀐊 􀐕 , 􀐊 􀐕

𝔽[􀐕𝐲􀐕, 𝐯􀐕 | 􀐊

􀍣:􀐕, 􀐊 􀍣:􀐕] = 𝔽[􀐕𝐲􀐕, 𝐯􀐕 | 􀐊 􀐕 , 􀐊 􀐕]

Structural result . . . (cont.)

A Mahajan (McGill) Control sharing info struc 8

Original model

􀐊

􀐕 = 􀐊 􀐕􀐊 􀍣:􀐕, 𝐯􀍣:􀐕􀍭􀍣

Implication of person-by-person optimality argument

􀐊

􀐕 = 􀐊 􀐕􀐊 􀐕 = 􀐊 􀐕􀐊 􀐕, 𝐯􀍣:􀐕􀍭􀍣

Design difficulty

Data at the controller is still increasing with time

A Mahajan (McGill) Control sharing info struc 9

A coordinator based on common information

General idea proposed in (Mahajan, Nayyar, and Teneketzis 2008)

𝑌􀍣

􀐕 , 𝐕􀍣:􀐕􀍭􀍣

𝑌􀍤

􀐕 , 𝐕􀍣:􀐕􀍭􀍣

􀍣

􀐕

􀍤

􀐕

􀍣

􀐕

􀍤

􀐕

A coordinator based on common information (cont.)

A Mahajan (McGill) Control sharing info struc 10

𝑌􀍣

􀐕

𝑌􀍤

􀐕

𝐕􀍣:􀐕􀍭􀍣 􀍣

􀐕

􀍤

􀐕

􀍣

􀐕

􀍤

􀐕

ℎ􀐕 􀍣

􀐕 , 􀍤 􀐕

where 􀐊

􀐕⋅ = 􀐊 􀐕⋅, 𝐯􀍣:􀐕􀍭􀍣

A coordinator based on common information (cont.)

A Mahajan (McGill) Control sharing info struc 11

Solution approach

The coordinated system is a POMDP Identify the structure of optimal coordination strategies for the coordinated system Show that the coordinated system is equivalent to the original model Translate the structure of optimal coordination strategies to the

• riginal model

A Mahajan (McGill) Control sharing info struc 12

The coordinated system

𝑌􀍣

􀐕

𝑌􀍤

􀐕

𝐕􀍣:􀐕􀍭􀍣 􀍣

􀐕

􀍤

􀐕

􀍣

􀐕

􀍤

􀐕

ℎ􀐕 􀍣

􀐕 , 􀍤 􀐕

State: 𝐲􀐕 = 􀍣

􀐕 , …, 􀐏 􀐕

Observations: 𝐯􀐕􀍭􀍣 = 􀍣

􀐕􀍭􀍣, …, 􀐏 􀐕􀍭􀍣

Control actions: 𝐞􀐕 = 􀍣

􀐕 , …, 􀐏 􀐕 ,

Coordination rule: ℎ􀐕 : (

􀐏

∏

􀐊􀍮􀍣

𝒱􀐊)

􀐕􀍭􀍣 : 􀐏

∏

􀐊􀍮􀍣

𝒴􀐊 → 𝒱􀐊

􀐕

𝐞􀐕 = ℎ􀐕𝐯􀍣:􀐕􀍭􀍣

Structure of optimal coordination strategy

Define Ξ􀐕 = ℙstate | history of observations = ℙ𝐲 | 𝐕􀍣:􀐕􀍭􀍣. Then, wlo, 𝐞􀐕 = ℎ􀐕𝜊􀐕

The coordinated system (cont.)

A Mahajan (McGill) Control sharing info struc 13

Dynamic programming decomposition

• 􀐕𝜊 = min

𝐞 𝔽 [􀐕𝐘􀐕, 𝐕􀐕 + 􀐕􀍬􀍣Ξ􀐕􀍬􀍣 | Ξ􀐕 = 𝜊]

Salient features

The optimization at each step is a functional optimization problem. (In our opinion) functional optimization at each step is the only way to circumvent the issue of signaling.

A Mahajan (McGill) Control sharing info struc 14

Translation of results back to the original system

𝑌􀍣

􀐕

𝑌􀍤

􀐕

𝐕􀍣:􀐕􀍭􀍣 􀍣

􀐕

􀍤

􀐕

􀍣

􀐕

􀍤

􀐕

ℎ􀐕 􀍣

􀐕 , 􀍤 􀐕

Structural result

wlo, 􀐊

􀐕 = 􀐊 􀐕􀐊 􀐕 = ℎ􀐊 􀐕𝜊􀐕􀐊 􀐕 = 􀐊 􀐕􀐊 􀐕, 𝜊􀐕

Dynamic programming decomposition

Solve the DP for coordinated system. Choose 􀐊

􀐕􀐊 􀐕, 𝜊􀐕 = ℎ􀐊 􀐕𝜊􀐕􀐊 􀐕

A Mahajan (McGill) Control sharing info struc 15

Further simplification of structural result

Recall main lemma:

The states processes are conditionally independent given the past control actions. ℙ𝐘􀍣:􀐕 = 𝐲􀍣:􀐕 | 𝐕􀍣:􀐕 =

􀐏

∏

􀐊􀍮􀍣

ℙ𝑌􀐊

􀍣:􀐕 = 􀐊 􀍣:􀐕 | 𝐕􀍣:􀐕

Implication

𝜊􀐕𝐲 = ℙ𝐘􀐕 = 𝐲 | 𝐕􀍣:􀐕􀍭􀍣 =

􀐏

∏

􀐊􀍮􀍣

𝜌􀐊

􀐕􀐊 􀐕

Further simplification of structural result (cont.)

A Mahajan (McGill) Control sharing info struc 16

Simplified structural result

wlo, 􀐊

􀐕 = 􀐊 􀐕􀐊 􀐕, 𝜊􀐕 = 􀐊 􀐕􀐊 􀐕, 𝝆􀐕

Significant reduction is size. 𝜊􀐕 ∈ Δ𝒴􀍣 × ⋯ × 𝒴􀐏 while 𝝆􀐕 ∈ Δ𝒴􀍣 × ⋯ × Δ𝒴􀐏

Simplified dynamic programming decomposition

• 􀐕𝝆 = min

𝐞 𝔽 [􀐕𝐘􀐕, 𝐕􀐕 + 􀐕􀍬􀍣𝚸􀐕􀍬􀍣 | 𝚸􀐕 = 𝝆]

A Mahajan (McGill) Control sharing info struc 17

Recap of structural results

Original: 􀐊

􀐕 = 􀐊 􀐕􀐊 􀍣:􀐕, 𝐯􀍣:􀐕􀍭􀍣

Using person-by-person approach 􀐊

􀐕 = 􀐊 􀐕􀐊 􀐕, 𝐯􀍣:􀐕􀍭􀍣

Using the common information approach of (NMT 2008, 2011) 􀐊

􀐕 = 􀐊 􀐕􀐊 􀐕, 𝜊􀐕,

𝜊􀐕 = ℙ𝐘􀐕 | 𝐯􀍣:􀐕􀍭􀍣 Using specific conditional independence due to the dynamics 􀐊

􀐕 = 􀐊 􀐕􀐊 􀐕, 𝝆􀐕,

𝜌􀐊

􀐕 = ℙ𝑌􀐊 􀐕 | 𝐯􀍣:􀐕􀍭􀍣

A Mahajan (McGill) Control sharing info struc 18

An Example: Two-user multiple access broadcast

Two-user with single slot buffer

􀐊

􀐕 ∈ {, }: # of packets in queue

􀐊

􀐕 ∈ {, }: # of arrivals ∼ Ber􀐊

􀐊

􀐕 ∈ {, }: # of transmitted packets

Multiple-access channel

Throughput:

􀐕 = 􀍣 􀐕 − 􀍤 􀐕 + − 􀍣 􀐕 􀍤 􀐕

• 􀐕 available to both users after one-step delay

State update: 􀐊

􀐕􀍬􀍣 = max (􀐊 􀐕 − 􀐊 􀐕 􀐕 + 􀐊 􀐕, )

An Example: Two-user multiple access broadcast (cont.)

A Mahajan (McGill) Control sharing info struc 19

Literature overview

Symmeric arrivals: Hlyuchj and Gallager, 1981 feasible lower bound Symmeric arrivals: Ooi and Wornell, 1996 genie aided upper bound that numerically matched lower bound. Asymmetric arrivals: Used as benchmark problem in AI community (Hansen et al, 2004, Bernstein et al, 2005, Shez Charpillet, 2006) for numerical algorithms for DEC-POMDPs.

An Example: Two-user multiple access broadcast (cont.)

A Mahajan (McGill) Control sharing info struc 20

Structure of optimal control policy

𝜌􀐊

􀐕 is equivalent to 𝜌􀐊 􀐕 =∶ 􀐊 􀐕 ∈ {, }

􀐊

􀐕 is equivalent to 􀐊 􀐕 =∶ 􀐊 􀐕 ∈ {, }

Structure of optimal policy 􀐊

􀐕 = 􀐊 􀐕 ⋅ 􀐊 􀐕,

where 􀍣

􀐕 , 􀍤 􀐕 = ℎ􀐊 􀐕􀍣 􀐕 , 􀍤 􀐕

An Example: Two-user multiple access broadcast (cont.)

A Mahajan (McGill) Control sharing info struc 21

Optimal policy for symmetric arrivals

Notation: for any ∈ [, ], let 𝐵 = − − − Characteristic polynomial: 𝜒􀐏 = + − 􀍤 − + − 􀐏􀍬􀍣. Let 𝛽􀐏 be the root of 𝜒􀐏 in [, ] and 𝜐 be the root of = − 􀍤 Optimal performance: 𝐾* = { − − 􀍤, if 𝛽􀍣 − 􀍤 − / + 􀍤 + 􀍥,

• therwise

An Example: Two-user multiple access broadcast (cont.)

A Mahajan (McGill) Control sharing info struc 22

Optimal policy

When > 𝜐 ℎ*􀍣, 􀍤 = { , if 􀍣 > 􀍤 , if 􀍣 < 􀍤 , or , if 􀍣 = 􀍤 When < 𝜐, let 𝑜 ∈ ℕ be such that 𝛽􀐏􀍬􀍣 < 𝛽􀐏. ℎ*􀍣, 􀍤 =

• ,

if 􀍣 𝐵􀐏 and 􀍤 𝐵􀐏 , if 􀍣 > max𝐵􀐏, 􀍤 , if 􀍤 > max𝐵􀐏, 􀍣 , or , if 􀍣 = 􀍤 =

Analytic proof of optimality of the policy proposed by Hlyuchj and Gallager, 1981.

A Mahajan (McGill) Control sharing info struc 23

Conclusion

Coupled subsystems with control-sharing

Non-classical information structure Use properties

• f

the system dynamics and the common information approach of (Mahajan, Nayyar, Teneketzis 2008) to find structure of optimal controller and a dynamic programming decomposition. Allows using standard tools from stochastic control to analyze specific applications.

Key take-home points

Subclasses of decentralized control problems with signaling are solvable! Each step of the DP is a functional optimization problem.