Dual Effect in Stochastic Optimization February 10, 2015 P. - - PowerPoint PPT Presentation

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization

Dual Effect in Stochastic Optimization

February 10, 2015

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 162 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization

Introduction

During the first part of the course, we have studied open-loop stochastic optimization problems, that is, problems in which the decisions correspond to deterministic variables which minimize a cost function defined as an expectation. min

u∈Uad E

• j(u, W )
• .

We now entre the realm of closed-loop stochastic optimization, that is, the case where on-line information is available to the decision maker. The decisions are thus functions of information and correspond to random variables. min

U∈Uad E

• j(U, W )
• .
• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 163 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization

Lecture Outline

1

Closed Loop Stochastic Optimization Problems Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

2

Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 164 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

1

Closed Loop Stochastic Optimization Problems Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

2

Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 165 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

1

Closed Loop Stochastic Optimization Problems Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

2

Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 166 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

Variables and Constraints

The decision variable U is now a random variable and belongs to a functional space U. A canonical example is: U = L2(Ω, A, P; U). The contraints on the r.v. U may be of different nature: point-wise constraints dealing with the possible values of U: U ∈ Upo =

• U ∈ U, U(ω) ∈ Uad P-a.s.
• ,

risk constraints, such as expectation or probability constraints: U ∈ Uri =

• U ∈ U, P
• Θ(U) ≤ θ
• ≥ π
• ,

measurability constraints which express the fact that a given amount of information Y is available to the decision maker: U ∈ Ume =

• U ∈ U, U is measurable w.r.t. Y
• .

We will mainly concentrate on the measurability constraints.

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 167 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

Compact Formulation of a Closed-Loop Problem

Given a probability space (Ω, A, P), the essential ingredients of a stochastic optimization problem are noise W : r.v. with values in a measurable space (W, W), decision U: r.v. with values in a measurable space (U, U), information Y : r.v. with values in a measurable space (Y, Y), a cost function: measurable mapping j : U × W → R. The σ-field generated by W (resp. Y ) is denoted F (resp. G). With all these elements at hand, the problem is set as follows: min

UY

E

• j(U, W )
• .

The notation U Y (or equivalently U G) is used to express that the r.v. U is measurable w.r.t. to the σ-field generated by Y .

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 168 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

Representation of Measurability Constraints

Consider the information structure of the stochastic optimization problem in a compact form, that is, the measurability constraints U Y . This information structure may be interpreted in different ways. From the functional point of view, using Doob’s Theorem, the decision U is expressed as a measurable function of Y : U = ϕ(Y ) . In this setting, the decision variable becomes the function ϕ. From the algebraic point of view, the constraints are expressed in terms of σ-field, that is, σ

• U
• ⊂ σ
• Y
• .

Question: how to take all these representations into account?

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 169 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

Static Information Structure (SIS)

This is the case when G = σ(Y ) is fixed, defined independently of the decision U. Therefore, the terminology “static” expresses that the σ-field G constraining the decision cannot be modified by the decision maker. It does not imply that no dynamics is present in the problem formulation.11 If the information Y is defined as a function of the noise W , that is, Y = h(W ), it generates a static information structure. Note that it may happen that Y does depend on U whereas the σ-field G it generates remains fixed. Remember from now that SIS will be the “easy” case.

11If time is involved in the optimization problem, a decision Ut has to be

taken at each time t, based on an information Yt, so that a measurability constraint Ut Yt is written at each time stage t.

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 170 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

Dynamic Information Structure (DIS) (1)

This is the situation when G = σ(Y ) depends on U. For example, in the case where Y = h(U, W ), the constraint reads U h(U, W ) , which yields a (seemingly) implicit measurability constraint. This is a source of huge complexity for stochastic optimization problems, known under the name of the dual effect of control. Indeed, the decision maker has to take care of the following double effect:

• n the one hand, his decision affects the cost E
• j(U, W )
• ,
• n the other hand, she makes the information more or less

constrained, that is, a less or more large admissible set.

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 171 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

Dynamic Information Structure (DIS) (2)

It will be easier to imagine such problems by explicitly introducing several agents which take decisions based on observations which may depend on decisions of other agents. Those agents may be a priori ordered. Then the notion of causality (who is “upstream” and who is “downstream”) becomes relevant, and it turns out that two notions are paramount for the level of difficulty of the problem:

1 who influences the available information of whom? 2 who knows more than whom?

We will illustrate these subtle notions and questions in the case of stochastic optimal control, for which an “agent” takes a decision at each time stage t of the time horizon

• 0, . . . , T − 1
• .
• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 172 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

1

Closed Loop Stochastic Optimization Problems Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

2

Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 173 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

Stochastic Optimal Control Problem. . .

An standard form for a stochastic optimization problem involving a dynamic process X over a time horizon

• 0, . . . , T
• is:

min

(U0,...,UT−1,X0,...,XT ) E

T−1

• t=0

Lt(Xt, Ut, Wt+1) + K(XT)

• subject to the dynamic constraints

X0 = f−1(W0) , Xt+1 = ft(Xt, Ut, Wt+1) , t = 0, . . . , T − 1 . We denote by Ft the σ-field generated by noises prior time t: Ft = σ

• W0, . . . , Wt
• , t = 0, . . . , T .

Nonanticipativity: Ft is the maximal information available at t.

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 174 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

and a Possible Information Structure

Information Structure An observation becomes available at time t: Zt = gt(Xt, Wt) , t = 0, . . . , T − 1 . Zt = Wt: observation of the noise, Zt = Xt: observation of the state. The information available at t is a function of past observations: Yt = Ct

• Z0, . . . , Zt
• , t = 0, . . . , T − 1 .

Yt = (Z0, . . . , Zt): perfect memory. Information Constraints Ut Yt ⇐ ⇒

• Ut

= ϕt

• Yt
• σ
• Ut
• ⊂ σ
• Yt
• , t = 0, . . . , T − 1 .
• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 175 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

Remarks About the Information Structure (1)

The problem is formulated in the Decision-Hazard framework: the decision Ut at time t must be chosen before Wt+1 occurs. In this setting, the information Yt has the following structure: Yt = Ct(Z0, . . . , Zt) = Ct

• g0(X0, W0), . . . , gt(Xt, Wt)
• = Ct
• g0
• f−1(W0), W0
• , . . . , gt
• ft−1(Xt−1, Ut−1, Wt), Wt
• .

. . = ht(U0, . . . , Ut−1, W0, . . . , Wt) . We are in a specific case of Dynamic Information Structure: information at time t depends on past noises, information at time t depends on (strictly) past controls.

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 176 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

Remarks About the Information Structure (2)

The information constraints are as follows: U0 σ

• h0(W0)
• ⊂ σ
• W0
• .

U1 σ

• h1(U0, W0, W1)
• ⊂ σ
• W0, W1
• .

. . . Ut σ

• ht(U0, . . . , Ut−1, W0, . . . , Wt)
• Gt

⊂ σ

• W0, . . . , Wt
• Ft

. The causality principle is fulfilled (no dependency on the future), but information depends on past controls, so that dual effect is possible and controls (U0, . . . , Ut−1) may be used to make the σ-field Gt as large as possible. Otherwise stated, decisions prior time t allow to transmit information up to time t.

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 177 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

Functional Approach: Dynamic Programming

Assume that the noise process (W0, . . . , WT) corresponds to a white noise. Then the three following information structures:

1 Yt = (W0, . . . , Wt), 2 Yt = (X0, . . . , Xt), 3 Yt = Xt,

lead to the same optimal solution of the problem. Moreover, this solution can be obtained by solving the Bellman equation: VT(x) = K(x) , Vt(x) = min

u∈U E

• Lt(x, u, Wt+1) + Vt+1
• ft(x, u, Wt+1)
• ,

U ♯

t = ϕ♯ t(Xt): functional approach applied to this specific DIS.

However, in the general case, the solution of the stochastic optimal control problem is not known (see Witsenhausen counterexample).

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 178 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

Algebraic Approach: Stochastic Programming

There are cases where a stochastic optimal control problem corresponds to a static information structure, for example Yt = ht(W0, . . . , Wt) , t = 0, . . . , T − 1 . There are also cases where the r.v. Yt depends on the controls whereas the associated σ-fields remain fixed. In all these situations, it is possible to use the algebraic approach and to look for the solution of the problem in terms of random variables satisfying fixed measurability contraints: σ

• Ut
• ⊂ σ
• Yt
• , t = 0, . . . , T − 1 .

First issue: characterize the class of problems that can be solved by this approach (lack of dual effect). Second issue: obtain a finite approximation of the problem, and more specifically discretize the information constraints.

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 179 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

1

Closed Loop Stochastic Optimization Problems Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

2

Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 180 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

A Simple Linear Quadratic Control Problem

Consider (Ω, A, P) a probability space, and six real-valued random variables related by the following dynamic equations: X0 = W0 , X1 = X0 + U0 , X2 = X1 − U1 . The optimization problem under consideration is min

U0Y0 , U1Y1

E

• k2U2

0 + X 2 2

• ,

Y0 and Y1 being the information available at t = 0 and t = 1. We will examine different choices for the information structure.

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 181 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

Problem Transformation

By an interchange of minimization and (conditional) expectation, the initial problem is equivalent to min

U0Y0

E

• k2U2

0 + min u1∈R E

• (X1 − u1)2

Y1

• .

The arg min of the inner optimization problem is the conditional expectation E

• X1
• Y1
• , and the associated optimal cost is, by

definition, the conditional variance: Var

• X1 | Y1
• = E
• X 2

1

• Y1
• −
• E
• X1
• Y1

2 . The solution of the initial problem is thus U ♯

0 = arg min U0Y0

E

• k2U2

0 + Var

• X0 + U0 | Y1

, U ♯

1 = E

• X0 + U ♯
• Y1
• .
• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 182 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

Information Patterns (1)

Full Noise Observation Y0 = W0 , Y1 = W0 . X0 = W0 and U0 Y0 imply that X1 = X0 + U0 Y1, so that Var

• X1 | Y1
• = 0 .

We thus deduce that U ♯

0 = 0 and U ♯ 1 = X0.

Full State Observation Y0 = X0 , Y1 = (X0, X1) . Obviously we have X1 Y1, so that U ♯

0 = 0 and U ♯ 1 = X0.

Note that this result remains true in the Markovian case Y1 = X1.

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 183 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

Information Patterns (2)

Classical Information Pattern (Noisy Observation of the State) Y0 = X0 , Y1 = (X0, X1 + W1) . We have X1 = X0 + U0 X0 Y1, so that U ♯

0 = 0 and U ♯ 1 = X0.

State-Control Observation Y0 = X0 , Y1 = (U0, X1 + W1) . It is shown that the problem admits only ε-optimal solutions, that is U0 = εY0 = εX0 and U1 = U0/ε = X0. For ε > 0, U0 = εX0 ⇒ σ(U0) = σ(X0), hence the result. For ε = 0, U0 ≡ 0 so that σ(Y1) = σ(X0 + W1) = σ(X0).

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 184 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

Information Patterns (3)

Witsenhausen Counterexample Y0 = X0 , Y1 = X1 + W1 . An optimal solution exists, but its expression is unknown! Intuitive point of view. The information Y0 available at time t = 0 is forgotten at time t = 1. The decision U0 may try to transmit information at time t = 1 (dual effect). For example, using the feedback law U0 = αX0, α ≫ 0, we have Y1 = X1 + W1 = (1 + α)X0 + W1 ≈ (1 + α)X0 . But such a feedback is expensive tradeoff information/cost.

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 185 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

1

Closed Loop Stochastic Optimization Problems Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

2

Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 186 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

1

Closed Loop Stochastic Optimization Problems Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

2

Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 187 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

Classical Measurability Framework

The standard measurability theory makes use of σ-fields. Definition A σ-field on Ω is a nonempty collection A of subsets of Ω which is stable under complementation and countable union (and hence under countable intersection). The following fundamental result is due to J. Doob. Theorem Consider two mappings Hi : (Ω, A) → (Yi, Yi), i = 1, 2. Assume that Y1 is a separable complete metric space. The mapping H1 is measurable w.r.t. H2 if and only if there exists a measurable mapping f : imH2 → imH1 such that H1 = f ◦ H2.

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 188 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

Information and σ-Fields

As noted by [Dubra & Echenique, 2004], and contrarily to a widely accepted idea, the use of σ-fields as the informational content of a signal is not without raising serious problems. Example On Ω = [0, 1], consider the two partitions C♭ =

• [0, 1/2], ]1/2, 1]
• and C♯ =
• {ω}
• ω∈[0,1] (complete partition). Of course, the set C♯

gives more information that C♭. Since C♭ is a finite partition, the σ-field it generates is σ(C♭) =

• ∅, [0, 1], [0, 1/2], ]1/2, 1]
• . It can

be seen that the σ-field generated by C♯ is made of subsets of Ω which are either countable or whose complement is countable. . . To summarize, we have that C♭ C♯, whereas σ(C♭) and σ(C♯) are not comparable.

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 189 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

Partition Fields or π-Fields

The inclusion order on σ-fields is thus not compatible with the

• rder on partitions: using σ-fields in order to express information

may be tricky.12 We now claim that partition fields are adequate to represent information. Definition A partition field (or π-field) on Ω is a nonempty collection G of subsets of Ω which is stable under complementation and unlimited union (and hence under unlimited intersection). A π-field may be a large collection of subsets: the π-field generated by all singletons of Ω is the collection of all subsets of Ω, that is, 2Ω. Partition fields are not used in Probability Theory because they are generally too large to support a probability law.

12However, the interest of σ-fields is that they can support a probability law.

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 190 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

Properties of Partition Fields

Definition Consider a collection G of subsets of Ω. An atom of G is a subset G ∈ G such that K ∈ G and K ⊂ G imply that K = ∅ or K = G. Theorem Consider G a π-field of Ω. The atoms of G form a partition of Ω, denoted by part(G), which generates G: π

• part(G)
• = G .

Theorem Let G and G′ be two π-fields of Ω. The π-field G is finer than G′ if and only if every atom of G′ is the union of G-atoms: G′ G ⇐ ⇒ part(G′) part(G) .

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 191 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

Measurability w.r.t. π-Fields (1)

Definition Let Ω be equipped with a π-field G, and let Y be another set equipped with the complete π-field Y = 2Y. The mapping H : Ω → Y is said to be measurable w.r.t. G if the π-field generated by H, that is, π(H) := H−1(Y), is such that π(H) G. Theorem Consider a mapping H : Ω → Y and a π-field G with associated partition part(G). The two following assertions are equivalent.

1 The mapping H is measurable w.r.t. the π-field G. 2 The mapping H is constant over each element of part(G).

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 192 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

Measurability w.r.t. π-Fields (2)

Definition Consider two mappings Hi : Ω → Yi, i = 1, 2. The mapping H1 is said to be measurable w.r.t. the mapping H2 if π(H1) π(H2). Theorem Consider two mappings Hi : Ω → Yi, i = 1, 2. The following conditions are equivalent characterizations of the fact that H1 is measurable w.r.t. H2.

1 ∀(ω, ω′) ∈ Ω × Ω , H2(ω) = H2(ω′) ⇒ H1(ω) = H1(ω′) 2 ∃! f : imH2 → imH1. such that H1 = f ◦ H2.

Similar conditions are available in the case where H1 and H2 are equivalent mappings: H1 ≡ H2 ⇔ (H1 H2 and H2 H1).

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 193 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

Measurability w.r.t. π-Fields (3)

H1 H−1

2

Ω H1 =

• H2

−1

f Figure: Measurability relation H1 H2

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 194 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

1

Closed Loop Stochastic Optimization Problems Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

2

Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 195 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

Problem Statement (1)

From now, the framework is the one of measurability w.r.t. π-fields and not w.r.t. σ-fields. Let Ω be a set equipped with the π-field G. The r.v. entering a stochastic optimization problem are a noise W : Ω → W: mapping with values in a space W, a decision U : Ω → U: mapping with values in a space U; the set of all possible decisions is denoted by U, a information Y : Ω → Y: mapping with values in a space Y; the information is given as a function h : U × W → Y: Y = h

• U, W
• ,

so that the information depends on the control (DIS). The information constraints of the problem write U Y so that the admissible set upon which the optimization problem relies is Uad =

• U ∈ U , U Y
• .
• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 196 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

Problem Statement (2)

The π-field generated by Y depends on U: dual effect holds true. We want to characterize the greatest set Unde ⊂ Uad such that the information π-field generated by any U ∈ Unde remains fixed: π

• h(U, W )
• = π
• h(U ′, W )
• ∀(U, U ′) ∈ Unde × Unde .

This condition is equivalently formulated as13 h

• U(ω), W (ω)
• = h
• U(ω′), W (ω′)
• ⇐

⇒ h

• U ′(ω), W (ω)
• = h
• U ′(ω′), W (ω′)
• ∀(ω, ω′) ∈ Ω × Ω .

13Recall that, in the π-field formalism, we have

U Y ⇐ ⇒

• Y (ω) = Y (ω′) ⇒ U(ω) = U(ω) ∀(ω, ω′) ∈ Ω × Ω
• .
• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 197 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

No Open-Loop Dual Effect

As a minimal requirement, we assume that the problem is such that all constant decision variables (open-loop controls) lead to the same information structure. We denote this set by ⊥U =

• U ∈ U , U(ω) = U(ω′)

∀(ω, ω′) ∈ Ω × Ω

• .

Of course we have: ⊥U ⊂ Uad. Definition There is No Open-Loop Dual Effect (NOLDE) for the stochastic system with observation function h : U × W → Y if we have π

• h(U, W )
• = π
• h(U ′, W )
• ∀(U, U ′) ∈ ⊥U × ⊥U .

Otherwise stated, the NOLDE property means that any mapping in the collection

• h(u, W )
• u∈U generates the same π-field.
• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 198 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

Characterization of Unde

Admissible set: Uad =

• U ∈ U , U h(U, W )
• .

We assume the NOLDE property, and we denote by ζ : Ω → Y the mapping such that ζ(ω) = h(u0, W (ω)) for a given u0 ∈ U. The no dual effect set Unde is given by Unde =

• U ∈ Uad , h(U, W ) ≡ ζ
• ,

We define the fixed information set Uζ by Uζ =

• U ∈ U , U ζ
• .

Theorem Under the NOLDE assumption, the set Unde is characterized by Unde = Uad ∩ Uζ .

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 199 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

Proof

1

Unde ⊂ Uad ∩ U ζ. Let U ∈ U nde.

By definition, U ∈ Uad, and U ∈ Uζ since h(U, W ) ≡ ζ.

2

Uad ∩ U ζ ⊂ U nde. Let U ∈ U ad ∩ U ζ.

Assume that h(U, W )(ω) = h(U, W )(ω′). Since U ∈ Uad, we have that U(ω) = U(ω′). Denoting by u this common value, we have h(u, W (ω)) = h(u, W (ω′)) and hence ζ(ω) = ζ(ω′). We deduce that ζ h(U, W ). Assume that ζ(ω) = ζ(ω′). Since U ∈ Uζ, we have that U(ω) = U(ω′). Denoting by u this common value, we have h(u, W (ω)) = h(u, W (ω′)) and h(U, W )(ω) = h(U, W )(ω′). We deduce that h(U, W ) ζ.

Ultimately, we obtain h(U, W ) ≡ ζ, that is, U ∈ U nde.

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 200 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

Examples

1 Let Ω = W = U = R, W = IdΩ and h(u, w) = w.

No open-loop dual effect holds, ζ being the identity mapping, and we have Uζ = U and Uad = U .

2 Let Ω = W = U = R, W = IdΩ and h(u, w) = u.

No open-loop dual effect holds, ζ being a constant mapping, and we have Uζ = ⊥U and Uad = U .

3 Let Ω = W = U = R, W = IdΩ and h(u, w) = u − w.

No open-loop dual effect holds, ζ being the identity mapping, and we have Uζ = U but Uad = U . Indeed, U0 = W is such that h

• U0, W
• = 0, hence U0 /

∈ Uad.

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 201 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

1

Closed Loop Stochastic Optimization Problems Stochastic Optimization Formulation Stochastic Optimal Control Problem Witsenhausen’s Counterexample

2

Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 202 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

Problem Statement (1)

We now consider a stochastic optimal control problem defined on {0, . . . , T}. Measurability is defined w.r.t. partition fields, so that the set Ω is equipped with a π-field G. The noise W = (W0, . . . , WT) : Ω → WT+1 is made up of the sequence of noises at each time t. The decision U = (U0, . . . , UT−1) : Ω → UT is made up of the sequence of decisions at each time t, and the set of decisions is denoted by UT. An information Yt is available at each time t, and is defined by a function ht : UT × WT+1 → Y: Yt = ht

• U, W
• .

The information constraints are gathered in the admissible set Uad =

• U ∈ UT , Ut Yt , t = 0, . . . , T −1
• .
• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 203 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

Problem Statement (2)

We denote by ⊥UT the set of open-loop controls: U ∈ ⊥UT if each decision variable Ut is a constant mapping. Assuming the NOLDE property at each t = 0 . . . T −1, we denote by ζt : Ω → Y the mapping such that ζt(ω) = ht(u0, W (ω)) for any given u0 ∈ U. The no dual effect set Unde is defined as Unde =

• U ∈ Uad , ht(U, W ) ≡ ζt , t = 0, . . . , T −1
• .

The fixed information set Uζ is defined as Uζ =

• U ∈ UT , Ut ζt , t = 0, . . . , T −1
• .

Question: does Unde = Uad ∩ Uζ still holds?

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 204 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

Precedence and Memory Relations

1 Precedence binary relation. We denote by [[t]] the smallest

set of time stages τ ∈ {0, . . . , T −1}, such that the mapping ht functionally depends on uτ, and we introduce the notation τ P t ⇐ ⇒ τ ∈ [[t]] . Otherwise stated, τ P t means that the decision variable at time τ influences the information variable at time t.

2 Memory binary relation. We denote by t the greatest set

• f time stages τ ∈ {0, . . . , T −1}, such that hτ ht (these

mappings are defined on UT × WT+1), and we introduce the notation τ M t ⇐ ⇒ τ ∈ t . Otherwise stated, τ M t means that the information available at time τ is remembered at time t.

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 205 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

A Meaningful Inclusion

Definition We say that the precedence binary relation P is included in the memory binary relation M if τ ∈ [[t]] = ⇒ τ ∈ t ∀(τ, t) ∈ {0, . . . , T −1}2 . We denote this property by P ⊂ M. It is equivalent to: [[t]] ⊂ t ∀t ∈ {0, . . . , T −1} . This property means that if a “agent” τ influences another “agent” t, then the information of “agent” τ is available to “agent” t. From an intuitive point of view, agent τ has no reason to influence agent t in order to transmit information because agent t already knows the information of agent τ: there is no need of dual effect.

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 206 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

Main Theorem

Theorem Let us assume that no open-loop dual effect holds true and that the precedence binary relation P is included in the memory binary relation M. Then Unde = Uad ∩ Uζ .

Sketch of proof.

1

Unde ⊂ Uad ∩ Uζ: obvious.

2

Uad ∩ Uζ ⊂ Unde: let U ∈ Uad ∩ Uζ. Assume ht

• U, W
• (ω) = ht
• U, W
• (ω′); then it also holds

true for any τ ∈ t, so that U(ω) = U(ω′) for any τ ∈ [[t]] and ζt(ω) = ζt(ω′); hence ζt ht

• U, W
• .

Assume ζt(ω) = ζt(ω′); . . . . . . hence ht

• U, W
• ζt.
• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 207 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

Application to a More Specific Problem (1)

We consider the causal Decision-Hazard information structure: Yt = ht(U0, . . . , Ut−1, W0, . . . , Wt) , and we moreover assume perfect memory: hτ ht ∀τ ≤ t . Then we have the two following properties:

1 [[t]] ⊂ {0, . . . , t − 1}, 2 t ⊃ {0, . . . , t},

so that the precedence relation is included in the memory relation. Assuming the NOLDE property, the last theorem applies.

• Remark. The perfect memory property, defined on UT × WT+1, means that

ht(u, w) = ht(u′, w ′) ⇒ hτ(u, w) = hτ(u′, w ′) ∀τ ≤ t . It is far stronger than some kind of “open-loop perfect memory”.

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 208 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

Application to a More Specific Problem (2)

In this specific case, we obtain the much better following result. Theorem Assume that both the NOLDE and the perfect memory properties hold true for the information structure under consideration. Then we have that Unde = Uad = Uζ .

Sketch of proof. The proof is done by induction on the two sets: Uad

t

=

• U ∈ UT, Uτ Yτ ∀τ ≤ t
• , Uζ

t =

• U ∈ UT, Uτ ζτ ∀τ ≤ t
• .

It is obvious that Uad = Uζ

0 .

Assuming that Uad

τ

= Uζ

τ

∀τ ≤ t, one can prove that Uad

t+1 = Uζ t+1.

Ultimately, we obtain Uad = Uad

T−1 = Uζ T−1 = Uζ.

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 209 / 267

Closed Loop Stochastic Optimization Problems Dual Effect in Stochastic Optimization Tools for Information Handling Dual Effect Free Stochastic Optimization Dual Effect for Stochastic Optimal Control Problems

Application to a More Specific Problem (3)

As a conclusion, for a rather popular information structure, and assuming both NOLDE and perfect memory properties, we have: (a) Uad = Unde. Every admissible decision variable belongs to the no dual effet set, so that there is no optimality loss to restrict the optimization process to Unde. (b) Uζ = Unde every decision variable mesurable w.r.t. ζ belongs to the no dual effet set, so that the original problem can be solved using a fixed information structure.

• Example. Consider the following additive information structure:

ht(u0, . . . , ut−1, w0, . . . , wt) = h1,t(u0, . . . , ut−1)+h2,t(w0, . . . , wt) . It always exhibits the NOLDE property. Assuming perfect memory, the previous theorem applies, so that we are in a good position to numerically solve the stochastic optimization problem.

• P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 210 / 267