[PPT] - Stochastic Gradient Method: Applications February 03, 2015 P. PowerPoint Presentation

SLIDE 1

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint

Stochastic Gradient Method: Applications

February 03, 2015

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 114 / 267

SLIDE 2

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint

Lecture Outline

1

Two Elementary Exercices on the Stochastic Gradient Two-Stage Recourse Problem Trade-off Between Investment and Operation

2

Option Pricing Problem and Variance Reduction Financial Problem Modeling Computing Efficiently the Price Two Algorithms

3

Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 115 / 267

SLIDE 3

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Two-Stage Recourse Problem Trade-off Between Investment and Operation

1

Two Elementary Exercices on the Stochastic Gradient Two-Stage Recourse Problem Trade-off Between Investment and Operation

2

Option Pricing Problem and Variance Reduction Financial Problem Modeling Computing Efficiently the Price Two Algorithms

3

Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 116 / 267

SLIDE 4

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Two-Stage Recourse Problem Trade-off Between Investment and Operation

1

Two Elementary Exercices on the Stochastic Gradient Two-Stage Recourse Problem Trade-off Between Investment and Operation

2

Option Pricing Problem and Variance Reduction Financial Problem Modeling Computing Efficiently the Price Two Algorithms

3

Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 117 / 267

SLIDE 5

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Two-Stage Recourse Problem Trade-off Between Investment and Operation

A Basic Two-Stage Recourse Problem

We consider the management of a water reservoir. Water is drawn from the reservoir by way of random consumers. In order to ensure the reservoir supply, 2 decisions are taken at successive time steps. A first supply decision q1 is taken without any knowledge of the effective consumption, the associated cost being equal to

1 2 c1

q1

2, with c1 > 0. Once the consumption d has been observed (realization of a r.v. D defined over a probability space (Ω, A, P)), a second supply decision q2 is taken in order to maintain the reservoir at its initial level, that is, q2 = d − q1, the cost associated to this second decision being equal to 1

2 c2

q2

2, with c2 > c1 > 0. The problem is to minimize the expected cost of operation.

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 118 / 267

SLIDE 6

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Two-Stage Recourse Problem Trade-off Between Investment and Operation

Mathematical Formulation and Solution

Problem Formulation q1 is a deterministic decision variable, whereas q2 is the realization of a random variable Q2. min

(q1,Q2)

1 2c1

q1

2 + 1 2E

c2
Q2

2 s.t. q1 + Q2 = D . Equivalent Problem min

q1∈R

1 2E

c1
q1

2 + c2

D − q1

2 Analytical solution: q♯

1 =

c2 c1 + c2 E

D
.
P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 119 / 267

SLIDE 7

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Two-Stage Recourse Problem Trade-off Between Investment and Operation

Stochastic Gradient Algorithm

Q(k+1)

1

= Q(k)

1

− 1 k

(c1 + c2)Q(k)

1

− c2D(k+1) . Algorithm (initialization)

// // Random generator // rand(’normal’); rand(’seed’,123); // // Random consumption // moy = 10.; ect = 5.; // // Criterion // c1 = 3.; c2 = 1.; // // Initialization // x = [ ]; y = [ ];

Algorithm (iterations)

// // Algorithm // qk = 0.; for k = 1:100 dk = moy + (ect*rand(1)); gk = ((c1+c2)*qk) - (c2*dk); ek = 1/k; qk = qk - (ek*gk); x = [x ; k]; y = [y ; qk]; end // // Trajectory plot // plot2d(x,y); xtitle(’Stochastic Gradient ’,’Iter.’,’Q1’);

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 120 / 267

SLIDE 8

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Two-Stage Recourse Problem Trade-off Between Investment and Operation

A Realization of the Algorithm

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 10 20 30 40 50 60 70 80 90 100 Stochastic Gradient Algorithm Iter. Q1

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 121 / 267

SLIDE 9

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Two-Stage Recourse Problem Trade-off Between Investment and Operation

More Realizations. . .

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 50 100 150 200 250 300 350 400 450 500 Stochastic Gradient Algorithm Iter. Q1

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 122 / 267

SLIDE 10

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Two-Stage Recourse Problem Trade-off Between Investment and Operation

Slight Modification of the problem

As in the basic two-stage recourse problem, a first supply decision q1 is taken without any knowledge of the effective consumption, the associated cost being equal to

1 2 c1

q1

2, a second supply decision q2 is taken once the consumption d has been observed (realization of a r.v. D), the cost of this second decision being equal to 1

2 c2

q2

2. The difference between supply and demand is penalized thanks to an additional cost term 1

2 c3

q1 + q2 − d
2. The new problem is :

min

(q1,Q2)

1 2E

c1
q1

2 + c2

Q2

2 + c3

q1 + Q2 − D

2 . Question: how to solve it using a stochastic gradient algorithm?

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 123 / 267

SLIDE 11

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Two-Stage Recourse Problem Trade-off Between Investment and Operation

1

Two Elementary Exercices on the Stochastic Gradient Two-Stage Recourse Problem Trade-off Between Investment and Operation

2

Option Pricing Problem and Variance Reduction Financial Problem Modeling Computing Efficiently the Price Two Algorithms

3

Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 124 / 267

SLIDE 12

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Two-Stage Recourse Problem Trade-off Between Investment and Operation

Trade-off Between Investment and Operation (1)

A company owns N production units and has to meet a given (non stochastic) demand d. For each unit i, the decision maker first takes an investment decision ui ∈ R, the associated cost being Ii(ui). Then a discrete disturbance wi ∈ {wi,a, wi,b, wi,c} occurs. Knowing all noises, the decision maker selects for each unit i an operating point vi ∈ R, which leads to a cost ci(vi, wi) and a production ei(vi, wi). The goal is to minimize the overall expected cost, subject to the following constraints: investment limitation: Θ(u1, . . . , uN) ≤ 0,

perating limitation: vi ≤ ϕi(ui) , i = 1 . . . , N,

demand satisfaction: N

i=1 ei(vi, wi) − d = 0.

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 125 / 267

SLIDE 13

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Two-Stage Recourse Problem Trade-off Between Investment and Operation

Trade-off Between Investment and Operation (2)

Questions.

1 Write down the optimization problem. Is it possible to apply

the stochastic gradient algorithm in a straightforward manner?

2 Extract the optimization subproblem obtained when both the

investment u = (u1, . . . , uN) and the noise w = (w1, . . . , wN) are fixed. The value of this subproblem is denoted f ♯(u, w).

Discuss the resolution of this subproblem. Give assumptions in order to have a “nice” function f ♯. Compute the partial derivatives of f ♯ w.r.t. u.

3 Reformulate the initial optimization problem using this new

function f ♯ and apply the stochastic gradient algorithm in the two following cases:

the investment limitation reduces to ui ∈ [ui, ui], i = 1, . . . , N, the investment limitation has the form Θ(u1, . . . , uN) ≤ 0.

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 126 / 267

SLIDE 14

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Financial Problem Modeling Computing Efficiently the Price Two Algorithms

1

Two Elementary Exercices on the Stochastic Gradient Two-Stage Recourse Problem Trade-off Between Investment and Operation

2

Option Pricing Problem and Variance Reduction Financial Problem Modeling Computing Efficiently the Price Two Algorithms

3

Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 127 / 267

SLIDE 15

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Financial Problem Modeling Computing Efficiently the Price Two Algorithms

1

Two Elementary Exercices on the Stochastic Gradient Two-Stage Recourse Problem Trade-off Between Investment and Operation

2

Option Pricing Problem and Variance Reduction Financial Problem Modeling Computing Efficiently the Price Two Algorithms

3

Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 128 / 267

SLIDE 16

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Financial Problem Modeling Computing Efficiently the Price Two Algorithms

The Financial Problem

The price of an option with payoff ψ(St, 0 ≤ t ≤ T) is given by P = E

e−rT ψ(St, 0 ≤ t ≤ T)
,

where the dynamics of the underlying n-dimensional asset S is described by the following stochastic differential equation dSt = St

rdt + σ(t, St)dWt
,

S0 = x , r being the interest rate and σ(t, y) being the volatility function.

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 129 / 267

SLIDE 17

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Financial Problem Modeling Computing Efficiently the Price Two Algorithms

Discretization

Most of the time, the exact solution is not available. To overcome the difficulty, one considers a discretized approximation of S (by using an Euler’s scheme), so that the price P is approximated by

P = E
e−rT ψ(

S t1, . . . , S td)

.

In such cases, the discretized function can be expressed in terms of the Brownian increments, or equivalently using a random normal

vector. A compact form for the discretized price is
P = E
φ(G )
,

where G is a n × d-dimensional Gaussian vector with identity covariance matrix and zero-mean.

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 130 / 267

SLIDE 18

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Financial Problem Modeling Computing Efficiently the Price Two Algorithms

1

Two Elementary Exercices on the Stochastic Gradient Two-Stage Recourse Problem Trade-off Between Investment and Operation

2

Option Pricing Problem and Variance Reduction Financial Problem Modeling Computing Efficiently the Price Two Algorithms

3

Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 131 / 267

SLIDE 19

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Financial Problem Modeling Computing Efficiently the Price Two Algorithms

A Parameterized Change of Variable

The problem is now to compute P = E

φ(G )
using Monte Carlo
simulations. By a change of variables, we obtain that
P = E
φ(G + θ)e−θ ,G − θ2

2

,

for any θ ∈ Rn×d. Let us denote by V (θ) the associated variance:

V (θ) = E
φ(G + θ)2e−2θ ,G −θ2

− E

φ(G )

2 , = E

φ(G )2e−θ ,G + θ2

2

− E
φ(G )

2 . The last expression shows that function V is strictly convex and differentiable without any specific assumptions on φ. Moreover, ∇ V (θ) = E

(θ − G )φ(G )2e−θ ,G + θ2

2

,

so that the gradient of V does not involves any derivative of φ.

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 132 / 267

SLIDE 20

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Financial Problem Modeling Computing Efficiently the Price Two Algorithms

Variance Minimization

The goal is to compute the parameter θ such that the variance

V (θ) related to

P is as small as possible, in order to optimize the convergence speed of the Monte Carlo simulations: min

θ∈Rd E

φ(G )2e−θ ,G + θ2

2

.

This optimization problem is well suited for being solved by the stochastic gradient algorithm: θ(k+1) = θ(k)−ǫ(k) θ(k)−G (k+1) φ

G (k+1)2e−θ(k) ,G(k+1)+ θ(k)2

2

, and its unique solution is denoted θ♯.

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 133 / 267

SLIDE 21

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Financial Problem Modeling Computing Efficiently the Price Two Algorithms

1

Two Elementary Exercices on the Stochastic Gradient Two-Stage Recourse Problem Trade-off Between Investment and Operation

2

Option Pricing Problem and Variance Reduction Financial Problem Modeling Computing Efficiently the Price Two Algorithms

3

Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 134 / 267

SLIDE 22

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Financial Problem Modeling Computing Efficiently the Price Two Algorithms

Non adaptive Algorithm

1 Using a N-sample of G , obtain an approximation θ(N) of θ♯

by N iterations of the stochastic gradient algorithm.

2 Once θ(N) has been obtained, use the standard Monte Carlo

method to compute an approximation of the price P by using another N-sample of G :

P

(N) = 1

N

k=1

φ(G (N+k) + θ(N))e−θ(N) ,G(N+k)− θ(N)2

2

. This algorithm requires 2N evaluations of the function φ, whereas a crude Monte Carlo method evaluates φ only N times. The non adaptive algorithm is efficient as soon as V (θ♯) ≤ V (0)/2.

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 135 / 267

SLIDE 23

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Financial Problem Modeling Computing Efficiently the Price Two Algorithms

Adaptive Algorithm

Combine the 2 previous algorithms, and compute simultaneously approximations of θ♯ and P by using the same N-sample of G :10

θ(k+1) = θ(k)− ǫ(k) θ(k)− G (k+1) φ

G (k+1)2e−θ(k) ,G(k+1)+

θ(k)2 2

,

P

(k+1) =

P

(k)−

1 k + 1

P(k)− φ(G (k+1)+ θ(k)) e−θ(k) ,G(k+1)−

θ(k)2 2

.

A Central Limit Theorem is available for this algorithm: √ N

P

(N) −

P

D

− → N

0,

V (θ♯)

.

10the last relation being the recursive form of:

P

(N) = 1

N

k=1

φ(G (k+1)+ θ(k)) e−θ(k) ,G(k+1)− θ(k)2

2

.

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 136 / 267

SLIDE 24

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

1

Two Elementary Exercices on the Stochastic Gradient Two-Stage Recourse Problem Trade-off Between Investment and Operation

2

Option Pricing Problem and Variance Reduction Financial Problem Modeling Computing Efficiently the Price Two Algorithms

3

Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 137 / 267

SLIDE 25

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

1

Two Elementary Exercices on the Stochastic Gradient Two-Stage Recourse Problem Trade-off Between Investment and Operation

2

Option Pricing Problem and Variance Reduction Financial Problem Modeling Computing Efficiently the Price Two Algorithms

3

Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 138 / 267

SLIDE 26

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

Mission to Mars

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 139 / 267

SLIDE 27

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

Satellite Model

dr dt = v , dv dt = −µ r r3 + F mκ , (8a) dm dt = − T g0Isp δ . (8b) (8a): 6-dimensional state vector (position r and velocity v). (8b): 1-dimensional state vector (mass m including fuel). κ involves the direction cosines of the thrust and the on-off switch δ of the engine (3 controls), and µ, F, T, g0, Isp are constants. The deterministic control problem is to drive the satellite from the initial condition at ti to a known final position rf and velocity vf at tf (given) while minimizing fuel consumption m(ti) − m(tf).

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 140 / 267

SLIDE 28

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

Deterministic Optimization Problem

Using equinoctial coordinates for the position and velocity ❀ state vector x ∈ R7, and cartesian coordinates for the thrust of the engine ❀ control vector u ∈ R3, the deterministic optimization problem writes as follows: min

u(·) K

x(tf)
subject to:

x(ti) = xi ,

x (t) = f
x(t), u(t)
,

u(t) ≤ 1 ∀t ∈ [ti, tf] , C

x(tf)
= 0 .
P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 141 / 267

SLIDE 29

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

Engine Failure

Sometimes, the engine may fail to work when needed: the satellite drifts away from the deterministic optimal trajectory. After the engine control is recovered, it is not always possible to drive the satellite to the final target at tf. By anticipating such possible failures and by modifying the trajectory followed before any such failure occurs, one may increase the possibility of eventually reaching the target. But such a deviation from the deterministic optimal trajectory results in a deterioration of the economic performance. The problem is thus to balance the increased probability of eventually reaching the target despite possible failures against the expected economic performance, that is, to quantify the price of safety one is ready to pay for.

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 142 / 267

SLIDE 30

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

Stochastic Formulation (1)

A failure is modeled using two random variables: tp : random initial time of the failure, td : random duration of the failure. For any realization (tξ

p, tξ d) of a failure:

1 u(·) denotes the control used prior to the failure

❀ u is defined over [ti, tf] but implemented over [ti, tξ

p]

and corresponds to an open-loop control,

2 the control during the failure is 0 over [tξ

p, tξ p + tξ d],

3 vξ(·) denotes the control used after the failure

❀ vξ is defined over [tξ

p + tξ d, tf] (if nonempty)

and corresponds to a closed-loop strategy V. The satellite dynamics in the stochastic formulation writes: xξ(ti) = xi ,

x ξ(t) = f ξ

xξ(t), u(t), vξ(t)

.
P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 143 / 267

SLIDE 31

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

Stochastic Formulation (2)

The problem is to minimize the expected cost (fuel consumption) w.r.t. the open-loop control u and the closed-loop strategy V, the probability to hit the target at tf being at least equal to p. min

u(·) min V(·) E

K
xξ(tf)
min

u(·) min V(·) E

K
xξ(tf)
C
xξ(tf)
= 0
subject to:

xξ(ti) = xi ,

x ξ(t) = f ξ

xξ(t), u(t), vξ(t)

,

u(t) ≤ 1 ∀t ∈ [ti, tf] , vξ(t) ≤ 1 ∀t ∈ [tξ

p + tξ d, tf] ,

P

C
xξ(tf)
= 0
≥ p .
P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 144 / 267

SLIDE 32

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

1

Two Elementary Exercices on the Stochastic Gradient Two-Stage Recourse Problem Trade-off Between Investment and Operation

2

Option Pricing Problem and Variance Reduction Financial Problem Modeling Computing Efficiently the Price Two Algorithms

3

Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 145 / 267

SLIDE 33

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

Indicator Function

Consider the real-valued indicator function: I(y) =

1

if y = 0,

therwise.

Then P

C
xξ(tf)
= 0
= E
I
C
xξ(tf)
,

and E

K
xξ(tf)
C
xξ(tf)
= 0
=

E

K
xξ(tf)
× I
C
xξ(tf)
E
I
C
xξ(tf)
.
P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 146 / 267

SLIDE 34

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

Problem Reformulation

The problem is (shortly) reformulated as min

u(·) min V(·)

E

K
xξ(tf)
× I
C
xξ(tf)
E
I
C
xξ(tf)
s.t.

E

I
C
xξ(tf)
≥ p .

Such a formulation is however not well-suited for a numerical implementation (e.g. Arrow-Hurwicz algorithm), because a ratio of expectations is not an expectation!

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 147 / 267

SLIDE 35

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

An Useful Lemma

Using compact notation, the previous problem writes: min

u

J(u) Θ(u) s.t. Θ(u) ≥ p , (9) in which J and Θ assume positive values.

1

If u♯ is a solution of (9) and if Θ(u♯) = p, then u♯ is also a solution of min

u

J(u) s.t. Θ(u) ≥ p . (10)

2

Conversely, if u♯ is a solution of (10), and if an optimal Kuhn-Tucker multiplier β♯ satisfies the condition β♯ ≥ J(u♯) Θ(u♯) , then u♯ is also a solution of (9).

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 148 / 267

SLIDE 36

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

Back to a Cost in Expectation

Using the previous lemma, we aim at solving a problem in which the cost and the constraint functions correspond to expectations: min

u(·) min V(·) E

K
xξ(tf)
× I
C
xξ(tf)
s.t.

E

I
C
xξ(tf)
≥ p ,
r equivalently (Interchange Theorem [R&W, 1998]):

min

u(·) E

min

vξ(·) K

xξ(tf)
× I
C
xξ(tf)
s.t.

E

I
C
xξ(tf)
≥ p .
P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 149 / 267

SLIDE 37

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

1

Two Elementary Exercices on the Stochastic Gradient Two-Stage Recourse Problem Trade-off Between Investment and Operation

2

Option Pricing Problem and Variance Reduction Financial Problem Modeling Computing Efficiently the Price Two Algorithms

3

Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 150 / 267

SLIDE 38

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

Lagrangian Formulation

min

u(·) E

min

vξ(·) K

xξ(tf)
× I
C
xξ(tf)
s.t.

p − E

I
C
xξ(tf)
≤ 0
µ

Assume there exists a saddle point for the associated Lagrangian. In order to solve max

µ≥0

min

u(·)

µ p + E
min

vξ(·)

K
xξ(tf)
− µ
× I
C
xξ(tf)
W (u, µ, ξ)
.

that is, max

µ≥0

min

u(·)

µ p + E
W (u, µ, ξ)
,

we use an adapted Arrow-Hurwicz algorithm ([Culioli, 1994]).

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 151 / 267

SLIDE 39

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

Algorithm Overview

Arrow-Hurwicz algorithm At iteration k,

1 draw a failure ξk = (tξk

p , tξk d ) according to its probability law,

2 compute the gradient of W w.r.t. u and update u(·):

uk+1 = ΠB

uk − εk ∇uW (uk, µk, ξk)
,

3 compute the gradient of W w.r.t. µ and update µ:

µk+1 = max

0, µk + ρk

p + ∇µW (uk+1, µk, ξk)

.
P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 152 / 267

SLIDE 40

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

First Difficulty: I Is Not a Smooth Function

At every iteration k, we must evaluate function W as well as its derivatives w.r.t. u(.) and µ. But W is not differentiable! I(y) =

1

if y = 0,

therwise, Ir(y) =

  

1 − y2

r2

2 if y ∈ [−r, r],

therwise.

0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0

There are specific rules to drive r to 0 as the iteration number k goes to infinity in order to obtain the best asymptotic Mean Quadratic Error of the gradient estimates ([Andrieu et al., 2007]).

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 153 / 267

SLIDE 41

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

Second Difficulty: Solving the Inner Problem

The approximated closed-loop problem to solve at each iteration is: Wrk(uk, ξk, µk) = min

vξ(·)

K
xξ(tf)
− µk

× Irk

C
xξ(tf)
.

In this setting, we have to check if the target is reached up to rk. Different cases have to be considered:

1 the target can be reached accurately, 2 the target can be reached up to rk only, 3 the target cannot be reached up to rk.

Note that if reaching the target is possible but too expensive (that is, if K

xξ(tf)
≥ µk), the best thing to do is to stop the engine!

In practice, the solution of the approximated problem is derived from the resolution of two standard optimal control problems. . .

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 154 / 267

SLIDE 42

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

Parameters Tuning

Gradient step length: εk = a b + k , ρk = c d + k , usual for a stochastic gradient algorithm. Smoothing parameter: rk = α β + k

1 3

, MQE reduced by a factor 1000 in about 100.000 iterations.

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 155 / 267

SLIDE 43

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

1

Two Elementary Exercices on the Stochastic Gradient Two-Stage Recourse Problem Trade-off Between Investment and Operation

2

Option Pricing Problem and Variance Reduction Financial Problem Modeling Computing Efficiently the Price Two Algorithms

3

Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 156 / 267

SLIDE 44

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

Example: Interplanetary Mission

ti = 0.70 and tf = 8.70 (normalized units), tp: exponential distribution: P

tp ≥ tf
≈ 0.58 = πf,

td: exponential distribution: P

0.035 ≤ td ≤ 0.125
≈ 0.80.
Comp. normale w
Comp. tangentielle s
Comp. radiale q

1 2 3 4 5 6 7 8 9 −0.4 −0.2 0.0 0.2 0.4 0.6 0.8 1.0

The deterministic optimal control has a “bang–off–bang” shape. Along the optimal trajectory, the probability to recover a failure is: pdet ≈ 0.94.

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 157 / 267

SLIDE 45

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

Masse finale / iterations 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000 0.674 0.675 0.676 0.677 0.678 0.679 0.680 0.681 Multiplicateur probabilite / iterations 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000 0.314 0.316 0.318 0.320 0.322 0.324 0.326

Comp. normale w
Comp. tangentielle s
Comp. radiale q

1 2 3 4 5 6 7 8 9 −0.4 −0.2 0.0 0.2 0.4 0.6 0.8 1.0

Figure: Probability level p = 0.750

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 158 / 267

SLIDE 46

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

Masse finale / iterations 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000 0.673 0.674 0.675 0.676 0.677 0.678 0.679 0.680 0.681 Multiplicateur probabilite / iterations 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000 0.318 0.320 0.322 0.324 0.326 0.328 0.330 0.332

Comp. normale w
Comp. tangentielle s
Comp. radiale q

1 2 3 4 5 6 7 8 9 −0.4 −0.2 0.0 0.2 0.4 0.6 0.8 1.0

Figure: Probability level p = 0.960

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 159 / 267

SLIDE 47

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

Masse finale / iterations 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000 0.674 0.675 0.676 0.677 0.678 0.679 0.680 0.681 Multiplicateur probabilite / iterations 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000 0.320 0.325 0.330 0.335 0.340 0.345 0.350 0.355 0.360 0.365

Comp. normale w
Comp. tangentielle s
Comp. radiale q

1 2 3 4 5 6 7 8 9 −0.4 −0.2 0.0 0.2 0.4 0.6 0.8 1.0

Figure: Probability level p = 0.990

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 160 / 267

SLIDE 48

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

The Price of Safety. . .

Consommation sans panne / Probabilite 0.85 0.90 0.95 1.00 0.3204 0.3206 0.3208 0.3210 0.3212 0.3214 0.3216 0.3218 0.3220 0.3222

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 161 / 267

SLIDE 49

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Satellite Model and Optimization Problem Probability and Conditional Expectation Handling Stochastic Arrow-Hurwicz Algorithm Numerical Results

H. Robbins and S. Monro.

A stochastic approximation method. Annals of Mathematical Statistics, 22:400–407, 1951.

B. T. Polyak and A. B. Juditsky.

Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838–855, 1992.

M. Duflo.

Algorithmes stochastiques. Springer Verlag, 1996.

H. J. Kushner and D. S. Clark.

Stochastic Approximation Methods for Constrained and Unconstrained Systems. Springer Verlag, 1978. J.-C. Culioli and G. Cohen. Decomposition-coordination algorithms in stochastic optimization. SIAM Journal on Control and Optimization, 28(6):1372–1403, 1990. J.-C. Culioli and G. Cohen. Optimisation stochastique sous contraintes en esp´ erance. CRAS, t.320, S´ erie I, pp. 75-758, 1995. Laetitia Andrieu, Guy Cohen and Felisa J. V´ azquez-Abad. Gradient-based simulation optimization under probability constraints. European Journal of Operational Research, 212, 345-351, 2011.

R. T. Rockafellar and R. J.-B. Wets.

Variational Analysis, volume 317 of A series of Comprehensive Studies in Mathematics. Springer Verlag, 1998.

P. Carpentier

Master MMMEF — Cours MNOS 2014-2015 161 / 267