Green Simulation Optimization Using Likelihood Ratio Estimators - - PowerPoint PPT Presentation

Green Simulation Optimization Using Likelihood Ratio Estimators

David J. Eckman

• M. Ben Feng

Cornell University University of Waterloo Operations Research and Info Eng Statistics and Actuarial Science ❞❥❡✽✽❅❝♦r♥❡❧❧✳❡❞✉ ❜❡♥✳❢❡♥❣❅✉✇❛t❡r❧♦♦✳❝❛

Winter Simulation Conference December 11, 2018

GREEN SIMULATION OPTIMIZATION ECKMAN AND FENG

Model

A simulation model h : Y → R maps a vectors of inputs, y, to a scalar output h(y). The expected performance of a design x, is given by µ(x) = Ex[h(Y )] =

• Y

h(y)f(y; x) dy, where the random vector Y |x ∼ f(·; x). The design affects the simulation output only through the likelihood of the inputs. This model is not always suitable, but sometimes it is possible to “push out” any dependence of h(·) on x to f(·; x).

GREEN SIMULATION GREEN SIMULATION OPTIMIZATION EXPERIMENTAL RESULTS CONCLUSIONS 2/15

GREEN SIMULATION OPTIMIZATION ECKMAN AND FENG

Examples

Example: An M/D/1 queue with mean interarrival time x.

• Y : vector of arrival times
• h(Y ): associated average waiting time
• µ(x): expected waiting time

GREEN SIMULATION GREEN SIMULATION OPTIMIZATION EXPERIMENTAL RESULTS CONCLUSIONS 3/15

GREEN SIMULATION OPTIMIZATION ECKMAN AND FENG

Examples

Example: An M/D/1 queue with mean interarrival time x.

• Y : vector of arrival times
• h(Y ): associated average waiting time
• µ(x): expected waiting time

Example: A stochastic activity network with mean task durations xi.

• Y : vector of task lengths
• h(Y ): associated project completion time
• µ(x): expected project completion time

GREEN SIMULATION GREEN SIMULATION OPTIMIZATION EXPERIMENTAL RESULTS CONCLUSIONS 3/15

GREEN SIMULATION OPTIMIZATION ECKMAN AND FENG

Unbiased Estimators of µ(x)

• 1. Standard Monte Carlo

Take r independent replications at design x and average the outputs:

• µSMC

r

(x) = 1 r

r

• j=1

h

• Y (j)

.

GREEN SIMULATION GREEN SIMULATION OPTIMIZATION EXPERIMENTAL RESULTS CONCLUSIONS 4/15

GREEN SIMULATION OPTIMIZATION ECKMAN AND FENG

Unbiased Estimators of µ(x)

• 1. Standard Monte Carlo

Take r independent replications at design x and average the outputs:

• µSMC

r

(x) = 1 r

r

• j=1

h

• Y (j)

.

• 2. Likelihood Ratio Method (importance sampling)

Take r independent replications at design x = x and average the weighted outputs:

• µLR

r

(x) = 1 r

r

• j=1

h

• Y (j) f
• Y (j); x
• f
• Y (j);

x

• likelihood ratio

.

GREEN SIMULATION GREEN SIMULATION OPTIMIZATION EXPERIMENTAL RESULTS CONCLUSIONS 4/15

GREEN SIMULATION OPTIMIZATION ECKMAN AND FENG

Green Simulation

Setting: Repeated experiments with a sequence of random designs X1, X2, . . . , Xn−1

• past

, Xn

• current

. A design may represent exogenous conditions (e.g., economic, weather).

Assumption

The current design is independent of outputs of past designs, i.e., no feedback loop. Main idea: Reuse simulation outputs from past designs to estimate the expected performance of the current design.

GREEN SIMULATION GREEN SIMULATION OPTIMIZATION EXPERIMENTAL RESULTS CONCLUSIONS 5/15

GREEN SIMULATION OPTIMIZATION ECKMAN AND FENG

Green Likelihood Ratio Estimators

Suppose we have taken r independent replications from each design X1, . . . , Xn. Green individual likelihood ratio (ILR) estimators for any point x ∈ X are given by

• µILR

n,r (x) = 1

n

n

• k=1

 1 r

r

• j=1

h

• Y (j)

k

f

• Y (j)

k

; x

• f
• Y (j)

k

; Xk

• 

 , and (objective function)

• ∇µ

ILR n,r (x) = 1

n

n

• k=1

 1 r

r

• j=1

h

• Y (j)

k

f

• Y (j)

k

; x

• f
• Y (j)

k

; Xk ∇x log f

• Y (j)

k

; x

• 

 , (gradient) where Y (j)

k

are i.i.d. ∼ f(·; Xk) for all j = 1, . . . , r and k = 1, . . . , n. Conditional on X1, . . . , Xn, the green ILR estimators are unbiased.

GREEN SIMULATION GREEN SIMULATION OPTIMIZATION EXPERIMENTAL RESULTS CONCLUSIONS 6/15

GREEN SIMULATION OPTIMIZATION ECKMAN AND FENG

Green Simulation Optimization

Consider the optimization problem: min

x∈X µ(x) = Ex[h(Y )].

A design now represents a vector of decision variables. An algorithm searches over the domain, X, visiting random designs X1, X2, . . ..

• Uses estimates of the objective function and/or gradient at the current design Xn to

identify the next design Xn+1.

• E.g., stochastic approximation, SPSA, and simulated annealing.

Main idea: Use green simulation estimates of these quantities.

• I.e.,

µILR

n,r (Xn) and

∇µ

ILR n,r (Xn).

GREEN SIMULATION GREEN SIMULATION OPTIMIZATION EXPERIMENTAL RESULTS CONCLUSIONS 7/15

GREEN SIMULATION OPTIMIZATION ECKMAN AND FENG

Green Simulation Optimization

Advantages:

• Computationally cheap to reuse outputs in this way.
• Recalculate the likelihood ratio and score each iteration.
• The green ILR estimator may have a smaller variance than the SMC estimator.

GREEN SIMULATION GREEN SIMULATION OPTIMIZATION EXPERIMENTAL RESULTS CONCLUSIONS 8/15

GREEN SIMULATION OPTIMIZATION ECKMAN AND FENG

Green Simulation Optimization

Advantages:

• Computationally cheap to reuse outputs in this way.
• Recalculate the likelihood ratio and score each iteration.
• The green ILR estimator may have a smaller variance than the SMC estimator.

Complications:

• 1. Correlated estimators
• 2. Conditionally dependent outputs
• 3. Conditionally biased estimators

“How do these issues manifest themselves in a search?”

GREEN SIMULATION GREEN SIMULATION OPTIMIZATION EXPERIMENTAL RESULTS CONCLUSIONS 8/15

GREEN SIMULATION OPTIMIZATION ECKMAN AND FENG

Correlated Estimators

Several forms of correlation:

• 1. The estimators

µILR

n,r (Xn) and

µILR

n′,r (Xn′) contain similar terms.

• 2. The estimators

∇µ

ILR n,r (Xn) and

∇µ

ILR n′,r (Xn′) contain similar terms.

• Gradient-based search trajectories may be smoother.
• 3. The estimators

µILR

n,r (Xn) and

∇µ

ILR n,r (Xn) contain similar terms.

GREEN SIMULATION GREEN SIMULATION OPTIMIZATION EXPERIMENTAL RESULTS CONCLUSIONS 9/15

GREEN SIMULATION OPTIMIZATION ECKMAN AND FENG

Conditionally Dependent Outputs

In most simulation optimization algorithms, the current design is determined by the

• utputs of past designs.
• The independence assumption of repeated experiments is violated.

Example: Gradient-based searches (with or without green simulation).

• Knowing the designs Xn−1 and Xn reveals additional information about the outputs

h(Y (1)

n−1), . . . , h(Y (r) n−1) used to estimate ∇µ(Xn−1).

Conditional on Xn−1, h(Y (j)

n−1) and Xn are conditionally dependent.

• Conditional on the visited designs X1, . . . Xn, the outputs h(Y (j)

k

) and h(Y (j′)

k

) are conditionally dependent for all k < n and j = j′.

GREEN SIMULATION GREEN SIMULATION OPTIMIZATION EXPERIMENTAL RESULTS CONCLUSIONS 10/15

GREEN SIMULATION OPTIMIZATION ECKMAN AND FENG

Conditionally Biased Estimators

From the conditional dependence of the outputs, E

• µILR

n,r (x) | X1 = x1, . . . , Xn = xn

• = µ(x), and

E

• ∇µ

ILR n,r (x) | X1 = x1, . . . , Xn = xn

• = ∇µ(x).

Conditional on the visited designs X1, . . . , Xn, the estimators µILR

n,r (x) and

∇µ

ILR n,r (x) are

conditionally biased. Biased estimates of the objective function and gradient at the current design could adversely affect simulation optimization algorithms.

GREEN SIMULATION GREEN SIMULATION OPTIMIZATION EXPERIMENTAL RESULTS CONCLUSIONS 11/15

GREEN SIMULATION OPTIMIZATION ECKMAN AND FENG

Example: Minimize a Quadratic

Minimize µ(x) = Ex[h(Y )] where Y |x ∼ N(x, σ2) and h(y) = y2.

• µ(x) = σ2 + x2 with a global minimizer x∗ = 0.

GREEN SIMULATION GREEN SIMULATION OPTIMIZATION EXPERIMENTAL RESULTS CONCLUSIONS 12/15

GREEN SIMULATION OPTIMIZATION ECKMAN AND FENG

Example: Minimize a Quadratic

Minimize µ(x) = Ex[h(Y )] where Y |x ∼ N(x, σ2) and h(y) = y2.

• µ(x) = σ2 + x2 with a global minimizer x∗ = 0.

Green ILR estimators are given by

• µILR

n,r (x) = 1

n

n

• k=1

 1 r

r

• j=1
• Y (j)

k

2 exp

• X2

k − 2Y (j) k

(Xk − x) − x2 2σ2   , and

• ∇µ

ILR n,r (x) = 1

n

n

• k=1

 1 r

r

• j=1
• Y (j)

k

2 exp

• X2

k − 2Y (j) k

(Xk − x) − x2 2σ2 Y (j)

k

− x σ2   . Ran 100 iterations of stochastic approximation with Xn+1 = Xn − 0.1 ∇µ

ILR n,r (Xn).

GREEN SIMULATION GREEN SIMULATION OPTIMIZATION EXPERIMENTAL RESULTS CONCLUSIONS 12/15

GREEN SIMULATION OPTIMIZATION ECKMAN AND FENG

Search Trajectories

• 2
• 1

1 2 Iterates 1 2 3 4 5 ILR Objective Estimate

Figure: *

r = 5 reps/design

• 2
• 1

1 2 Iterates 1 2 3 4 5 ILR Objective Estimate

Figure: *

r = 50 reps/design

GREEN SIMULATION GREEN SIMULATION OPTIMIZATION EXPERIMENTAL RESULTS CONCLUSIONS 13/15

GREEN SIMULATION OPTIMIZATION ECKMAN AND FENG

Conditional Biases

20 40 60 80 100 Iteration Number

• 0.3
• 0.2
• 0.1

Bias of ILR Objective Estimate

r=5 r=50

Figure: *

Bias of Objective Function Estimator

20 40 60 80 100 Iteration Number

• 0.2

0.2 0.4 0.6 Bias of ILR Gradient Estimate

r=5 r=50

Figure: *

Bias of Gradient Estimator

GREEN SIMULATION GREEN SIMULATION OPTIMIZATION EXPERIMENTAL RESULTS CONCLUSIONS 14/15

GREEN SIMULATION OPTIMIZATION ECKMAN AND FENG

Ongoing Work

Are these issues of practical significance? Can the use of green simulation estimators cause a simulation optimization algorithm to fail to converge? Under what conditions are green simulation estimators asymptotically unbiased and consistent as r → ∞ or n → ∞?

GREEN SIMULATION GREEN SIMULATION OPTIMIZATION EXPERIMENTAL RESULTS CONCLUSIONS 15/15

GREEN SIMULATION OPTIMIZATION ECKMAN AND FENG

Acknowledgments

This material is based upon work supported by the National Science Foundation under grant DGE-1650441. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

GREEN SIMULATION GREEN SIMULATION OPTIMIZATION EXPERIMENTAL RESULTS CONCLUSIONS 15/15