Green Simulation Optimization Using Likelihood Ratio Estimators - PowerPoint PPT Presentation

Green Simulation Optimization Using Likelihood Ratio Estimators David J. Eckman M. Ben Feng Cornell University University of Waterloo Operations Research and Info Eng Statistics and Actuarial Science ❞❥❡✽✽❅❝♦r♥❡❧❧✳❡❞✉ ❜❡♥✳❢❡♥❣❅✉✇❛t❡r❧♦♦✳❝❛ Winter Simulation Conference December 11, 2018

G REEN S IMULATION O PTIMIZATION E CKMAN AND F ENG Model A simulation model h : Y �→ R maps a vectors of inputs, y , to a scalar output h ( y ) . The expected performance of a design x , is given by � µ ( x ) = E x [ h ( Y )] = h ( y ) f ( y ; x ) dy, Y where the random vector Y | x ∼ f ( · ; x ) . The design affects the simulation output only through the likelihood of the inputs. This model is not always suitable, but sometimes it is possible to “push out” any dependence of h ( · ) on x to f ( · ; x ) . G REEN S IMULATION G REEN S IMULATION O PTIMIZATION E XPERIMENTAL R ESULTS C ONCLUSIONS 2/15

G REEN S IMULATION O PTIMIZATION E CKMAN AND F ENG Examples Example: An M/D/ 1 queue with mean interarrival time x . • Y : vector of arrival times • h ( Y ) : associated average waiting time • µ ( x ) : expected waiting time G REEN S IMULATION G REEN S IMULATION O PTIMIZATION E XPERIMENTAL R ESULTS C ONCLUSIONS 3/15

G REEN S IMULATION O PTIMIZATION E CKMAN AND F ENG Examples Example: An M/D/ 1 queue with mean interarrival time x . • Y : vector of arrival times • h ( Y ) : associated average waiting time • µ ( x ) : expected waiting time Example: A stochastic activity network with mean task durations x i . • Y : vector of task lengths • h ( Y ) : associated project completion time • µ ( x ) : expected project completion time G REEN S IMULATION G REEN S IMULATION O PTIMIZATION E XPERIMENTAL R ESULTS C ONCLUSIONS 3/15

G REEN S IMULATION O PTIMIZATION E CKMAN AND F ENG Unbiased Estimators of µ ( x ) 1. Standard Monte Carlo Take r independent replications at design x and average the outputs: � Y ( j ) � r � ( x ) = 1 µ SMC � h . r r j =1 G REEN S IMULATION G REEN S IMULATION O PTIMIZATION E XPERIMENTAL R ESULTS C ONCLUSIONS 4/15

G REEN S IMULATION O PTIMIZATION E CKMAN AND F ENG Unbiased Estimators of µ ( x ) 1. Standard Monte Carlo Take r independent replications at design x and average the outputs: � Y ( j ) � r � ( x ) = 1 µ SMC � h . r r j =1 2. Likelihood Ratio Method (importance sampling) Take r independent replications at design � x � = x and average the weighted outputs: � � � Y ( j ) ; x � Y ( j ) � f r � ( x ) = 1 � µ LR � � � h . r r � f Y ( j ) ; � x j =1 � �� � likelihood ratio G REEN S IMULATION G REEN S IMULATION O PTIMIZATION E XPERIMENTAL R ESULTS C ONCLUSIONS 4/15

G REEN S IMULATION O PTIMIZATION E CKMAN AND F ENG Green Simulation Setting: Repeated experiments with a sequence of random designs X 1 , X 2 , . . . , X n − 1 , X n . ���� � �� � current past A design may represent exogenous conditions (e.g., economic, weather). Assumption The current design is independent of outputs of past designs, i.e., no feedback loop. Main idea: Reuse simulation outputs from past designs to estimate the expected performance of the current design. G REEN S IMULATION G REEN S IMULATION O PTIMIZATION E XPERIMENTAL R ESULTS C ONCLUSIONS 5/15

G REEN S IMULATION O PTIMIZATION E CKMAN AND F ENG Green Likelihood Ratio Estimators Suppose we have taken r independent replications from each design X 1 , . . . , X n . Green individual likelihood ratio (ILR) estimators for any point x ∈ X are given by � �   Y ( j ) � � f ; x � n � r n,r ( x ) = 1  1 k Y ( j )  , and µ ILR � � � h (objective function) k n r Y ( j ) f ; X k k =1 j =1 k � �   Y ( j ) � � f � � � n � r ; x n,r ( x ) = 1  1 ILR k Y ( j ) Y ( j ) �  , � � ∇ x log f ∇ µ h ; x (gradient) k k n r Y ( j ) f ; X k k =1 j =1 k where Y ( j ) are i.i.d. ∼ f ( · ; X k ) for all j = 1 , . . . , r and k = 1 , . . . , n . k Conditional on X 1 , . . . , X n , the green ILR estimators are unbiased . G REEN S IMULATION G REEN S IMULATION O PTIMIZATION E XPERIMENTAL R ESULTS C ONCLUSIONS 6/15

G REEN S IMULATION O PTIMIZATION E CKMAN AND F ENG Green Simulation Optimization Consider the optimization problem: min x ∈X µ ( x ) = E x [ h ( Y )] . A design now represents a vector of decision variables. An algorithm searches over the domain, X , visiting random designs X 1 , X 2 , . . . . • Uses estimates of the objective function and/or gradient at the current design X n to identify the next design X n +1 . • E.g., stochastic approximation, SPSA, and simulated annealing. Main idea: Use green simulation estimates of these quantities. ILR n,r ( X n ) and � µ ILR • I.e., � ∇ µ n,r ( X n ) . G REEN S IMULATION G REEN S IMULATION O PTIMIZATION E XPERIMENTAL R ESULTS C ONCLUSIONS 7/15

G REEN S IMULATION O PTIMIZATION E CKMAN AND F ENG Green Simulation Optimization Advantages: • Computationally cheap to reuse outputs in this way. • Recalculate the likelihood ratio and score each iteration. • The green ILR estimator may have a smaller variance than the SMC estimator. G REEN S IMULATION G REEN S IMULATION O PTIMIZATION E XPERIMENTAL R ESULTS C ONCLUSIONS 8/15

G REEN S IMULATION O PTIMIZATION E CKMAN AND F ENG Green Simulation Optimization Advantages: • Computationally cheap to reuse outputs in this way. • Recalculate the likelihood ratio and score each iteration. • The green ILR estimator may have a smaller variance than the SMC estimator. Complications: 1. Correlated estimators 2. Conditionally dependent outputs 3. Conditionally biased estimators “How do these issues manifest themselves in a search?” G REEN S IMULATION G REEN S IMULATION O PTIMIZATION E XPERIMENTAL R ESULTS C ONCLUSIONS 8/15

G REEN S IMULATION O PTIMIZATION E CKMAN AND F ENG Correlated Estimators Several forms of correlation: µ ILR µ ILR 1. The estimators � n,r ( X n ) and � n ′ ,r ( X n ′ ) contain similar terms. ILR ILR 2. The estimators � n,r ( X n ) and � ∇ µ ∇ µ n ′ ,r ( X n ′ ) contain similar terms. • Gradient-based search trajectories may be smoother . ILR n,r ( X n ) and � µ ILR 3. The estimators � ∇ µ n,r ( X n ) contain similar terms. G REEN S IMULATION G REEN S IMULATION O PTIMIZATION E XPERIMENTAL R ESULTS C ONCLUSIONS 9/15

G REEN S IMULATION O PTIMIZATION E CKMAN AND F ENG Conditionally Dependent Outputs In most simulation optimization algorithms, the current design is determined by the outputs of past designs. • The independence assumption of repeated experiments is violated. Example: Gradient-based searches (with or without green simulation). • Knowing the designs X n − 1 and X n reveals additional information about the outputs h ( Y (1) n − 1 ) , . . . , h ( Y ( r ) n − 1 ) used to estimate ∇ µ ( X n − 1 ) . Conditional on X n − 1 , h ( Y ( j ) n − 1 ) and X n are conditionally dependent . ) and h ( Y ( j ′ ) • Conditional on the visited designs X 1 , . . . X n , the outputs h ( Y ( j ) ) are k k conditionally dependent for all k < n and j � = j ′ . G REEN S IMULATION G REEN S IMULATION O PTIMIZATION E XPERIMENTAL R ESULTS C ONCLUSIONS 10/15

G REEN S IMULATION O PTIMIZATION E CKMAN AND F ENG Conditionally Biased Estimators From the conditional dependence of the outputs, � � µ ILR � n,r ( x ) | X 1 = x 1 , . . . , X n = x n � = µ ( x ) , and E � � ILR � ∇ µ n,r ( x ) | X 1 = x 1 , . . . , X n = x n � = ∇ µ ( x ) . E ILR n,r ( x ) and � µ ILR Conditional on the visited designs X 1 , . . . , X n , the estimators � ∇ µ n,r ( x ) are conditionally biased . Biased estimates of the objective function and gradient at the current design could adversely affect simulation optimization algorithms. G REEN S IMULATION G REEN S IMULATION O PTIMIZATION E XPERIMENTAL R ESULTS C ONCLUSIONS 11/15

G REEN S IMULATION O PTIMIZATION E CKMAN AND F ENG Example: Minimize a Quadratic Minimize µ ( x ) = E x [ h ( Y )] where Y | x ∼ N ( x, σ 2 ) and h ( y ) = y 2 . • µ ( x ) = σ 2 + x 2 with a global minimizer x ∗ = 0 . G REEN S IMULATION G REEN S IMULATION O PTIMIZATION E XPERIMENTAL R ESULTS C ONCLUSIONS 12/15

G REEN S IMULATION O PTIMIZATION E CKMAN AND F ENG Example: Minimize a Quadratic Minimize µ ( x ) = E x [ h ( Y )] where Y | x ∼ N ( x, σ 2 ) and h ( y ) = y 2 . • µ ( x ) = σ 2 + x 2 with a global minimizer x ∗ = 0 . Green ILR estimators are given by  � � � � 2 n r k − 2 Y ( j ) � � X 2 ( X k − x ) − x 2 n,r ( x ) = 1  1 Y ( j ) µ ILR  , and k � exp k 2 σ 2 n r k =1 j =1  � � � � � � 2 n r k − 2 Y ( j ) Y ( j ) � � X 2 ( X k − x ) − x 2 − x n,r ( x ) = 1  1 ILR Y ( j ) �  . k k ∇ µ exp k 2 σ 2 σ 2 n r k =1 j =1 ILR Ran 100 iterations of stochastic approximation with X n +1 = X n − 0 . 1 � ∇ µ n,r ( X n ) . G REEN S IMULATION G REEN S IMULATION O PTIMIZATION E XPERIMENTAL R ESULTS C ONCLUSIONS 12/15

G REEN S IMULATION O PTIMIZATION E CKMAN AND F ENG Search Trajectories 5 5 ILR Objective Estimate ILR Objective Estimate 4 4 3 3 2 2 1 1 0 0 -2 -1 0 1 2 -2 -1 0 1 2 Iterates Iterates Figure: * Figure: * r = 5 reps/design r = 50 reps/design G REEN S IMULATION G REEN S IMULATION O PTIMIZATION E XPERIMENTAL R ESULTS C ONCLUSIONS 13/15