effective scenarios in multistage distributionally robust
play

Effective Scenarios in Multistage Distributionally Robust - PowerPoint PPT Presentation

Effective Scenarios in Multistage Distributionally Robust Optimization with Total Variation Distance G uzin Bayraksan Department of Integrated Systems Engineering The Ohio State University April 4, 2019 Joint work with Hamed Rahimian


  1. Multistage Distributionally Robust Stochastic Program (DRSP) Toward a Nested Formulation of Multistage DRSP Given a scenario tree and a nominal distribution on the tree ξ T x ∗ T − 1 . . . ξ t +1 x ∗ T − 1 x ∗ t ξ T q t +1 | ξ [ t ] ξ t ξ T x ∗ t x ∗ T − 1 x ∗ . . . ξ t +1 t − 1 x ∗ T − 1 ξ T x ∗ . . . ξ 1 ξ 2 1 ξ T x ∗ T − 1 x ∗ . . . ξ t +1 t − 1 x ∗ T − 1 x ∗ t ξ T ξ t ξ T x ∗ t x ∗ T − 1 . . . ξ t +1 x ∗ T − 1 ξ T Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 9

  2. Multistage Distributionally Robust Stochastic Program (DRSP) Toward a Nested Formulation of Multistage DRSP Given a scenario tree and a nominal distribution on the tree ξ T x ∗ T − 1 . . . ξ t +1 x ∗ T − 1 x ∗ t ξ T q t +1 | ξ [ t ] ξ t ξ T x ∗ t x ∗ T − 1 x ∗ . . . ξ t +1 t − 1 x ∗ T − 1 ξ T x ∗ . . . ξ 1 ξ 2 1 ξ T x ∗ T − 1 x ∗ . . . ξ t +1 t − 1 x ∗ T − 1 x ∗ t ξ T q t +1 | ξ [ t ] ξ t ξ T x ∗ t x ∗ T − 1 . . . ξ t +1 x ∗ T − 1 ξ T Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 9

  3. Multistage Distributionally Robust Stochastic Program (DRSP) Nested Formulation of Multistage DRSP � � x 1 ∈X 1 g 1 ( x 1 , ξ 1 ) + E q 2 | ξ [1] min x 2 ∈X 2 g 2 ( x 2 , ξ 2 ) + E q 3 | ξ [2] min . . . + �� � � x T ∈X T g T ( x T , ξ T ) min E q T | ξ [ T − 1] . . . , where P t | ξ [ t − 1] is the conditional ambiguity set for stage- t probability measure, conditioned on ξ [ t − 1] . Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 10

  4. Multistage Distributionally Robust Stochastic Program (DRSP) Nested Formulation of Multistage DRSP max max  p 2 ∈P 2 | ξ [1] p 3 ∈P 3 | ξ [2] � � E p 2 � E p 3  x 1 ∈X 1 g 1 ( x 1 , ξ 1 ) + min  min x 2 ∈X 2 g 2 ( x 2 , ξ 2 ) + . . . +   max  p T ∈P T | ξ [ T − 1] � � � �  E p T x T ∈X T g T ( x T , ξ T ) min . . .  ,   where P t | ξ [ t − 1] is the conditional ambiguity set for stage- t probability measure, conditioned on ξ [ t − 1] . Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 10

  5. Multistage Distributionally Robust Stochastic Program (DRSP) Nested Formulation of Multistage DRSP max max  p 2 ∈P 2 | ξ [1] p 3 ∈P 3 | ξ [2] � � E p 2 � E p 3  x 1 ∈X 1 g 1 ( x 1 , ξ 1 ) + min  min x 2 ∈X 2 g 2 ( x 2 , ξ 2 ) + . . . +   max  p T ∈P T | ξ [ T − 1] � � � �  E p T x T ∈X T g T ( x T , ξ T ) min . . .  ,   where P t | ξ [ t − 1] is the conditional ambiguity set for stage- t probability measure, conditioned on ξ [ t − 1] . Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 10

  6. Multistage Distributionally Robust Stochastic Program (DRSP) How to Construct the Ambiguity Set (Multistage)? Moment- based sets: distributions with similar moments (Shapiro, 2012), (Xin et al., 2013), (Xin and Goldberg, 2015) Distance- based sets: sufficiently close distributions to a nominal distribution with respect to a distance Nested distance (Wasserstein metric): (Pflug and Pichler, 2014), (Analui and Pflug, 2014) Modified χ 2 distance: (Philpott et al., 2017) L ∞ norm: (Huang et al., 2017) General theory: (Shapiro, 2016; 2017; 2018) Total variation distance Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 11

  7. Multistage Distributionally Robust Stochastic Program (DRSP) How to Construct the Ambiguity Set (Multistage)? Moment- based sets: distributions with similar moments (Shapiro, 2012), (Xin et al., 2013), (Xin and Goldberg, 2015) Distance- based sets: sufficiently close distributions to a nominal distribution with respect to a distance Nested distance (Wasserstein metric): (Pflug and Pichler, 2014), (Analui and Pflug, 2014) Modified χ 2 distance: (Philpott et al., 2017) L ∞ norm: (Huang et al., 2017) General theory: (Shapiro, 2016; 2017; 2018) Total variation distance Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 11

  8. Multistage Distributionally Robust Stochastic Program (DRSP) Multistage DRSP with Total Variation Distance (DRSP-V) At stage t , given ξ [ t − 1] , instead of considering one (“nominal”) distribution q t | ξ [ t − 1] , Consider all distributions p t in � p t : V( p t , q t | ξ [ t − 1] ) := 1 � � � P t | ξ [ t − 1] = � p t − q t | ξ [ t − 1] � d ν ≤ γ t , � � 2 Ξ t | ξ [ t − 1] � p t d ν = 1 , Ξ t | ξ [ t − 1] � p t ≥ 0 , where Ξ t | ξ [ t − 1] is the sample space of stage t , given ξ [ t − 1] . Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 12

  9. Multistage Distributionally Robust Stochastic Program (DRSP) Multistage DRSP with Total Variation Distance (DRSP-V) At stage t , given ξ [ t − 1] , instead of considering one (“nominal”) distribution q t | ξ [ t − 1] , Consider all distributions p t in � p t : V( p t , q t | ξ [ t − 1] ) := 1 � � � P t | ξ [ t − 1] = � p t − q t | ξ [ t − 1] � d ν ≤ γ t , � � 2 Ξ t | ξ [ t − 1] � p t d ν = 1 , Ξ t | ξ [ t − 1] � p t ≥ 0 , where Ξ t | ξ [ t − 1] is the sample space of stage t , given ξ [ t − 1] . ◮ all distributions sufficiently close to the nominal distribution Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 12

  10. Multistage Distributionally Robust Stochastic Program (DRSP) Multistage DRSP with Total Variation Distance (DRSP-V) At stage t , given ξ [ t − 1] , instead of considering one (“nominal”) distribution q t | ξ [ t − 1] , Consider all distributions p t in � p t : V( p t , q t | ξ [ t − 1] ) := 1 � � � P t | ξ [ t − 1] = � p t − q t | ξ [ t − 1] � d ν ≤ γ t , � � 2 Ξ t | ξ [ t − 1] � p t d ν = 1 , Ξ t | ξ [ t − 1] � p t ≥ 0 , where Ξ t | ξ [ t − 1] is the sample space of stage t , given ξ [ t − 1] . ◮ ensure it is a probability measure Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 12

  11. Multistage Distributionally Robust Stochastic Program (DRSP) Aim Q1: How do we formulate this problem? What uncertain scenarios are important to a multistage DRSP Q2: model? How to define important scenarios? How to identify important scenarios? But . . . Let’s take a look at static/two-stage case first Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 13

  12. Two-Stage DRSP with Total Variation Distance Outline Introduction 1 Multistage Distributionally Robust Stochastic Program (DRSP) 2 Two-Stage DRSP with Total Variation Distance 3 4 Effective Scenarios in Multistage DRSP Solution Approach — A Decomposition Algorithm 5 Computational Results 6 Conclusion and Future Research 7 Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 13

  13. Two-Stage DRSP with Total Variation Distance Static/Two-Stage DRSP � � min f ( x ) := max p ∈P E p [ h ( x , ω )] , x ∈X where X ⊆ R n is a deterministic and non-empty convex compact set, Ω is sample space, assumed finite h : X × Ω �→ R is an integrable convex random function, i.e., for any x ∈ X , h ( x , · ) is integrable, and h ( · , ω ) is convex q -almost surely, Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 14

  14. Two-Stage DRSP with Total Variation Distance Static/Two-Stage DRSP � � min f ( x ) := max p ∈P E p [ h ( x , ω )] , x ∈X where q denotes a nominal probability distribution, which may be obtained from data, e.g., empirical distribution, P is the ambiguity set of distributions, a subset of all probability distributions on Ω, which may be obtained, e.g., via the total variation distance to the nominal distribution Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 14

  15. Two-Stage DRSP with Total Variation Distance Assessment Problem of “Removed” Scenarios Consider “removing” a set F ⊂ Ω of scenarios: P A := { p ∈ P : p ω = 0 , ω ∈ F} . Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 15

  16. Two-Stage DRSP with Total Variation Distance Assessment Problem of “Removed” Scenarios Consider “removing” a set F ⊂ Ω of scenarios: P A := { p ∈ P : p ω = 0 , ω ∈ F} . Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 15

  17. Two-Stage DRSP with Total Variation Distance Assessment Problem of “Removed” Scenarios Consider “removing” a set F ⊂ Ω of scenarios: P A := { p ∈ P : p ω = 0 , ω ∈ F} . The Assessment problem of scenarios in F is � � f A ( x ; F ) = � min max p ω h ω ( x ) , x ∈ X p ∈P A ( F ) ω ∈F c where If Inner Max of the Assessment Problem is Infeasible: f A ( x ; F ) = −∞ Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 15

  18. Two-Stage DRSP with Total Variation Distance Assessment Problem of “Removed” Scenarios Consider “removing” a set F ⊂ Ω of scenarios: P A := { p ∈ P : p ω = 0 , ω ∈ F} . The Assessment problem of scenarios in F is � � f A ( x ; F ) = � min max p ω h ω ( x ) , x ∈ X p ∈P A ( F ) ω ∈F c where If Inner Max of the Assessment Problem is Infeasible: f A ( x ; F ) = −∞ Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 15

  19. Two-Stage DRSP with Total Variation Distance Assessment Problem of “Removed” Scenarios Consider “removing” a set F ⊂ Ω of scenarios: P A := { p ∈ P : p ω = 0 , ω ∈ F} . The Assessment problem of scenarios in F is � � f A ( x ; F ) = � min max p ω h ω ( x ) , x ∈ X p ∈P A ( F ) ω ∈F c where If Inner Max of the Assessment Problem is Infeasible: f A ( x ; F ) = −∞ Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 15

  20. Two-Stage DRSP with Total Variation Distance Effective/Ineffective Scenarios in DRSP (Rahimian, B., Homem-de-Mello, 2018) Definition ( Effective Subset of Scenarios) At an optimal solution x ∗ , a subset F ⊂ Ω is called effective if by its “removal” the optimal value of the Assessment problem is strictly smaller than the optimal value of DRSP; i.e., if x ∈X f A ( x ; F ) < min min x ∈X f ( x ) . Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 16

  21. Two-Stage DRSP with Total Variation Distance Effective/Ineffective Scenarios in DRSP (Rahimian, B., Homem-de-Mello, 2018) Definition ( Effective Subset of Scenarios) At an optimal solution x ∗ , a subset F ⊂ Ω is called effective if by its “removal” the optimal value of the Assessment problem is strictly smaller than the optimal value of DRSP; i.e., if x ∈X f A ( x ; F ) < min min x ∈X f ( x ) . Definition ( Ineffective Subset of Scenarios) A subset F ⊂ Ω that is not effective is called ineffective. Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 16

  22. Two-Stage DRSP with Total Variation Distance DRSP with Total Variation Distance n � min x ∈X max p ω h ( x , ω ) p ∈P ω =1 where � n � 1 � � P = | p ω − q ω | ≤ γ, p ω = 1 , p ω ≥ 0 , ∀ ω , 2 ω ∈ Ω ω =1 Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 17

  23. Two-Stage DRSP with Total Variation Distance Risk-Averse Interpretation Proposition ( Risk-Averse Interpretation of DRSP with Total Variation)  E q [ h ( x , ω )] , if γ = 0 ,    γ sup ω ∈ Ω h ( x , ω ) + (1 − γ ) CVaR γ [ h ( x , ω )] , if 0 < γ < 1 , f γ ( x ) = sup h ( x , ω ) , if γ ≥ 1 ,    ω ∈ Ω By (Jiang and Guan, 2016) . Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 18

  24. Two-Stage DRSP with Total Variation Distance How to Find Effective/Ineffective Scenarios for DRSP? How can we determine the effectiveness of a scenario? Resolve for any scenario ω ∈ Ω Form the corresponding Assessment problem, Resolve the corresponding Assessment problem, Compare the optimal values to determine the effectiveness of the scenario. Exploit the structure of the ambiguity set Propose easy-to-check conditions (based on optimal solution and worst-case distribution) to identify the effectiveness of a scenario Low computational cost We might not be able to identify the effectiveness of all scenarios Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 19

  25. Two-Stage DRSP with Total Variation Distance Notation Consider an optimal solution ( x ∗ , p ∗ ) ∈ X × P to DRSP-V: x ∗ ∈ argmin x ∈X E p ∗ [ h ( x , ω )] p ∗ := p ∗ ( x ∗ ) ∈ argmax p ∈P E p [ h ( x ∗ , ω )] Define Ω 1 ( x ∗ ) := [ ω ∈ Ω : h ( x ∗ , ω ) < VaR γ [ h ( x ∗ , ω )]] Ω 2 ( x ∗ ) := [ ω ∈ Ω : h ( x ∗ , ω ) = VaR γ [ h ( x ∗ , ω )]] Ω 3 ( x ∗ ) := [ ω ∈ Ω : VaR γ [ h ( x ∗ , ω )] < h ( x ∗ , ω ) < sup ω ∈ Ω h ( x ∗ , ω )] Ω 4 ( x ∗ ) := [ ω ∈ Ω : h ( x ∗ , ω ) = sup ω ∈ Ω h ( x ∗ , ω )] Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 20

  26. Two-Stage DRSP with Total Variation Distance Ineffective Scenarios Theorem ( Easy-to-Check Conditions for Ineffective Scenarios, (Rahimian, B., Homem-de-Mello, 2018) ) Suppose ( x ∗ , p ∗ ) solves DRSP-V. Then, a scenario ω ′ with q ω ′ ≤ γ , is ineffective if any of the following conditions holds: ω ′ ∈ Ω 1 ( x ∗ ) , ω ′ ∈ Ω 2 ( x ∗ ) and q ω ′ = 0 , ω ′ ∈ Ω 2 ( x ∗ ) and � ω ∈ Ω 2 ( x ∗ ) p ∗ ω = 0 , ω ′ ∈ Ω 3 ( x ∗ ) and q ω ′ = 0 . Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 21

  27. Two-Stage DRSP with Total Variation Distance Effective Scenarios Theorem ( Easy-to-Check Conditions for Effective Scenarios) Suppose ( x ∗ , p ∗ ) solves DRSP-V. Then, a scenario ω ′ is effective if any of the following conditions holds: q ω ′ > γ , Ω 2 ( x ∗ ) = { ω ′ } and p ∗ ω ′ > 0 , ω ′ ∈ Ω 3 ( x ∗ ) and q ω ′ > 0 , ω ′ ∈ Ω 4 ( x ∗ ) and q ω ′ > 0 , Ω 4 ( x ∗ ) = { ω ′ } . Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 22

  28. Two-Stage DRSP with Total Variation Distance Effective Scenarios Theorem ( Easy-to-Check Conditions for Effective Scenarios) Suppose ( x ∗ , p ∗ ) solves DRSP-V. Then, a scenario ω ′ is effective if any of the following conditions holds: q ω ′ > γ , Ω 2 ( x ∗ ) = { ω ′ } and p ∗ ω ′ > 0 , ω ′ ∈ Ω 3 ( x ∗ ) and q ω ′ > 0 , ω ′ ∈ Ω 4 ( x ∗ ) and q ω ′ > 0 , Ω 4 ( x ∗ ) = { ω ′ } . ◮ Trivially Effective ! Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 22

  29. Two-Stage DRSP with Total Variation Distance Beyond Previous Theorems: Identify Undetermined Scenarios Theorem ( Easy-to-Check Conditions to Identify Undetermined Scenarios) Suppose ( x ∗ , p ∗ ) solves DRO-V. For a scenario ω ′ ∈ Ω 2 ( x ∗ ) with q ω ′ > 0 , suppose that the effectiveness of scenario ω ′ is not identified by the previous theorems. Let F = { ω ′ } . If 1 VaR γ F [ h ( x ∗ , ω ) |F c ] < VaR γ [ h ( x ∗ , ω )] , and 2 either there exists a scenario � � VaR γ F [ h ( x ∗ , ω ) |F c ] < h ( x ∗ , ω ) < VaR γ [ h ( x ∗ , ω )] ω ∈ with � � x ∗ , VaR γ F [ h ( x ∗ ω ) , |F c ] q ω > 0 or Ψ |F c > γ F , then scenario ω ′ is effective. Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 23

  30. Two-Stage DRSP with Total Variation Distance Effective/Ineffective Scenarios Summary (Two-Stage) Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 24

  31. Effective Scenarios in Multistage DRSP Outline Introduction 1 Multistage Distributionally Robust Stochastic Program (DRSP) 2 Two-Stage DRSP with Total Variation Distance 3 4 Effective Scenarios in Multistage DRSP Solution Approach — A Decomposition Algorithm 5 Computational Results 6 Conclusion and Future Research 7 Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 24

  32. Effective Scenarios in Multistage DRSP Effective/Ineffective Scenarios in Multistage DRSP What happens in the Multistage case? Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 25

  33. Effective Scenarios in Multistage DRSP Relation to Multistage Risk-Averse Optimization � x 1 ∈X 1 g 1 ( x 1 , ξ 1 ) + min max x 2 ∈X 2 g 2 ( x 2 , ξ 2 ) + min max . . . E p 2 p 2 ∈P 2 | ξ [1] p 3 ∈P 3 | ξ [2] �� � � · · · + max x T ∈X T g T ( x T , ξ T ) min E p T . . . p T ∈P T | ξ [ T − 1] Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 26

  34. Effective Scenarios in Multistage DRSP Relation to Multistage Risk-Averse Optimization Proposition ( Risk-Averse Interpretation of Multistage DRSP-V) Multistage DRSP-V can be written as � � � � �� x 1 ∈X 1 g 1 ( x 1 , ξ 1 )+ R 2 | ξ [1] min x 2 ∈X 2 g 2 ( x 2 , ξ 2 ) + R 3 | ξ [2] min . . . + R T | ξ [ T − 1] x T ∈X T g T ( x T , ξ T ) min . . . , where R ’s are the (real-valued) coherent conditional risk mappings  E q t + 1 | ξ [ t ] [ · ] , if γ = 0 ,    γ sup ξ t +1 ∈ Ξ t +1 | ξ [ t ] [ · ] + (1 − γ ) CVaR γ [ · ] , if 0 < γ < 1 , R t +1 | ξ [ t ] [ · ] =  sup ξ t +1 ∈ Ξ t +1 | ξ [ t ] [ · ] , if γ ≥ 1 .   where · is Q t +1 ( x [ t ] , ξ [ t +1] ) is the cost-to-go function. Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 27

  35. Effective Scenarios in Multistage DRSP Effective/Ineffective Scenarios in Multistage DRSP? Now we have a scenario tree. What to do? ξ T x ∗ T − 1 . . . ξ t +1 x ∗ T − 1 x ∗ t ξ T ξ t ξ T x ∗ t x ∗ T − 1 x ∗ . . . ξ t +1 t − 1 x ∗ T − 1 ξ T x ∗ . . . ξ 1 ξ 2 1 ξ T x ∗ T − 1 x ∗ . . . ξ t +1 t − 1 x ∗ T − 1 x ∗ t ξ T ξ t ξ T x ∗ t x ∗ T − 1 . . . ξ t +1 x ∗ T − 1 ξ T Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 28

  36. Effective Scenarios in Multistage DRSP Effective/Ineffective Scenarios in Multistage DRSP? Questions What is the effectiveness of a scenario (path)? What is the effectiveness of a realization in stage t + 1? Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 29

  37. Effective Scenarios in Multistage DRSP Effective/Ineffective Scenarios in Multistage DRSP? Questions What is the effectiveness of a scenario (path)? What is the effectiveness of a realization in stage t + 1? Main Idea Look at realizations conditioned on their history of decisions and stochastic process → At an optimal policy x ∗ , if we look at stage t , given x ∗ [ t − 1] and ξ [ t ] , previous definitions on effective/ineffective scenarios hold conditionally . Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 29

  38. Effective Scenarios in Multistage DRSP Effective/Ineffective Scenarios in Multistage DRSP? ξ T x ∗ T − 1 . . . ξ t +1 x ∗ T − 1 x ∗ t ξ T ξ t ξ T x ∗ t x ∗ T − 1 x ∗ ξ t +1 . . . t − 1 x ∗ T − 1 ξ T x ∗ . . . ξ 1 ξ 2 1 ξ T x ∗ T − 1 x ∗ . . . ξ t +1 t − 1 x ∗ T − 1 x ∗ t ξ T ξ t ξ T x ∗ t x ∗ T − 1 . . . ξ t +1 x ∗ T − 1 ξ T Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 30

  39. Effective Scenarios in Multistage DRSP Effective/Ineffective Scenarios in Multistage DRSP? ξ T x ∗ T − 1 . . . ξ t +1 x ∗ T − 1 x ∗ t ξ T ξ t ξ T x ∗ t x ∗ T − 1 x ∗ . . . ξ t +1 t − 1 x ∗ T − 1 ξ T x ∗ . . . ξ 1 ξ 2 1 ξ T x ∗ T − 1 x ∗ ξ t +1 . . . t − 1 x ∗ T − 1 x ∗ t ξ T ξ t ξ T x ∗ t x ∗ T − 1 . . . ξ t +1 x ∗ T − 1 ξ T Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 30

  40. Effective Scenarios in Multistage DRSP Effective/Ineffective Scenarios in Multistage DRSP? ξ T x ∗ T − 1 . . . ξ t +1 x ∗ T − 1 x ∗ t ξ T x ∈ X max min E p [ h ( x , ω )] ξ t p ∈P ξ T x ∗ t x ∗ T − 1 x ∗ . . . ξ t +1 t − 1 x ∗ T − 1 ξ T x ∗ . . . ξ 1 ξ 2 1 ξ T x ∗ T − 1 x ∗ ξ t +1 . . . t − 1 x ∗ T − 1 x ∗ t ξ T ξ t ξ T x ∗ t x ∗ T − 1 . . . ξ t +1 x ∗ T − 1 ξ T Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 30

  41. Effective Scenarios in Multistage DRSP Effective/Ineffective Scenarios in Multistage DRSP? ξ T x ∗ T − 1 . . . ξ t +1 x ∗ T − 1 x ∗ x ∗ t t ξ T x ∈ X max min E p [ h ( x , ω )] ξ t p ∈P ξ T x ∗ t x ∗ T − 1 x ∗ . . . ξ t +1 t − 1 x ∗ T − 1 ξ T x ∗ . . . ξ 1 ξ 2 1 ξ T x ∗ T − 1 x ∗ ξ t +1 . . . t − 1 x ∗ T − 1 x ∗ t ξ T ξ t ξ T x ∗ t x ∗ T − 1 . . . ξ t +1 x ∗ T − 1 ξ T Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 30

  42. Effective Scenarios in Multistage DRSP Effective Scenarios in Multistage DRSP: Conditional Effectiveness Definition ( Conditionally Effective Realization) At an optimal policy x ∗ := [ x ∗ 1 , . . . , x ∗ T ], a realization of ξ t +1 in stage t + 1 is called conditionally effective, given x ∗ [ t − 1] and ξ [ t ] , if by its removal the optimal stage- t cost function (immediate cost + cost-to-go function) of the new problem is strictly smaller than the optimal value of the original stage- t problem in multistage DRSP. Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 31

  43. Effective Scenarios in Multistage DRSP Effective Scenarios in Multistage DRSP: Effectiveness of a Scenario Path Definition ( Effective Scenario Path) At an optimal policy x ∗ := [ x ∗ 1 , . . . , x ∗ T ], a scenario path { ξ t } T t =1 is called effective if by its “removal” the optimal value of the new problem is strictly smaller than the optimal value of multistage DRSP. NOTE: Removing a scenario path is defined by forcing the probability of ξ T to be zero. Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 32

  44. Effective Scenarios in Multistage DRSP Difference Between Conditional Effective Realizations and Effective Scenario Paths ξ T x ∗ T − 1 . . . ξ t +1 x ∗ T − 1 x ∗ t ξ T min x ∈ X max E p [ h ( x , ω )] ξ t p ∈P ξ T x ∗ t x ∗ T − 1 x ∗ . . . ξ t +1 t − 1 x ∗ T − 1 ξ T x ∗ . . . ξ 1 ξ 2 1 ξ T x ∗ T − 1 x ∗ . . . ξ t +1 t − 1 x ∗ T − 1 x ∗ t ξ T ξ t ξ T x ∗ t x ∗ T − 1 . . . ξ t +1 x ∗ T − 1 ξ T Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 33

  45. Effective Scenarios in Multistage DRSP Difference Between Conditional Effective Realizations and Effective Scenario Paths ξ T x ∗ T − 1 . . . ξ t +1 x ∗ T − 1 x ∗ x ∗ t t ξ T min x ∈ X max E p [ h ( x , ω )] ξ t p ∈P ξ T x ∗ t x ∗ T − 1 x ∗ . . . ξ t +1 t − 1 x ∗ T − 1 ξ T x ∗ . . . ξ 1 ξ 2 1 ξ T x ∗ T − 1 x ∗ . . . ξ t +1 t − 1 x ∗ T − 1 x ∗ t ξ T ξ t ξ T x ∗ t x ∗ T − 1 . . . ξ t +1 x ∗ T − 1 ξ T Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 33

  46. Effective Scenarios in Multistage DRSP Difference Between Conditional Effective Realizations and Effective Scenario Paths ξ T x ∗ x ∗ T − 1 T − 1 . . . ξ t +1 x ∗ x ∗ T − 1 T − 1 x ∗ x ∗ t t ξ T x ∈ X max min E p [ h ( x , ω )] ξ t p ∈P ξ T x ∗ x ∗ t t x ∗ x ∗ T − 1 T − 1 x ∗ x ∗ . . . ξ t +1 t − 1 t − 1 x ∗ x ∗ T − 1 T − 1 ξ T x ∗ x ∗ . . . ξ 1 ξ 2 1 1 ξ T x ∗ x ∗ T − 1 T − 1 x ∗ x ∗ . . . ξ t +1 t − 1 t − 1 x ∗ x ∗ T − 1 T − 1 x ∗ x ∗ t t ξ T ξ t ξ T x ∗ x ∗ t t x ∗ x ∗ T − 1 T − 1 . . . ξ t +1 x ∗ x ∗ T − 1 T − 1 ξ T Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 33

  47. Effective Scenarios in Multistage DRSP Difference Between Conditional Effective Realizations and Effective Scenario Paths ξ T x ∗ x ∗ T − 1 T − 1 . . . ξ t +1 x ∗ x ∗ T − 1 T − 1 x ∗ x ∗ t t ξ T x ∈ X max min E p [ h ( x , ω )] ξ t p ∈P ξ T x ∗ x ∗ t t x ∗ x ∗ T − 1 T − 1 x ∗ x ∗ . . . ξ t +1 t − 1 t − 1 x ∗ x ∗ T − 1 T − 1 ξ T x ∗ x ∗ . . . ξ 1 ξ 2 1 1 ξ T x ∗ x ∗ T − 1 T − 1 x ∗ x ∗ . . . ξ t +1 t − 1 t − 1 x ∗ x ∗ T − 1 T − 1 x ∗ x ∗ t t ξ T ξ t ξ T x ∗ x ∗ t t x ∗ x ∗ T − 1 T − 1 . . . ξ t +1 x ∗ x ∗ T − 1 T − 1 ξ T Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 33

  48. Effective Scenarios in Multistage DRSP Difference Between Conditional Effective Realizations and Effective Scenario Paths ξ T x ∗ x ∗ T − 1 T − 1 . . . ξ t +1 x ∗ x ∗ x ∗ x ∗ T − 1 T − 1 t t ξ T x ∈ X max min E p [ h ( x , ω )] ξ t p ∈P ξ T x ∗ x ∗ t t x ∗ x ∗ T − 1 T − 1 x ∗ x ∗ . . . ξ t +1 t − 1 t − 1 x ∗ x ∗ T − 1 T − 1 ξ T x ∗ x ∗ . . . ξ 1 ξ 2 1 1 ξ T x ∗ x ∗ T − 1 T − 1 x ∗ x ∗ . . . ξ t +1 t − 1 t − 1 x ∗ x ∗ T − 1 T − 1 x ∗ x ∗ t t ξ T ξ t ξ T x ∗ x ∗ t t x ∗ x ∗ T − 1 T − 1 ξ t +1 . . . x ∗ x ∗ T − 1 T − 1 ξ T Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 33

  49. Effective Scenarios in Multistage DRSP How to Find Effective/Ineffective Scenarios for Multistage DRSP-V? Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 34

  50. Effective Scenarios in Multistage DRSP How to Find Effective/Ineffective Scenarios for Multistage DRSP-V? Resolve? Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 34

  51. Effective Scenarios in Multistage DRSP How to Find Effective/Ineffective Scenarios for Multistage DRSP-V? Resolve? Suppose each node has n children. Then, we would have to solve many problems! Effectiveness of Scenario Paths : n T − 1 problems at stage T Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 34

  52. Effective Scenarios in Multistage DRSP How to Find Effective/Ineffective Scenarios for Multistage DRSP-V? Resolve? Suppose each node has n children. Then, we would have to solve many problems! Effectiveness of Scenario Paths : n T − 1 problems at stage T Conditionally Effectiveness of Realizations : n + . . . + n T − 1 problems at stage 2 + . . . + stage T Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 34

  53. Effective Scenarios in Multistage DRSP How to Find Effective/Ineffective Scenarios for Multistage DRSP-V? Resolve? Suppose each node has n children. Then, we would have to solve many problems! Effectiveness of Scenario Paths : n T − 1 problems at stage T Conditionally Effectiveness of Realizations : n + . . . + n T − 1 problems at stage 2 + . . . + stage T → AIM: Propose easy-to-check conditions Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 34

  54. Effective Scenarios in Multistage DRSP Use Conditional Effectiveness of Realizations in Multistage DRSP-V AIM: Propose easy-to-check conditions Theorem [Conditionally Multistage ← Two-stage] Our easy-to-check conditions to identify effective/ineffective scenarios in static/two-stage DRSP-V are valid conditions to identify conditionally effective/ineffective scenarios in multistage DRSP-V. Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 35

  55. Effective Scenarios in Multistage DRSP Effectiveness of Scenario Paths in Multistage DRSP-V Consider a scenario path { ξ t } T t =1 . Theorem If ξ t is conditionally effective by our easy-to-check conditions, for all t = 1 , . . . , T , then, the scenario path { ξ t } T t =1 is effective. Theorem If ξ T is not trivially conditionally effective (i.e., too large nominal conditional probability) and there exists t , t = 1 , . . . , T , such that ξ t is conditionally ineffective by our easy-to-check conditions, then, the scenario path { ξ t } T t =1 is ineffective. Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 36

  56. Effective Scenarios in Multistage DRSP Easy-To-Check Conditions for Effectiveness of Scenario Paths ξ 1 CI CE ξ 1 ξ 2 2 2 CU CE CE CU ξ 1 , 1 ξ 1 , 2 ξ 2 , 1 ξ 2 , 2 3 3 3 3 Effective Unknown Ineffective Ineffective Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 37

  57. Solution Approach — A Decomposition Algorithm Outline Introduction 1 Multistage Distributionally Robust Stochastic Program (DRSP) 2 Two-Stage DRSP with Total Variation Distance 3 4 Effective Scenarios in Multistage DRSP Solution Approach — A Decomposition Algorithm 5 Computational Results 6 Conclusion and Future Research 7 Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 37

  58. Solution Approach — A Decomposition Algorithm Dynamic Programming Formulation  � �  min min g 1 ( x 1 , ξ 1 ) + max E p 2 g 2 ( x 2 , ξ 2 ) + . . . + max E p T min g T ( x T , ξ T )  x 1 ∈X 1 p 2 ∈P 2 | ξ [1] x 2 ∈X 2 p T ∈P T | ξ [ T − 1] x T ∈X T Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 38

  59. Solution Approach — A Decomposition Algorithm Dynamic Programming Formulation  � �  min min g 1 ( x 1 , ξ 1 ) + max E p 2 g 2 ( x 2 , ξ 2 ) + . . . + max E p T min g T ( x T , ξ T )  x 1 ∈X 1 p 2 ∈P 2 | ξ [1] x 2 ∈X 2 p T ∈P T | ξ [ T − 1] x T ∈X T � �� � Q 2 ( x 1 ,ξ [2] ) First-stage cost function � � x 1 ∈X 1 g 1 ( x 1 , ξ 1 ) + min max Q 2 ( x 1 , ξ [2] ) E p 2 p 2 ∈P 2 | ξ [1] Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 38

  60. Solution Approach — A Decomposition Algorithm Dynamic Programming Formulation           � �   min g 1 ( x 1 , ξ 1 ) + max E p 2 min g 2 ( x 2 , ξ 2 ) + . . . + max E p T min g T ( x T , ξ T )     x 1 ∈X 1 p 2 ∈P 2 | ξ [1] x 2 ∈X 2 p T ∈P T | ξ [ T − 1] x T ∈X T   � �� �    Q T ( x T − 1 ,ξ [ T ] )    � �� � Q 3 ( x 2 ,ξ [3] ) � �� � Q 2 ( x 1 ,ξ [2] ) First-stage cost function � � x 1 ∈X 1 g 1 ( x 1 , ξ 1 ) + min max Q 2 ( x 1 , ξ [2] ) E p 2 p 2 ∈P 2 | ξ [1] stage-t cost function � � Q t ( x t − 1 , ξ [ t ] ) := min x t ∈X t g t ( x t , ξ t ) + max E p t +1 Q t +1 ( x t , ξ [ t +1] ) p t +1 ∈P t +1 | ξ [ t ] Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 38

  61. Solution Approach — A Decomposition Algorithm A Cutting Plane Approach stage-t cost function � � Q t ( x t − 1 , ξ [ t ] ) = min x t ∈X t g t ( x t , ξ t ) + max Q t +1 ( x t , ξ [ t +1] ) E p t +1 p t +1 ∈P t +1 | ξ [ t ] Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 39

  62. Solution Approach — A Decomposition Algorithm A Cutting Plane Approach stage-t cost function Q t ( x t − 1 , ξ [ t ] ) = min x t ∈X t g t ( x t , ξ t ) + α t � � s.t. α t ≥ max Q t +1 ( x t , ξ [ t +1] ) E p t +1 p t +1 ∈P t +1 | ξ [ t ] Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 39

  63. Solution Approach — A Decomposition Algorithm A Cutting Plane Approach stage-t cost function Q t ( x t − 1 , ξ [ t ] ) = min x t ∈X t g t ( x t , ξ t ) + α t � � s.t. α t ≥ E p t +1 Q t +1 ( x t , ξ [ t +1] ) p t +1 ∈ P t +1 | ξ [ t ] , Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 39

  64. Solution Approach — A Decomposition Algorithm A Cutting Plane Approach stage-t cost function Q t ( x t − 1 , ξ [ t ] ) = min x t ∈X t g t ( x t , ξ t ) + α t � � s.t. α t ≥ E p t +1 Q t +1 ( x t , ξ [ t +1] ) , p t +1 ∈ P t +1 | ξ [ t ] For multistage DRSP-V, P t +1 | ξ [ t ] is a polyhedron = ⇒ Finite convergence Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 39

  65. Solution Approach — A Decomposition Algorithm A Cutting Plane Approach stage-t cost function Q t ( x t − 1 , ξ [ t ] ) = min x t ∈X t g t ( x t , ξ t ) + α t � � s.t. α t ≥ E p t +1 Q t +1 ( x t , ξ [ t +1] ) , p t +1 ∈ P t +1 | ξ [ t ] For multistage DRSP-V, P t +1 | ξ [ t ] is a polyhedron = ⇒ Finite convergence This idea can be applied to any polyhedral ambiguity set, with finite convergence guaranteed Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 39

  66. Solution Approach — A Decomposition Algorithm How to Generate Distributional Cuts? Distribution Separation Problem For a fixed x t ∈ X t , solve � � max Q t +1 ( x t , ξ [ t +1] ) E p t +1 p t +1 ∈P t +1 | ξ [ t ] Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 40

  67. Solution Approach — A Decomposition Algorithm How to Generate Distributional Cuts? Distribution Separation Problem For a fixed x t ∈ X t , solve � max p t +1 Q t +1 ( x t , · ) d ν p t +1 ∈P t +1 | ξ [ t ] Ξ t +1 | ξ [ t ] Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 40

  68. Solution Approach — A Decomposition Algorithm How to Generate Distributional Cuts? Distribution Separation Problem For a fixed x t ∈ X t , solve � max p t +1 Q t +1 ( x t , · ) d ν p t +1 ∈P t +1 | ξ [ t ] Ξ t +1 | ξ [ t ] For multistage DRSP-V, P t +1 | ξ [ t ] is a polytope = ⇒ Optimum is obtained at an extreme point Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 40

  69. Solution Approach — A Decomposition Algorithm How to Generate Distributional Cuts? Distribution Separation Problem For a fixed x t ∈ X t , solve � max p t +1 Q t +1 ( x t , · ) d ν p t +1 ∈P t +1 | ξ [ t ] Ξ t +1 | ξ [ t ] For multistage DRSP-V, P t +1 | ξ [ t ] is a polytope = ⇒ Optimum is obtained at an extreme point Challenge We do not have Q t +1 ( x t , ξ [ t +1] ) Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 40

  70. Solution Approach — A Decomposition Algorithm How to Generate Distributional Cuts? Distribution Separation Problem For a fixed x t ∈ X t , solve � p t +1 ¯ max Q t +1 ( x t , · ) d ν p t +1 ∈P t +1 | ξ [ t ] Ξ t +1 | ξ [ t ] For multistage DRSP-V, P t +1 | ξ [ t ] is a polytope = ⇒ Optimum is obtained at an extreme point Challenge We do not have Q t +1 ( x t , ξ [ t +1] ) But... We can use an inner (upper) approximation ¯ Q t +1 ( x t , ξ [ t +1] ) Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 40

  71. Solution Approach — A Decomposition Algorithm Primal Decomposition Algorithm Main Idea Combine Nested L-shaped method and Distribution Separation problem Forward Pass Obtain x = [ x 1 , . . . , x T ] Use inner approximations on Q t +1 ( x t , ξ [ t +1] ), t = T − 1 , . . . , 1 to obtain p = [ p T , . . . , p 2 ] Backward Pass Refine outer approximations on Q t +1 ( x t , ξ [ t +1] ) and � � max p t +1 ∈P t +1 | ξ [ t ] E p t +1 Q t +1 ( x t , ξ [ t +1] ) Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 41

  72. Computational Results Outline Introduction 1 Multistage Distributionally Robust Stochastic Program (DRSP) 2 Two-Stage DRSP with Total Variation Distance 3 4 Effective Scenarios in Multistage DRSP Solution Approach — A Decomposition Algorithm 5 Computational Results 6 Conclusion and Future Research 7 Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 41

  73. Computational Results Test Problems We considered two sets of problems: SGPF—A Bond Investment Planning problem described by (Frauendorfer, Marohn, and Sch¨ Aurle, 1997) to maximize profit under uncertain returns Water Resources Allocation—Allocate Colorado River water among different users under water demand and supply uncertainties at minimum cost? (Zhang, Rahimian, Bayraksan, 2016) We implemented our primal decomposition algorithm in C++ on top of SUTIL 0.1 (A Stochastic Programming Utility Library) (Czyzyk, Linderoth, and Shen, 2008) and solved problems with CPLEX 12.7. Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 42

  74. Computational Results SGPF3Y3 (3 Stages, 5 2 = 25 Scenarios) Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 43

  75. Computational Results SGPF3Y3 (3 Stages, 5 2 = 25 Scenarios) Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 43

  76. Computational Results SGPF3Y3 (3 Stages, 5 2 = 25 Scenarios) Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 43

  77. Computational Results SGPF3Y3 (3 Stages, 5 2 = 25 Scenarios) Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 43

  78. Computational Results SGPF3Y3 (3 Stages, 5 2 = 25 Scenarios) Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 43

  79. Computational Results SGPF3Y6 (6 Stages, 5 5 = 3125 Scenarios) # of scenario path γ ineffective effective undetermined 0.00 0 3125 0 0.05 0 3125 0 0.10 0 3125 0 0.15 0 3125 0 0.20 994 2131 0 0.25 2101 1024 0 0.30 2101 1024 0 0.35 2101 1024 0 0.40 2745 380 0 0.45 2793 183 149 0.50 2829 214 82 0.55 2873 234 18 0.60 3076 37 12 0.65 3081 24 20 0.70 3083 24 18 0.75 3089 36 0 0.80 3116 9 0 0.85 3116 9 0 0.90 3116 9 0 0.95 3116 9 0 1.00 3116 9 0 Rahimian, Bayraksan & Homem-de-Mello Effective Scen.s in Multistage DRSP ECOM 2019 44

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend


More recommend