A Comparison of Two Parallel Ranking and Selection Procedures Eric - - PowerPoint PPT Presentation

▶

Oct 10, 2022 569 likes •857 views

A Comparison of Two Parallel Ranking and Selection Procedures Eric C. Ni Shane G. Henderson School of Operations Research and Information Engineering, Cornell University Susan R. Hunter School of Industrial Engineering, Purdue University

SLIDE 1

A Comparison of Two Parallel Ranking and Selection Procedures

Eric C. Ni Shane G. Henderson

School of Operations Research and Information Engineering, Cornell University

Susan R. Hunter

School of Industrial Engineering, Purdue University

Dragos Florin Ciocan

INSEAD

Winter Simulation Conference, Savannah, GA December 8, 2014

SLIDE 2

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

1 Introduction 2 Parallel Procedures 3 Numerical Experiments 4 New PGS Procedure 5 Conclusion

Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 1/18

SLIDE 3

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

1 Introduction 2 Parallel Procedures 3 Numerical Experiments 4 New PGS Procedure 5 Conclusion

Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 2/18

SLIDE 4

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

Ranking and Selection (R&S)

max

i∈S

µi = E[Y (i; ξ)]

Optimize a function through a stochastic simulation.
Feasible region is finite: k = |S| < ∞.
User-specified parameter δ.
Types of statistical guarantee (assuming µ1 ≤ . . . ≤ µk):

◮ Correct selection (CS):

P[select system k|µk ≥ µk−1 + δ] ≥ 1 − α;

◮ Good selection (GS):

P[select system i : µi ≥ µk − δ] ≥ 1 − α.

Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 3/18

SLIDE 5

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

Paths to GS guarantee

GS procedures:

Multiple comparisons with the best (Rinott, 1978; Nelson

and Matejcik, 1995; Nelson et al., 2001)

◮ Simultaneous confidence intervals on µi − maxj=i µj of

width δ for all i.

Best-arm selection algorithms (Jamieson et al., 2013;

Jamieson and Nowak, 2014)

◮ Guarantees to find the true best system if it is unique.

CS procedures:

Sequential screening (Paulson, 1964; Fabian, 1974; Kim

and Nelson, 2001, 2006; Hong, 2006).

◮ The assumption µk ≥ µk−1 + δ is essential in proving

statistical validity.

Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 4/18

SLIDE 6

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

1 Introduction 2 Parallel Procedures 3 Numerical Experiments 4 New PGS Procedure 5 Conclusion

Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 5/18

SLIDE 7

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

Previous work on parallel R&S

Web services-based parallel simulation optimization (Yoo

et al., 2009; Luo et al., 2000)

Parallel version of a screening-based procedure, with

asymptotic CS guarantee (Luo et al., 2013)

Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 5/18

SLIDE 8

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

A parallel R&S procedure NHH

Ni et al. (2013)

Stage 0: Simulate all systems to estimate simulation

completion times.

Stage 1: (If variances need to be estimated) Independently
f Stage 0, systems are simulated in parallel to obtain

variance estimates.

Stage 2: Remaining systems are iteratively simulated and

screened (both in parallel) until one system remains. Highly scalable. Inherits CS guarantee from its sequential predecessor (Hong, 2006).

Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 6/18

SLIDE 9

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

(Parallel) Procedure NSGS

Two-stage procedure proposed by Nelson et al. (2001).

Stage 0: We simulate all systems to estimate simulation

completion times.

Stage 1: (If variances need to be estimated) Independently
f Stage 0, systems are simulated in parallel to obtain

variance estimates. Perform a round of screening (in parallel) with Stage 1 output.

Stage 2: (The Rinott step) Simulate up to ⌈(hSi/δ)2⌉

replications of each system i and choose the system with the highest sample mean. Parallel implementation inherits GS guarantee from Nelson et al. (2001), as the Rinott step leads to MCB confidence intervals.

Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 7/18

SLIDE 10

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

Procedure complexity

Let ∆i := µk − µi. The expected number of replications required to eliminate a system i by system k is approximately

O(σ2

i ∆−2 i

log(∆−2

i )) using best-arm algorithms (Jamieson

et al. (2013))

O((σ2

i + σiσk)[max(∆i, δ)]−1δ−1) using NHH

O(σ2

i δ−2) using NSGS (and its parallel version)

Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 8/18

SLIDE 11

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

1 Introduction 2 Parallel Procedures 3 Numerical Experiments 4 New PGS Procedure 5 Conclusion

Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 9/18

SLIDE 12

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

Numerical example

Our parallel procedures are applied to a throughput-maximization problem (SimOpt.org).

Table 1: Summary of three instances of the test problem

Number of Highest

Num. of systems in [µk − δ, µk]

Instance systems k mean µk δ = 0.01 δ = 0.1 δ = 1 1 3249 5.78 6 21 256 2 57624 15.70 12 43 552 3 1016127 41.66 28 97 866

Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 9/18

SLIDE 13

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

Numerical example

1 2 3 4 5 6 7

Mean µ

0.0 0.2 0.4 0.6 0.8 1.0

Standard Deviation σ Instance 1: 3249 systems

5 10 15 20

Mean µ

0.0 0.5 1.0 1.5 2.0 2.5 3.0

Standard Deviation σ Instance 2: 57624 systems

Figure 1: Mean-standard deviation profiles of two instances

Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 10/18

SLIDE 14

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

Procedure Performance

Table 2: Summary of procedure costs on 3 instances of the throughput maximization problem with α = 0.05, n0 = 20

Total simulation Total Per Number of replications running replication systems k n1 δ Procedure (×106) time (s) time (µs) 3249 60 0.01 NHH 3.9 23 375 NSGS 15.0 182 758 0.1 NHH 1.1 7.0 396 NSGS 0.4 2.9 480 57624 80 0.01 NHH 92.0 536 365 NSGS 1,600.0 10,327 403 0.1 NHH 30.0 179 371 NSGS 22.0 158 455 1016127 100 0.01 NHH 3,600.0 1,901 726 NSGS N/A (too costly) 0.1 NHH 720.0 324 572 NSGS 2,300.0 1,724 564 Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 11/18

SLIDE 15

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

3 15 63 255 1023 Number of workers 101 102 103 Wallclock time (s) NHH procedure Perfect scaling Actual performance 3 15 63 255 1023 Number of workers 101 102 103 Wallclock time (s) NSGS procedure Perfect scaling Actual performance

Figure 2: Scaling result on 57,624 systems, δ = 0.1

Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 12/18

SLIDE 16

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

1 Introduction 2 Parallel Procedures 3 Numerical Experiments 4 New PGS Procedure 5 Conclusion

Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 13/18

SLIDE 17

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

A parallel R&S procedure with PGS

Two main stages:

Stage 1: (In parallel) Simulate and periodically screen

systems until one system remains, or a pre-specified termination criterion is met.

◮ The screening method and termination criterion are jointly

chosen such that P[System k survives Stage 1] is sufficiently large, say 1 − α/2.

Stage 2: (The Rinott step) Simulate up to ⌈(hSi/δ)2⌉

replications of each system i and choose the system with the highest sample mean.

◮ Stage 2 guarantees good selection amongst those surviving

Stage 1.

Theorem 1

The new procedure provides a good selection guarantee P[select system i : µi ≥ µk − δ] ≥ 1 − α if simulation outcomes Y (i; ξ) ∼ Normal(µi, σ2

i ).

Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 13/18

SLIDE 18

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

Implementing the PGS procedure

MapReduce

Standard distributed programming model
Map() procedure to process data in parallel

◮ Simulate surviving systems

Reduce() procedure to summarize

◮ Screen using the additional statistics

Easy to program but incurs higher overhead
Runs on the cloud (e.g. Amazon EC2)

Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 14/18

SLIDE 19

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

Procedure Performance

Table 3: Summary of procedure costs

Total simulation Total Per Number of replications running replication systems k n1 δ Procedure (×106) time (s) time (µs) 3249 60 0.01 NHH 3.9 23 375 NSGS 15.0 182 758 GS 13.0 0.1 NHH 1.1 7.0 396 NSGS 0.4 2.9 480 GS 2.4 720 18,000 57624 80 0.01 NHH 92.0 536 365 NSGS 1,600.0 10,327 403 GS 280.0 0.1 NHH 30.0 179 371 NSGS 22.0 158 455 GS 12.0 1,878 9,400

NHH and NSGS on XSEDE high-performance cluster
GS on Amazon EC2 using Hadoop MapReduce

Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 15/18

SLIDE 20

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

1 Introduction 2 Parallel Procedures 3 Numerical Experiments 4 New PGS Procedure 5 Conclusion

Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 16/18

SLIDE 21

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

Conclusion

Efficiency of procedures is sensitive to problem

configuration and δ.

NHH tends to cost significantly fewer simulation

replications then parallel NSGS in many instances, but NSGS guarantees good selection.

A new procedure with provable good selection guarantee.

◮ Saves simulation effort in some cases. ◮ Extends to cloud computing platforms. Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 16/18

SLIDE 22

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

Thank you! Questions?

17/18

SLIDE 23

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

Sketch of Proof: Theorem 1

Let B∆ denote a Brownian motion with drift ∆.

Lemma (Hong (2006), Theorem 1)

Let m(r) and n(r) be nondecreasing functions of r = 1, 2, . . . and i, i′ be any two systems. Define Z(m, n) =

σ2

i /m + σ2 i′/n

−1 [ ¯

Xi(m) − ¯ Xi′(n)] and Z ′(m, n) = Bµi−µi′([σ2

i /m + σ2 i′/n]−1). Then the random

sequences Z(m(r), n(r)) and Z ′(m(r), n(r)) have the same joint distribution.

Lemma

P[min0≤t≤T B0(t) < −A] = 2P[B0(T) < −A] = 2¯ Φ(A/ √ T) for all A, T > 0. Based on the above, we can design procedures to limit the probability of eliminating the best system in Stage 1.

18/18

SLIDE 24

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

References I

V. Fabian. Note on anderson’s sequential procedures with

triangular boundary. Annals of Statistics, 2(1):170–176, 1974.

L. Jeff Hong. Fully sequential indifference-zone selection

procedures with variance-dependent sampling. Naval Research Logistics (NRL), 53(5):464–476, 2006.

K. Jamieson and R. Nowak. Best-arm identification algorithms

for multi-armed bandits in the fixed confidence setting. In Information Sciences and Systems (CISS), 2014 48th Annual Conference on, pages 1–6, 2014. ID: 1. Kevin Jamieson, Matthew Malloy, Robert Nowak, and SÃľbastien Bubeck. lil’ucb: An optimal exploration algorithm for multi-armed bandits. arXiv preprint arXiv:1312.7308, 2013.

References 1/4

SLIDE 25

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

References II

Seong-Hee Kim and Barry L. Nelson. A fully sequential procedure for indifference-zone selection in simulation. ACM Transactions on Modeling and Computer Simulation, 11(3): 251–273, 2001. Seong-Hee Kim and Barry L. Nelson. On the asymptotic validity of fully sequential selection procedures for steady-state simulation. Operations Research, 54(3):475–488, 2006. Jun Luo, Jeff L. Hong, Barry L. Nelson, and Yang Wu. Fully sequential procedures for large-scale ranking-and-selection problems in parallel computing environments. Working Paper, 2013.

References 2/4

SLIDE 26

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

References III

Yuh-Chuyn Luo, Chun-Hung Chen, E. Yucesan, and Insup Lee. Distributed web-based simulation optimization. In Proceedings of the 2000 Winter Simulation Conference, volume 2, pages 1785–1793, 2000.

B. L. Nelson and F. J. Matejcik. Using common random

numbers for indifference-zone selection and multiple comparisons in simulation. Management Science, 41(12): 1935–1945, 1995. Barry L. Nelson, Julie Swann, David Goldsman, and Wheyming

Song. Simple procedures for selecting the best simulated

system when the number of alternatives is large. Operations Research, 49(6):950–963, 2001.

References 3/4

SLIDE 27

Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures

References IV

Eric C. Ni, Susan R. Hunter, and Shane G. Henderson. Ranking and selection in a high performance computing

environment. In Proceedings of the 2013 Winter Simulation

Conference, pages 833–845, 2013.

E. Paulson. A sequential procedure for selecting the population

with the largest mean from k normal populations. Annals of Mathematical Statistics, 35(1):174–180, 1964. Yosef Rinott. On two-stage selection procedures and related probability-inequalities. Communications in Statistics - Theory and Methods, 7(8):799–811, 1978. Taejong Yoo, Hyunbo Cho, and Enver Yücesan. Web services-based parallel replicated discrete event simulation for large-scale simulation optimization. Simulation, 85(7): 461–475, July 2009.

References 4/4