Emprircal Performace Models B.A. Rachunok School of Industrial - - PowerPoint PPT Presentation

▶

Aug 27, 2023 6 likes •413 views

Emprircal Performace Models B.A. Rachunok School of Industrial Engineering Purdue University September 10, 2016 B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 1 / 12 History Early work by

SLIDE 1

Emprircal Performace Models

B.A. Rachunok

School of Industrial Engineering Purdue University

September 10, 2016

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 1 / 12

SLIDE 2

History

◮ Early work by Cheeseman et al. (1991) →Varied paramters and

looked at results

◮ Fink (1998) →Used regression to predict which of three algorithms

would work best

◮ Leyton-Brown et al. (2003) →Predicted runtimes for several solvers. ◮ I read the big papers by Leyton-Brown et al. and their work will be

the focus of this presentation

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 2 / 12

SLIDE 3

History

◮ Early work by Cheeseman et al. (1991) →Varied paramters and

looked at results

◮ Fink (1998) →Used regression to predict which of three algorithms

would work best

◮ Leyton-Brown et al. (2003) →Predicted runtimes for several solvers. ◮ I read the big papers by Leyton-Brown et al. and their work will be

the focus of this presentation

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 2 / 12

SLIDE 4

History

◮ Early work by Cheeseman et al. (1991) →Varied paramters and

looked at results

◮ Fink (1998) →Used regression to predict which of three algorithms

would work best

◮ Leyton-Brown et al. (2003) →Predicted runtimes for several solvers. ◮ I read the big papers by Leyton-Brown et al. and their work will be

the focus of this presentation

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 2 / 12

SLIDE 5

History

◮ Early work by Cheeseman et al. (1991) →Varied paramters and

looked at results

◮ Fink (1998) →Used regression to predict which of three algorithms

would work best

◮ Leyton-Brown et al. (2003) →Predicted runtimes for several solvers. ◮ I read the big papers by Leyton-Brown et al. and their work will be

the focus of this presentation

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 2 / 12

SLIDE 6

EPH Motivation

◮ We already have complexity theory to analyze algorithm performance. ◮ Why this too?

Consider

◮ TSP is O(n!) (if solved via brute force) ◮ However I can solve instances with 5000 points on my gross old

laptop?

◮ Imagine an algorithm with O(n4e10000) performance? ◮ . . . still polynomial

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 3 / 12

SLIDE 7

EPH Motivation

◮ We already have complexity theory to analyze algorithm performance. ◮ Why this too?

Consider

◮ TSP is O(n!) (if solved via brute force) ◮ However I can solve instances with 5000 points on my gross old

laptop?

◮ Imagine an algorithm with O(n4e10000) performance? ◮ . . . still polynomial

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 3 / 12

SLIDE 8

EPH Motivation

◮ We already have complexity theory to analyze algorithm performance. ◮ Why this too?

Consider

◮ TSP is O(n!) (if solved via brute force) ◮ However I can solve instances with 5000 points on my gross old

laptop?

◮ Imagine an algorithm with O(n4e10000) performance? ◮ . . . still polynomial

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 3 / 12

SLIDE 9

EPH Motivation

◮ We already have complexity theory to analyze algorithm performance. ◮ Why this too?

Consider

◮ TSP is O(n!) (if solved via brute force) ◮ However I can solve instances with 5000 points on my gross old

laptop?

◮ Imagine an algorithm with O(n4e10000) performance? ◮ . . . still polynomial

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 3 / 12

SLIDE 10

EPH Motivation

◮ We already have complexity theory to analyze algorithm performance. ◮ Why this too?

Consider

◮ TSP is O(n!) (if solved via brute force) ◮ However I can solve instances with 5000 points on my gross old

laptop?

◮ Imagine an algorithm with O(n4e10000) performance? ◮ . . . still polynomial

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 3 / 12

SLIDE 11

EPH Motivation

◮ We already have complexity theory to analyze algorithm performance. ◮ Why this too?

Consider

◮ TSP is O(n!) (if solved via brute force) ◮ However I can solve instances with 5000 points on my gross old

laptop?

◮ Imagine an algorithm with O(n4e10000) performance? ◮ . . . still polynomial

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 3 / 12

SLIDE 12

EPH Motivation

◮ We already have complexity theory to analyze algorithm performance. ◮ Why this too?

Consider

◮ TSP is O(n!) (if solved via brute force) ◮ However I can solve instances with 5000 points on my gross old

laptop?

◮ Imagine an algorithm with O(n4e10000) performance? ◮ . . . still polynomial

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 3 / 12

SLIDE 13

Motivation Pt 2

◮ Knowing how a particular instance of a problem will work gives us

some options

◮ Potentially select the ”best” predicted solver for a given instance

(more on this)

◮ Provide some insights into how solvers handle instances (eg 4.3

clauses-to-literal ratio)

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 4 / 12

SLIDE 14

Motivation Pt 2

◮ Knowing how a particular instance of a problem will work gives us

some options

◮ Potentially select the ”best” predicted solver for a given instance

(more on this)

◮ Provide some insights into how solvers handle instances (eg 4.3

clauses-to-literal ratio)

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 4 / 12

SLIDE 15

Motivation Pt 2

◮ Knowing how a particular instance of a problem will work gives us

some options

◮ Potentially select the ”best” predicted solver for a given instance

(more on this)

◮ Provide some insights into how solvers handle instances (eg 4.3

clauses-to-literal ratio)

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 4 / 12

SLIDE 16

Clause-to-Literal Ratio

Graph taken from Mitchell, Selman, Levesque 1992

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 5 / 12

SLIDE 17

Marker time

To the board

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 6 / 12

SLIDE 18

Types of Models

◮ Ridge Regression ◮ Neural Networks ◮ Gaussian Process Regression ◮ Regression Trees ◮ INSERT WHATEVER I USED HERE

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 7 / 12

SLIDE 19

Types of Models

◮ Ridge Regression ◮ Neural Networks ◮ Gaussian Process Regression ◮ Regression Trees ◮ INSERT WHATEVER I USED HERE

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 7 / 12

SLIDE 20

Types of Models

◮ Ridge Regression ◮ Neural Networks ◮ Gaussian Process Regression ◮ Regression Trees ◮ INSERT WHATEVER I USED HERE

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 7 / 12

SLIDE 21

Types of Models

◮ Ridge Regression ◮ Neural Networks ◮ Gaussian Process Regression ◮ Regression Trees ◮ INSERT WHATEVER I USED HERE

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 7 / 12

SLIDE 22

Types of Models

◮ Ridge Regression ◮ Neural Networks ◮ Gaussian Process Regression ◮ Regression Trees ◮ INSERT WHATEVER I USED HERE

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 7 / 12

SLIDE 23

Types of Features for MIP

◮ Number of constraints and variables ◮ Variable types ◮ ≤, ≥, or = for the RHS ◮ Mean of objective function coefficients

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 8 / 12

SLIDE 24

Types of Features for MIP

◮ Number of constraints and variables ◮ Variable types ◮ ≤, ≥, or = for the RHS ◮ Mean of objective function coefficients

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 8 / 12

SLIDE 25

Types of Features for MIP

◮ Number of constraints and variables ◮ Variable types ◮ ≤, ≥, or = for the RHS ◮ Mean of objective function coefficients

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 8 / 12

SLIDE 26

Types of Features for MIP

◮ Number of constraints and variables ◮ Variable types ◮ ≤, ≥, or = for the RHS ◮ Mean of objective function coefficients

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 8 / 12

SLIDE 27

Types of Features for TSP

◮ Number of nodes ◮ Cluster distance ◮ Area spanned by nodes ◮ Centroid of points ◮ Nearest Neighbor path length

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 9 / 12

SLIDE 28

Types of Features for TSP

◮ Number of nodes ◮ Cluster distance ◮ Area spanned by nodes ◮ Centroid of points ◮ Nearest Neighbor path length

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 9 / 12

SLIDE 29

Types of Features for TSP

◮ Number of nodes ◮ Cluster distance ◮ Area spanned by nodes ◮ Centroid of points ◮ Nearest Neighbor path length

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 9 / 12

SLIDE 30

Types of Features for TSP

◮ Number of nodes ◮ Cluster distance ◮ Area spanned by nodes ◮ Centroid of points ◮ Nearest Neighbor path length

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 9 / 12

SLIDE 31

Types of Features for TSP

◮ Number of nodes ◮ Cluster distance ◮ Area spanned by nodes ◮ Centroid of points ◮ Nearest Neighbor path length

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 9 / 12

SLIDE 32

Example Time

◮ now to the example

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 10 / 12

SLIDE 33

Who cares?

◮ Why do we care about this at all? ◮ Consider two prediction models for two solvers ◮ Consider three prediction models for three solvers ◮ . . . ◮ Consider n prediction models for n solvers

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 11 / 12

SLIDE 34

Who cares?

◮ Why do we care about this at all? ◮ Consider two prediction models for two solvers ◮ Consider three prediction models for three solvers ◮ . . . ◮ Consider n prediction models for n solvers

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 11 / 12

SLIDE 35

Who cares?

◮ Why do we care about this at all? ◮ Consider two prediction models for two solvers ◮ Consider three prediction models for three solvers ◮ . . . ◮ Consider n prediction models for n solvers

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 11 / 12

SLIDE 36

Who cares?

◮ Why do we care about this at all? ◮ Consider two prediction models for two solvers ◮ Consider three prediction models for three solvers ◮ . . . ◮ Consider n prediction models for n solvers

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 11 / 12

SLIDE 37

Who cares?

◮ Why do we care about this at all? ◮ Consider two prediction models for two solvers ◮ Consider three prediction models for three solvers ◮ . . . ◮ Consider n prediction models for n solvers

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 11 / 12

SLIDE 38

Who cares cont’d . . .

◮ If we can cheaply compute features and make predictions ◮ . . . then we can solve our instance with the best algorithm from a

portfolio

◮ eg SATZILLA

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 12 / 12

SLIDE 39

Who cares cont’d . . .

◮ If we can cheaply compute features and make predictions ◮ . . . then we can solve our instance with the best algorithm from a

portfolio

◮ eg SATZILLA

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 12 / 12

SLIDE 40

Who cares cont’d . . .

◮ If we can cheaply compute features and make predictions ◮ . . . then we can solve our instance with the best algorithm from a

portfolio

◮ eg SATZILLA

B.A. Rachunok (IE590 - Purdue University) Empirical Performance Models September 10, 2016 12 / 12