Neural Networks for Predicting Algorithm Runtime Distributions - - PowerPoint PPT Presentation

▶
neural networks for predicting algorithm runtime
SMART_READER_LITE
LIVE PREVIEW

Neural Networks for Predicting Algorithm Runtime Distributions - - PowerPoint PPT Presentation

Jan 18, 2023 301 likes •594 views

Neural Networks for Predicting Algorithm Runtime Distributions Katharina Eggensperger, Marius Lindauer & Frank Hutter Paper ID #2772 Eggensperger, Lindauer and Hutter DistNet: Runtime Distribution Prediction #2772 IJCAI2018 Motivation

slide-1
SLIDE 1

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Neural Networks for Predicting Algorithm Runtime Distributions

Katharina Eggensperger, Marius Lindauer & Frank Hutter Paper ID #2772

slide-2
SLIDE 2

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Motivation

Algorithm portfolios yield state-of-the-art performance for SAT, ASP, Planning, … → to build these we can make use of runtime predictions Other applications:

  • Optimal restarts
  • Algorithm selection
  • Algorithm configurations

2

slide-3
SLIDE 3

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Describing the Runtime of an Algorithm?

3

solve(instance, seed): # do something return solution, runtime runtime

slide-4
SLIDE 4

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Describing the Runtime of an Algorithm?

3

solve(instance, seed): # do something return solution, runtime runtime

slide-5
SLIDE 5

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Contributions

1 Study how to predict parametric RTDs 2 Propose DistNet, a practical neural network for predicting RTDs 3 Evaluate DistNet and show that it can learn from only a few samples per instance

4

slide-6
SLIDE 6

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Typical Pipeline for Runtime prediction

5

slide-7
SLIDE 7

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Typical Pipeline for Runtime prediction

5

slide-8
SLIDE 8

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Typical Pipeline for Runtime prediction

5

slide-9
SLIDE 9

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Typical Pipeline for Runtime prediction

5

slide-10
SLIDE 10

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Typical Pipeline for Runtime prediction

6

slide-11
SLIDE 11

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Empirical RTDs

7

Clasp-factoring LPG-Zenotravel SAPS-CV-VAR p(solved by this time)

slide-12
SLIDE 12

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Typical Pipeline for Runtime prediction

8

slide-13
SLIDE 13

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Considered Parametric Distribution

9

slide-14
SLIDE 14

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Quantifying the Quality of Runtime Distributions

10

(1)

  • bserved runtimes

distribution parameter

slide-15
SLIDE 15

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Quantifying the Quality of Runtime Distributions

10

(1) (2)

  • bserved runtimes

distribution parameter

slide-16
SLIDE 16

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Typical Pipeline for Runtime prediction

11

slide-17
SLIDE 17

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Predicting multiple Runtime Distributions

Option 1

For each training instance → fit the parametric distribution’s parameter on observed runtimes. Then for all training instances, for each distribution parameter: fit a model

12

slide-18
SLIDE 18

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Predicting multiple Runtime Distributions

Option 1

For each training instance → fit the parametric distribution’s parameter on observed runtimes. Then for all training instances, for each distribution parameter: fit a model

12

Problematic, because models

  • can only be as good as each fitted distribution
  • do not know about interaction between their outputs
  • typically minimize loss in the parameter space
slide-19
SLIDE 19

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Predicting multiple Runtime Distributions

Option 2

For each training instance → fit the parametric distribution’s parameter on observed runtimes. Then for all training instances, for each distribution parameter: fit a model with multiple outputs

12

Problematic, because model

  • can only be as good as each fitted distribution
  • does not know about interaction between their outputs
  • typically minimizes loss in the parameter space
slide-20
SLIDE 20

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Predicting multiple Runtime Distributions

DistNet

For each training instance → fit the parametric distribution’s parameter on observed runtimes. Then for all training instances, for each distribution parameter: fit a neural network using negative log-likelihood as a loss function

12

slide-21
SLIDE 21

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Results

We compared

  • DistNet
  • independent Random Forests (iRF)
  • multi-output Random Forests (mRF)
  • n 7 scenarios from SAT solving and AI

planning.

16 Table: Averaged negative log-likelihood achieved for predicting RTDs for unseen instances

[...]

Figure: Averaged negative log-likelihood. Smaller values are better.

slide-22
SLIDE 22

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Results

We compared

  • DistNet
  • independent Random Forests (iRF)
  • multi-output Random Forests (mRF)
  • n 7 scenarios from SAT solving and AI

planning.

16 Table: Averaged negative log-likelihood achieved for predicting RTDs for unseen instances

[...]

Figure: Averaged negative log-likelihood. Smaller values are better.

slide-23
SLIDE 23

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Results

We compared

  • DistNet
  • independent Random Forests (iRF)
  • multi-output Random Forests (mRF)
  • n 7 scenarios from SAT solving and AI

planning. → Predicting parameters for RTDs is possible → Joint predictions work better → DistNet provides more robust predictions which are often better than those of competitors

16 Table: Averaged negative log-likelihood achieved for predicting RTDs for unseen instances

[...]

Figure: Averaged negative log-likelihood. Smaller values are better.

slide-24
SLIDE 24

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

DistNet on a Low Number of Observations

17

averaged NLLH

multi-ouput Random Forest Distribution fitted on all samples

DistNet

slide-25
SLIDE 25

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Wrap-Up

We have proposed DistNet, which + jointly learns distribution parameters + directly optimizes the loss function of interest + performs well even if only few observations per instance are available

18

slide-26
SLIDE 26

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Wrap-Up

We have proposed DistNet, which + jointly learns distribution parameters + directly optimizes the loss function of interest + performs well even if only few observations per instance are available Open Questions:

  • How to automatically determine a well fitting distribution family?
  • How to handle heterogeneous datasets?

18

slide-27
SLIDE 27

DistNet: Runtime Distribution Prediction #2772 Eggensperger, Lindauer and Hutter IJCAI’2018

Wrap-Up

We have proposed DistNet, which + jointly learns distribution parameters + directly optimizes the loss function of interest + performs well even if only few observations per instance are available Open Questions:

  • How to automatically determine a well fitting distribution family?
  • How to handle heterogeneous datasets?

Code and data: https://www.automl.org/distnet/

18