Quantifying Algorithmic Improvements Lars Kotthofg University of - - PowerPoint PPT Presentation

Quantifying Algorithmic Improvements

Lars Kotthofg University of Wyoming larsko@uwyo.edu1 Leiden, 10 July 2018

1joint work with Alexandre Fréchette, Tomasz Michalak, Talal Rahwan,

Holger H. Hoos, Kevin Leyton-Brown

Motivation

▷ you’re blending a great new cheese and make lots of money ▷ you want to distribute the money fairly to the producers of the constituent cheeses cheese A cheese B cheese C volume fraction in blend 0.2 0.45 0.35 cost fraction in blend 0.5 0.1 0.4 popularity 10 3 5 average fractions? 0.35 0.275 0.375 popularity times volume? 2 1.35 1.75 popularity times price? 5 0.3 2 How much worse would the blend be without each cheese? Would you still have made lots of money? What if one of the constituent cheeses is based on another constituent cheese?

1

Motivation

▷ you’re blending a great new cheese and make lots of money ▷ you want to distribute the money fairly to the producers of the constituent cheeses cheese A cheese B cheese C volume fraction in blend 0.2 0.45 0.35 cost fraction in blend 0.5 0.1 0.4 popularity 10 3 5 average fractions? 0.35 0.275 0.375 popularity times volume? 2 1.35 1.75 popularity times price? 5 0.3 2 How much worse would the blend be without each cheese? Would you still have made lots of money? What if one of the constituent cheeses is based on another constituent cheese?

1

Motivation

▷ you’re blending a great new cheese and make lots of money ▷ you want to distribute the money fairly to the producers of the constituent cheeses cheese A cheese B cheese C volume fraction in blend 0.2 0.45 0.35 cost fraction in blend 0.5 0.1 0.4 popularity 10 3 5 average fractions? 0.35 0.275 0.375 popularity times volume? 2 1.35 1.75 popularity times price? 5 0.3 2 How much worse would the blend be without each cheese? Would you still have made lots of money? What if one of the constituent cheeses is based on another constituent cheese?

1

Motivation

▷ you’re blending a great new cheese and make lots of money ▷ you want to distribute the money fairly to the producers of the constituent cheeses cheese A cheese B cheese C volume fraction in blend 0.2 0.45 0.35 cost fraction in blend 0.5 0.1 0.4 popularity 10 3 5 average fractions? 0.35 0.275 0.375 popularity times volume? 2 1.35 1.75 popularity times price? 5 0.3 2 How much worse would the blend be without each cheese? Would you still have made lots of money? What if one of the constituent cheeses is based on another constituent cheese?

1

Motivation

▷ you’re blending a great new cheese and make lots of money ▷ you want to distribute the money fairly to the producers of the constituent cheeses cheese A cheese B cheese C volume fraction in blend 0.2 0.45 0.35 cost fraction in blend 0.5 0.1 0.4 popularity 10 3 5 average fractions? 0.35 0.275 0.375 popularity times volume? 2 1.35 1.75 popularity times price? 5 0.3 2 How much worse would the blend be without each cheese? Would you still have made lots of money? What if one of the constituent cheeses is based on another constituent cheese?

1

Motivation

▷ you’re blending a great new cheese and make lots of money ▷ you want to distribute the money fairly to the producers of the constituent cheeses cheese A cheese B cheese C volume fraction in blend 0.2 0.45 0.35 cost fraction in blend 0.5 0.1 0.4 popularity 10 3 5 average fractions? 0.35 0.275 0.375 popularity times volume? 2 1.35 1.75 popularity times price? 5 0.3 2 How much worse would the blend be without each cheese? Would you still have made lots of money? What if one of the constituent cheeses is based on another constituent cheese?

1

Motivation

▷ you’re blending a great new cheese and make lots of money ▷ you want to distribute the money fairly to the producers of the constituent cheeses cheese A cheese B cheese C volume fraction in blend 0.2 0.45 0.35 cost fraction in blend 0.5 0.1 0.4 popularity 10 3 5 average fractions? 0.35 0.275 0.375 popularity times volume? 2 1.35 1.75 popularity times price? 5 0.3 2 How much worse would the blend be without each cheese? Would you still have made lots of money? What if one of the constituent cheeses is based on another constituent cheese?

1

Motivation

▷ you’re blending a great new cheese and make lots of money ▷ you want to distribute the money fairly to the producers of the constituent cheeses cheese A cheese B cheese C volume fraction in blend 0.2 0.45 0.35 cost fraction in blend 0.5 0.1 0.4 popularity 10 3 5 average fractions? 0.35 0.275 0.375 popularity times volume? 2 1.35 1.75 popularity times price? 5 0.3 2 How much worse would the blend be without each cheese? Would you still have made lots of money? What if one of the constituent cheeses is based on another constituent cheese?

1

Motivation

▷ you’re blending a great new cheese and make lots of money ▷ you want to distribute the money fairly to the producers of the constituent cheeses cheese A cheese B cheese C volume fraction in blend 0.2 0.45 0.35 cost fraction in blend 0.5 0.1 0.4 popularity 10 3 5 average fractions? 0.35 0.275 0.375 popularity times volume? 2 1.35 1.75 popularity times price? 5 0.3 2 How much worse would the blend be without each cheese? Would you still have made lots of money? What if one of the constituent cheeses is based on another constituent cheese?

1

Analyzing Algorithms – Setting

▷ sorting lists with quicksort algorithm ▷ difgerent methods for choosing pivot, which partitions the unsorted list ▷ measure time to sort list ▷ score proportional to speed

2

Contributions – Standalone Performance

798602199 798501630 798470169 798466233 798461169 798360514 794178118 784476788 671833

dual pivot (2009) median 9 (1993) median 9 random (1993) mid (1978) median 3 random (1978) random (1961) median 3 (1978) first (1961) insertion (1946) dual pivot (2009) median 9 (1993) median 9 random (1993) mid (1978) median 3 random (1978) random (1961) median 3 (1978) first (1961) insertion (1946)

Standalone Performance

3

Contributions – Standalone Performance

798602199 798501630 798470169 798466233 798461169 798360514 794178118 784476788 671833

dual pivot (2009) median 9 (1993) median 9 random (1993) mid (1978) median 3 random (1978) random (1961) median 3 (1978) first (1961) insertion (1946) dual pivot (2009) median 9 (1993) median 9 random (1993) mid (1978) median 3 random (1978) random (1961) median 3 (1978) first (1961) insertion (1946)

Standalone Performance

How well do they complement each other?

3

Contributions – Marginal Performance

How much does an algorithm contribute to the state of the art (defjned by a coalition of all other algorithms)?

φi = v(Ci ∪ {i}) − v(Ci)

Xu, Hutter, Hoos, Leyton-Brown. “Evaluating Component Solver Contributions to Portfolio-Based Algorithm Selectors.” SAT 2012

4

Contributions – Marginal Performance

798602199 798501630 798470169 798466233 798461169 798360514 794178118 784476788 671833 98900 18 5 5 3 1

dual pivot (2009) median 9 (1993) median 9 random (1993) mid (1978) median 3 random (1978) random (1961) median 3 (1978) first (1961) insertion (1946) dual pivot (2009) median 9 (1993) median 3 random (1978) median 9 random (1993) mid (1978) median 3 (1978) first (1961) insertion (1946) random (1961)

Standalone Performance Marginal Performance 5

Contributions – Marginal Performance

798602199 798501630 798470169 798466233 798461169 798360514 794178118 784476788 671833 98900 18 5 5 3 1

dual pivot (2009) median 9 (1993) median 9 random (1993) mid (1978) median 3 random (1978) random (1961) median 3 (1978) first (1961) insertion (1946) dual pivot (2009) median 9 (1993) median 3 random (1978) median 9 random (1993) mid (1978) median 3 (1978) first (1961) insertion (1946) random (1961)

Standalone Performance Marginal Performance

…most get almost nothing?

5

Shapley Value

How much does an algorithm contribute to all possible coalitions

• f other algorithms?

φi = 1 |Π| ∑

π∈ΠN

v(Cπ

i ∪ {i}) − v(Cπ i )

We can compute this in polynomial time.

• Shapley. “A Value for n-person Games.” In Contributions to the Theory of

Games, 1953. Fréchette, Alexandre, Lars Kotthofg, Talal Rahwan, Holger H. Hoos, Kevin Leyton-Brown, and Tomasz P. Michalak. “Using the Shapley Value to Analyze Algorithm Portfolios.” In 30th AAAI Conference on Artifjcial Intelligence, 2016.

6

Properties

Effjciency The total value is distributed among algorithms. Dummy An algorithm that make no contribution in any case does not have any value. Symmetry Identical algorithms have the same value. Additivity The sum of values of an algorithm under two difgerent performance measures is the same as its value under a combined measure.

7

Contributions – Shapley Value

798602199 798501630 798470169 798466233 798461169 798360514 794178118 784476788 671833 100267058 100167412 100153715 100153384 100151097 100131186 99434662 98059604 84173 98900 18 5 5 3 1

dual pivot (2009) median 9 (1993) median 9 random (1993) mid (1978) median 3 random (1978) random (1961) median 3 (1978) first (1961) insertion (1946) dual pivot (2009) median 9 (1993) median 3 random (1978) median 9 random (1993) mid (1978) median 3 (1978) first (1961) insertion (1946) random (1961)

Standalone Performance Shapley Value Marginal Performance

8

Contributions – Shapley Value

798602199 798501630 798470169 798466233 798461169 798360514 794178118 784476788 671833 100267058 100167412 100153715 100153384 100151097 100131186 99434662 98059604 84173 98900 18 5 5 3 1

dual pivot (2009) median 9 (1993) median 9 random (1993) mid (1978) median 3 random (1978) random (1961) median 3 (1978) first (1961) insertion (1946) dual pivot (2009) median 9 (1993) median 3 random (1978) median 9 random (1993) mid (1978) median 3 (1978) first (1961) insertion (1946) random (1961)

Standalone Performance Shapley Value Marginal Performance

…but later algorithms were developed based on earlier ones.

8

Contributions – Temporal Marginal Performance

How much does an algorithm contribute to the state of the art (defjned by a coalition of all other algorithms available at the time)?

φ≻

i = v≻(Ci ∪ {i}) − v≻(Ci)

where ≻ is a relation that encodes temporal precedence.

9

Contributions – Temporal Marginal Performance

798602199 798501630 798470169 798466233 798461169 798360514 794178118 784476788 671833 13212030 671833 98900 20497 15703 6907 552 541 137

dual pivot (2009) median 9 (1993) median 9 random (1993) mid (1978) median 3 random (1978) random (1961) median 3 (1978) first (1961) insertion (1946) random (1961) insertion (1946) dual pivot (2009) median 9 (1993) mid (1978) median 3 random (1978) median 3 (1978) median 9 random (1993) first (1961)

Standalone Performance Temporal Marginal Performance

10

Temporal Shapley Value

How much does an algorithm contribute to all possible coalitions

• f other algorithms, taking temporal precedence into account?

φ≻

i =

1 |Π≻| ∑

π∈Π≻

v≻(Cπ

i ∪ {i}) − v≻(Cπ i )

We can compute this in polynomial time as well.

Kotthofg, Lars, Alexandre Fréchette, Tomasz P. Michalak, Talal Rahwan, Holger H. Hoos, and Kevin Leyton-Brown. “Quantifying Algorithmic Improvements over Time.” In 27th International Joint Conference on Artifjcial Intelligence (IJCAI) Special Track on the Evolution of the Contours of AI, 2018.

11

Contributions – Temporal Shapley Value

798602199 798501630 798470169 798466233 798461169 798360514 794178118 784476788 671833 100267058 100167412 100153715 100153384 100151097 100131186 99434662 98059604 84173 405450356 392238462 671833 98900 57198 50411 22506 10074 2550 13212030 671833 98900 20497 15703 6907 552 541 137

dual pivot (2009) median 9 (1993) median 9 random (1993) mid (1978) median 3 random (1978) random (1961) median 3 (1978) first (1961) insertion (1946) random (1961) insertion (1946) dual pivot (2009) median 9 (1993) mid (1978) median 3 random (1978) median 3 (1978) median 9 random (1993) first (1961)

Standalone Performance Shapley Value Temporal Shapley Value Temporal Marginal Performance

12

Quicksort Over Time

1e+02 1e+05 1e+08 1946 1961 1978 1993 2009

year Sum of temporal Shapley values

13

SAT Competition

• 144.25268

144.08601 139.91915 57.41773 56.33454 51.75097 44.43418 43.43408 35.08748 28.8338 27.08395 19.16712 19.13952 16.3337 12.43352 10.19402 9.69765 9.0321 8.30224 8.0001 7.75017 7.03675 6.64388 4.87001 4.50018 4.26672 4.26672 3.91678 2.00002 2.00001 1.83337 1.16668 0.83336 4e−05 78.42398 63.90765 55.75744 55.15192 52.43065 50.72959 45.33413 44.61692 44.50427 36.26344 31.60638 30.53198 30.41135 28.4814 25.76449 21.82523 20.7125 20.49854 20.15654 19.71357 18.71084 17.82205 16.86642 16.31361 15.36641 14.86641 14.18306 13.18303 9.83857 8.74145 4.81501 3.56499 1.90815 1.88997 0.53389 0.45055 0.14286

gnovelty+_2007 ranov_2007 adaptnovelty_2007 TNM_2009 sparrow2011_2011 sapsrt_2007 March−KS_2007 KCNFS_2007 dimetheus_2.100_2014 hybridGM3_2009 iPAWS_2009 sattime2011_2011 BalancedZ_2014 adaptg2wsat2011_2011 DEWSATZ−1A_2007 CSCCSat2014_SC2014_2014 CCgscore_2014 probSAT_sc14_2014 Ncca+_v1.05_2014 CSHCrandMC_2013 gnovelty+2_2009 YalSAT_03l_2014 CCA2014_2.0_2014 sattime_2014 MPhaseSAT_M−2011−02−16_2011 minisat−SAT_2007 MXC_2007 gNovelty+−T_2009 csls−pnorm−8cores_2011 march_br_sat+unsat_2013 march_rw−2011−03−02_2011 MiraXT−v3_2007 march_hi_2011 gNovelty+GCwa_1.0_2013 minipure_1.0.1_2013 Solver43a_a_2013 Solver43b_b_2013 strangenight_satcomp11−st_2013 dimetheus_2.100_2014 BalancedZ_2014 CCgscore_2014 CSCCSat2014_SC2014_2014 probSAT_sc14_2014 Ncca+_v1.05_2014 CCA2014_2.0_2014 sattime_2014 YalSAT_03l_2014 sparrow2011_2011 TNM_2009 sattime2011_2011 adaptg2wsat2011_2011 MPhaseSAT_M−2011−02−16_2011 CSHCrandMC_2013 ranov_2007 iPAWS_2009 gnovelty+_2007 adaptnovelty_2007 hybridGM3_2009 gnovelty+2_2009 gNovelty+−T_2009 march_br_sat+unsat_2013 gNovelty+GCwa_1.0_2013 march_rw−2011−03−02_2011 march_hi_2011 March−KS_2007 KCNFS_2007 csls−pnorm−8cores_2011 sapsrt_2007 DEWSATZ−1A_2007 minipure_1.0.1_2013 MXC_2007 minisat−SAT_2007 MiraXT−v3_2007 Solver43a_a_2013 Solver43b_b_2013 strangenight_satcomp11−st_2013 Temporal Shapley Value Shapley Value

14

SAT Competition Over Time

200 400 600 2007 2009 2011 2013 2014

year Sum of temporal Shapley Values

15

MiniZinc (CP) Competition

• 9987.86155

360.7717 1261.4996 29828.71058 4376.2985 922.81764 49647.2861 170.56476 21435.6316 5163.9305 197.036 16878.17865 12376.60633 193.93191 4775.12936 17831.07372 14097.37673 8886.77925 91361.24195 150.02742 20393.03799 192.78811 3267.92072 12513.1466 9828.08541 32189.36215 34970.99061 7059.47499 15552.43907 6190.81406 7087.25379 6756.47091 13073.56821 14389.56267 5262.55503 15095.99322 16977.5178 36090.70992 6071.40394 18324.39121 15746.33068 15810.6008 52.19571 6853.37461 11324.41791 Choco_2014 Choco_2016 Choco3_2015 Chuffed_2015 Chuffed_2016 Concrete_2016 G12Chuffed_2014 G12FD_2015 G12FD_2016 Gecode_2014 Gecode_2015 Gecode_2016 JaCoP_2014 JaCoP_2015 JaCoP_2016 LCG−Glucose_2016 OpturionCPX_2014 OpturionCPX_2015 OR−Tools_2015 ORTools_2014 PicatCP_2014 PicatCP_2016 SICStus_2014 SICStus_2016 Choco_2014 Choco_2016 Choco3_2015 Chuffed_2015 Chuffed_2016 Concrete_2016 G12Chuffed_2014 G12FD_2015 G12FD_2016 Gecode_2014 Gecode_2015 Gecode_2016 JaCoP_2014 JaCoP_2015 JaCoP_2016 LCG−Glucose_2016 OpturionCPX_2014 OpturionCPX_2015 OR−Tools_2015 ORTools_2014 PicatCP_2014 PicatCP_2016 SICStus_2014 SICStus_2016 Temporal Shapley Value Shapley Value

16

MiniZinc Competition Over Time

50000 100000 150000 200000 2014 2015 2016

year Sum of temporal Shapley Values

17

Summary

▷ standalone performance does not indicate how algorithms complement each other ▷ marginal performance is not fair ▷ Shapley Value

▷ provides better characterization of algorithms’ performance ▷ rewards algorithms that introduce novel and complementary concepts ▷ enables better analysis of algorithms’ performance

▷ Temporal Shapley Value

▷ takes when an algorithm was conceived into account ▷ all desirable properties of Shapley Value ▷ rewards earlier algorithms, which may have inspired later algorithms

18

I’m hiring!

Several funded graduate positions available.

19