Estimation of Skill Distribution from a Tournament Ali Jadbabaie, - - PowerPoint PPT Presentation

▶
estimation of skill distribution from a tournament
SMART_READER_LITE
LIVE PREVIEW

Estimation of Skill Distribution from a Tournament Ali Jadbabaie, - - PowerPoint PPT Presentation

Dec 28, 2022 386 likes •630 views

Estimation of Skill Distribution from a Tournament Ali Jadbabaie, Anuran Makur, and Devavrat Shah Laboratory for Information & Decision Systems Massachusetts Institute of Technology Conference on Neural Information Processing Systems

slide-1
SLIDE 1

Estimation of Skill Distribution from a Tournament

Ali Jadbabaie, Anuran Makur, and Devavrat Shah

Laboratory for Information & Decision Systems Massachusetts Institute of Technology

Conference on Neural Information Processing Systems (NeurIPS) 6-12 December 2020

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 1 / 12

slide-2
SLIDE 2

Outline

1

Introduction Motivation and Goal Experiments

2

Contributions

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 2 / 12

slide-3
SLIDE 3

Motivation and Goal

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 3 / 12

slide-4
SLIDE 4

Motivation and Goal

Can we measure the level of skill in a game based on win-loss data from tournaments?

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 3 / 12

slide-5
SLIDE 5

Cricket World Cups

Estimated Skill Densities from Tournament Data

skill value ,

  • 0.2

0.2 0.4 0.6 0.8 1 1.2

skill PDF P,

0.5 1 1.5 2 2.5 3 3.5 4

2003 2007 2011 2015 2019

Negative Differential Entropies

  • f Estimated Skill Densities

time (years)

2004 2006 2008 2010 2012 2014 2016 2018

negative entropy !h(P,)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 4 / 12

slide-6
SLIDE 6

Cricket World Cups

Estimated Skill Densities from Tournament Data

skill value ,

  • 0.2

0.2 0.4 0.6 0.8 1 1.2

skill PDF P,

0.5 1 1.5 2 2.5 3 3.5 4

2003 2007 2011 2015 2019

Negative Differential Entropies

  • f Estimated Skill Densities

time (years)

2004 2006 2008 2010 2012 2014 2016 2018

negative entropy !h(P,)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Entropy skill score: Measures holistic variation of skill levels of teams High score = more “luck”, low score = more skill

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 4 / 12

slide-7
SLIDE 7

Cricket World Cups

Estimated Skill Densities from Tournament Data

skill value ,

  • 0.2

0.2 0.4 0.6 0.8 1 1.2

skill PDF P,

0.5 1 1.5 2 2.5 3 3.5 4

2003 2007 2011 2015 2019

Negative Differential Entropies

  • f Estimated Skill Densities

time (years)

2004 2006 2008 2010 2012 2014 2016 2018

negative entropy !h(P,)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Entropy skill score: Measures holistic variation of skill levels of teams High score = more “luck”, low score = more skill Observation: Skill scores of cricket world cup tournaments is decreasing

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 4 / 12

slide-8
SLIDE 8

Soccer World Cups

Estimated Skill Densities from Tournament Data

skill value ,

  • 0.2

0.2 0.4 0.6 0.8 1 1.2

skill PDF P,

0.5 1 1.5 2 2.4 3 3.5 4 4.5 5

2002 2006 2010 2014 2018

Negative Differential Entropies

  • f Estimated Skill Densities

time (years)

2002 2004 2006 2008 2010 2012 2014 2016 2018

negative entropy !h(P,)

0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 5 / 12

slide-9
SLIDE 9

Soccer World Cups

Estimated Skill Densities from Tournament Data

skill value ,

  • 0.2

0.2 0.4 0.6 0.8 1 1.2

skill PDF P,

0.5 1 1.5 2 2.4 3 3.5 4 4.5 5

2002 2006 2010 2014 2018

Negative Differential Entropies

  • f Estimated Skill Densities

time (years)

2002 2004 2006 2008 2010 2012 2014 2016 2018

negative entropy !h(P,)

0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5

Observation: Soccer world cups have remained unpredictable over the years

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 5 / 12

slide-10
SLIDE 10

Soccer Leagues in 2018-2019

Estimated Skill Densities from Tournament Data

skill value ,

  • 0.2

0.2 0.4 0.6 0.8 1 1.2

skill PDF P,

0.5 1 1.5 2 2.4 3 3.5 4 4.5 5

English Premier League Spanish La Liga German Bundesliga French Ligue 1 Italian Serie A FIFA World Cup 2018

Negative Differential Entropies

  • f Estimated Skill Densities

Soccer leagues in 2018-2019

World English Spanish German French Italian

negative entropy !h(P,)

0.2 0.4 0.6 0.8 1 1.2

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 6 / 12

slide-11
SLIDE 11

Soccer Leagues in 2018-2019

Estimated Skill Densities from Tournament Data

skill value ,

  • 0.2

0.2 0.4 0.6 0.8 1 1.2

skill PDF P,

0.5 1 1.5 2 2.4 3 3.5 4 4.5 5

English Premier League Spanish La Liga German Bundesliga French Ligue 1 Italian Serie A FIFA World Cup 2018

Negative Differential Entropies

  • f Estimated Skill Densities

Soccer leagues in 2018-2019

World English Spanish German French Italian

negative entropy !h(P,)

0.2 0.4 0.6 0.8 1 1.2

Observation: Recover ranking of soccer leagues that is consistent with fan experience

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 6 / 12

slide-12
SLIDE 12

US Mutual Funds

Estimated Skill Densities from Tournament Data

skill value ,

0.2 0.4 0.6 0.8 1

skill PDF P,

1 2 3 4 5 6 7 8

2005 2006 2007 2008 2009 2010 2011 skill value ,

0.2 0.4 0.6 0.8 1

skill PDF P,

1 2 3 4 5 6 7 8

2012 2013 2014 2015 2016 2017 2018

Negative Entropies of Estimated Skill Densities

time (years)

2006 2008 2010 2012 2014 2016 2018

negative entropy !h(P,)

0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 7 / 12

slide-13
SLIDE 13

US Mutual Funds

Estimated Skill Densities from Tournament Data

skill value ,

0.2 0.4 0.6 0.8 1

skill PDF P,

1 2 3 4 5 6 7 8

2005 2006 2007 2008 2009 2010 2011 skill value ,

0.2 0.4 0.6 0.8 1

skill PDF P,

1 2 3 4 5 6 7 8

2012 2013 2014 2015 2016 2017 2018

Negative Entropies of Estimated Skill Densities

time (years)

2006 2008 2010 2012 2014 2016 2018

negative entropy !h(P,)

0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4

Observation: Skill score is minimum during the Great Recession in 2008

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 7 / 12

slide-14
SLIDE 14

Outline

1

Introduction

2

Contributions Formal Setup Estimation Algorithm Theoretical Results

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 8 / 12

slide-15
SLIDE 15

Formal Setup

Unknown probability density of skill levels Pα on R+ ✶ ✶

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 9 / 12

slide-16
SLIDE 16

Formal Setup

Unknown probability density of skill levels Pα on R+ Teams {1, . . . , n} play tournament with unknown i.i.d. skill levels α1, . . . , αn ∼ Pα ✶ ✶

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 9 / 12

slide-17
SLIDE 17

Formal Setup

Unknown probability density of skill levels Pα on R+ Teams {1, . . . , n} play tournament with unknown i.i.d. skill levels α1, . . . , αn ∼ Pα For any teams i = j, with probability p ∈ (0, 1], observe k independent pairwise games Z1(i, j), . . . , Zk(i, j), where Zm(i, j) = ✶{j beats i in mth game} ✶

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 9 / 12

slide-18
SLIDE 18

Formal Setup

Unknown probability density of skill levels Pα on R+ Teams {1, . . . , n} play tournament with unknown i.i.d. skill levels α1, . . . , αn ∼ Pα For any teams i = j, with probability p ∈ (0, 1], observe k independent pairwise games Z1(i, j), . . . , Zk(i, j), where Zm(i, j) = ✶{j beats i in mth game} Bradley-Terry-Luce (BTL) or multinomial logit model [BT52,Luc59,McF73]: P(Zm(i, j) = 1 | α1, . . . , αn) = αj αi + αj ✶

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 9 / 12

slide-19
SLIDE 19

Formal Setup

Unknown probability density of skill levels Pα on R+ Teams {1, . . . , n} play tournament with unknown i.i.d. skill levels α1, . . . , αn ∼ Pα For any teams i = j, with probability p ∈ (0, 1], observe k independent pairwise games Z1(i, j), . . . , Zk(i, j), where Zm(i, j) = ✶{j beats i in mth game} Bradley-Terry-Luce (BTL) or multinomial logit model [BT52,Luc59,McF73]: P(Zm(i, j) = 1 | α1, . . . , αn) = αj αi + αj Goal: Learn Pα from observation matrix Z ∈ [0, 1]n×n with Z(i, j) =      ✶{games observed between i, j} 1 k

k

  • m=1

Zm(i, j) , i = j 0 , i = j

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 9 / 12

slide-20
SLIDE 20

Estimation Algorithm

Assume Pα is bounded, in an η-H¨

  • lder class, and has support in [δ, 1].

Algorithm Estimating Pα from Z Input: Observation matrix Z Output: Estimator P∗ of unknown Pα

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 10 / 12

slide-21
SLIDE 21

Estimation Algorithm

Assume Pα is bounded, in an η-H¨

  • lder class, and has support in [δ, 1].

Algorithm Estimating Pα from Z Input: Observation matrix Z Output: Estimator P∗ of unknown Pα Step 1: Skill parameter estimation using rank centrality algorithm [NOS12,NOS17]

1: Construct stochastic matrix S ∈ Rn×n with S(i, j) = Z(i,j)

2np

for i = j, whose rows sum to 1

2: Compute leading left eigenvector ˆ

π∗ of S such that ˆ π∗ = ˆ π∗S

3: Compute skill level estimates ˆ

αi =

ˆ π∗(i) ˆ π∗∞ for i = 1, . . . , n

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 10 / 12

slide-22
SLIDE 22

Estimation Algorithm

Assume Pα is bounded, in an η-H¨

  • lder class, and has support in [δ, 1].

Algorithm Estimating Pα from Z Input: Observation matrix Z Output: Estimator P∗ of unknown Pα Step 1: Skill parameter estimation using rank centrality algorithm [NOS12,NOS17]

1: Construct stochastic matrix S ∈ Rn×n with S(i, j) = Z(i,j)

2np

for i = j, whose rows sum to 1

2: Compute leading left eigenvector ˆ

π∗ of S such that ˆ π∗ = ˆ π∗S

3: Compute skill level estimates ˆ

αi =

ˆ π∗(i) ˆ π∗∞ for i = 1, . . . , n

Step 2: Kernel density estimation using Parzen-Rosenblatt method [Ros56,Par62]

4: Compute bandwidth h = Θ

  • log(n)

1 2η+2 n− 1 2η+2

5: Construct

P∗ using appropriate kernel K : [−1, 1] → R: P∗(x)

1 nh

n

i=1 K

ˆ

αi−x h

  • 6: return

P∗

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 10 / 12

slide-23
SLIDE 23

Theoretical Results

Summary of Minimax Estimation Results: (in red) Estimation problem Loss function Upper bound Lower bound Smooth skill density mean squared error ˜ O(n−1+ε) Ω(n−1) [IK82,Tsy09] BTL skill parameters relative ℓ∞-norm ˜ O(n−1/2) [CFMW19] ˜ Ω(n−1/2) BTL skill parameters ℓ1-norm O(n−1/2) [CFMW19] ˜ Ω(n−1/2) Note: ˜ O and ˜ Ω hide polylog(n) terms, and ε > 0 is any arbitrarily small constant.

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 11 / 12

slide-24
SLIDE 24

Thank You!

  • A. Jadbabaie, A. Makur, D. Shah (MIT)

Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 12 / 12