Estimation of Skill Distribution from a Tournament Ali Jadbabaie, Anuran Makur, and Devavrat Shah Laboratory for Information & Decision Systems Massachusetts Institute of Technology Conference on Neural Information Processing Systems (NeurIPS) 6-12 December 2020 A. Jadbabaie, A. Makur, D. Shah (MIT) Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 1 / 12
Outline Introduction 1 Motivation and Goal Experiments Contributions 2 A. Jadbabaie, A. Makur, D. Shah (MIT) Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 2 / 12
Motivation and Goal A. Jadbabaie, A. Makur, D. Shah (MIT) Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 3 / 12
Motivation and Goal Can we measure the level of skill in a game based on win-loss data from tournaments? A. Jadbabaie, A. Makur, D. Shah (MIT) Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 3 / 12
Cricket World Cups Estimated Skill Densities Negative Differential Entropies from Tournament Data of Estimated Skill Densities 4 0.9 2003 0.8 3.5 2007 negative entropy ! h ( P , ) 0.7 2011 3 2015 0.6 skill PDF P , 2.5 2019 0.5 2 0.4 1.5 0.3 1 0.2 0.5 0.1 0 0 -0.2 0 0.2 0.4 0.6 0.8 1 1.2 2004 2006 2008 2010 2012 2014 2016 2018 skill value , time (years) A. Jadbabaie, A. Makur, D. Shah (MIT) Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 4 / 12
Cricket World Cups Estimated Skill Densities Negative Differential Entropies from Tournament Data of Estimated Skill Densities 4 0.9 2003 0.8 3.5 2007 negative entropy ! h ( P , ) 0.7 2011 3 2015 0.6 skill PDF P , 2.5 2019 0.5 2 0.4 1.5 0.3 1 0.2 0.5 0.1 0 0 -0.2 0 0.2 0.4 0.6 0.8 1 1.2 2004 2006 2008 2010 2012 2014 2016 2018 skill value , time (years) Entropy skill score: Measures holistic variation of skill levels of teams High score = more “luck”, low score = more skill A. Jadbabaie, A. Makur, D. Shah (MIT) Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 4 / 12
Cricket World Cups Estimated Skill Densities Negative Differential Entropies from Tournament Data of Estimated Skill Densities 4 0.9 2003 0.8 3.5 2007 negative entropy ! h ( P , ) 0.7 2011 3 2015 0.6 skill PDF P , 2.5 2019 0.5 2 0.4 1.5 0.3 1 0.2 0.5 0.1 0 0 -0.2 0 0.2 0.4 0.6 0.8 1 1.2 2004 2006 2008 2010 2012 2014 2016 2018 skill value , time (years) Entropy skill score: Measures holistic variation of skill levels of teams High score = more “luck”, low score = more skill Observation: Skill scores of cricket world cup tournaments is decreasing A. Jadbabaie, A. Makur, D. Shah (MIT) Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 4 / 12
Soccer World Cups Estimated Skill Densities Negative Differential Entropies from Tournament Data of Estimated Skill Densities 5 1.5 2002 4.5 1.4 2006 negative entropy ! h ( P , ) 4 1.3 2010 3.5 1.2 2014 skill PDF P , 2018 3 1.1 1 2.4 2 0.9 1.5 0.8 1 0.7 0.5 0.6 0 0.5 -0.2 0 0.2 0.4 0.6 0.8 1 1.2 2002 2004 2006 2008 2010 2012 2014 2016 2018 skill value , time (years) A. Jadbabaie, A. Makur, D. Shah (MIT) Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 5 / 12
Soccer World Cups Estimated Skill Densities Negative Differential Entropies from Tournament Data of Estimated Skill Densities 5 1.5 2002 4.5 1.4 2006 negative entropy ! h ( P , ) 4 1.3 2010 3.5 1.2 2014 skill PDF P , 2018 3 1.1 1 2.4 2 0.9 1.5 0.8 1 0.7 0.5 0.6 0 0.5 -0.2 0 0.2 0.4 0.6 0.8 1 1.2 2002 2004 2006 2008 2010 2012 2014 2016 2018 skill value , time (years) Observation: Soccer world cups have remained unpredictable over the years A. Jadbabaie, A. Makur, D. Shah (MIT) Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 5 / 12
Soccer Leagues in 2018-2019 Estimated Skill Densities Negative Differential Entropies from Tournament Data of Estimated Skill Densities 5 1.2 4.5 1 negative entropy ! h ( P , ) 4 3.5 English Premier League 0.8 skill PDF P , Spanish La Liga 3 German Bundesliga 0.6 2.4 French Ligue 1 2 Italian Serie A FIFA World Cup 2018 0.4 1.5 1 0.2 0.5 0 0 -0.2 0 0.2 0.4 0.6 0.8 1 1.2 World English Spanish German French Italian skill value , Soccer leagues in 2018-2019 A. Jadbabaie, A. Makur, D. Shah (MIT) Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 6 / 12
Soccer Leagues in 2018-2019 Estimated Skill Densities Negative Differential Entropies from Tournament Data of Estimated Skill Densities 5 1.2 4.5 1 negative entropy ! h ( P , ) 4 3.5 English Premier League 0.8 skill PDF P , Spanish La Liga 3 German Bundesliga 0.6 2.4 French Ligue 1 2 Italian Serie A FIFA World Cup 2018 0.4 1.5 1 0.2 0.5 0 0 -0.2 0 0.2 0.4 0.6 0.8 1 1.2 World English Spanish German French Italian skill value , Soccer leagues in 2018-2019 Observation: Recover ranking of soccer leagues that is consistent with fan experience A. Jadbabaie, A. Makur, D. Shah (MIT) Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 6 / 12
US Mutual Funds Negative Entropies of Estimated Skill Densities from Tournament Data Estimated Skill Densities 8 8 1.4 2005 2012 1.3 7 7 2006 2013 negative entropy ! h ( P , ) 1.2 2007 2014 6 6 2008 2015 1.1 skill PDF P , skill PDF P , 5 5 2009 2016 1 2010 2017 4 4 2011 2018 0.9 3 3 0.8 2 2 0.7 1 1 0.6 0 0 0.5 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 2006 2008 2010 2012 2014 2016 2018 skill value , skill value , time (years) A. Jadbabaie, A. Makur, D. Shah (MIT) Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 7 / 12
US Mutual Funds Negative Entropies of Estimated Skill Densities from Tournament Data Estimated Skill Densities 8 8 1.4 2005 2012 1.3 7 7 2006 2013 negative entropy ! h ( P , ) 1.2 2007 2014 6 6 2008 2015 1.1 skill PDF P , skill PDF P , 5 5 2009 2016 1 2010 2017 4 4 2011 2018 0.9 3 3 0.8 2 2 0.7 1 1 0.6 0 0 0.5 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 2006 2008 2010 2012 2014 2016 2018 skill value , skill value , time (years) Observation: Skill score is minimum during the Great Recession in 2008 A. Jadbabaie, A. Makur, D. Shah (MIT) Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 7 / 12
Outline Introduction 1 Contributions 2 Formal Setup Estimation Algorithm Theoretical Results A. Jadbabaie, A. Makur, D. Shah (MIT) Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 8 / 12
✶ ✶ Formal Setup Unknown probability density of skill levels P α on R + A. Jadbabaie, A. Makur, D. Shah (MIT) Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 9 / 12
✶ ✶ Formal Setup Unknown probability density of skill levels P α on R + Teams { 1 , . . . , n } play tournament with unknown i.i.d. skill levels α 1 , . . . , α n ∼ P α A. Jadbabaie, A. Makur, D. Shah (MIT) Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 9 / 12
✶ Formal Setup Unknown probability density of skill levels P α on R + Teams { 1 , . . . , n } play tournament with unknown i.i.d. skill levels α 1 , . . . , α n ∼ P α For any teams i � = j , with probability p ∈ (0 , 1], observe k independent pairwise games Z 1 ( i , j ) , . . . , Z k ( i , j ), where Z m ( i , j ) = ✶ { j beats i in m th game } A. Jadbabaie, A. Makur, D. Shah (MIT) Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 9 / 12
✶ Formal Setup Unknown probability density of skill levels P α on R + Teams { 1 , . . . , n } play tournament with unknown i.i.d. skill levels α 1 , . . . , α n ∼ P α For any teams i � = j , with probability p ∈ (0 , 1], observe k independent pairwise games Z 1 ( i , j ) , . . . , Z k ( i , j ), where Z m ( i , j ) = ✶ { j beats i in m th game } Bradley-Terry-Luce (BTL) or multinomial logit model [BT52,Luc59,McF73]: α j P ( Z m ( i , j ) = 1 | α 1 , . . . , α n ) = α i + α j A. Jadbabaie, A. Makur, D. Shah (MIT) Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 9 / 12
Formal Setup Unknown probability density of skill levels P α on R + Teams { 1 , . . . , n } play tournament with unknown i.i.d. skill levels α 1 , . . . , α n ∼ P α For any teams i � = j , with probability p ∈ (0 , 1], observe k independent pairwise games Z 1 ( i , j ) , . . . , Z k ( i , j ), where Z m ( i , j ) = ✶ { j beats i in m th game } Bradley-Terry-Luce (BTL) or multinomial logit model [BT52,Luc59,McF73]: α j P ( Z m ( i , j ) = 1 | α 1 , . . . , α n ) = α i + α j Goal: Learn P α from observation matrix Z ∈ [0 , 1] n × n with k � ✶ { games observed between i , j } 1 Z m ( i , j ) , i � = j Z ( i , j ) = k m =1 0 , i = j A. Jadbabaie, A. Makur, D. Shah (MIT) Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 9 / 12
Estimation Algorithm Assume P α is bounded, in an η -H¨ older class, and has support in [ δ, 1]. Algorithm Estimating P α from Z Input: Observation matrix Z P ∗ of unknown P α Output: Estimator � A. Jadbabaie, A. Makur, D. Shah (MIT) Estimation of Skill Distribution from a Tournament NeurIPS 2020 SPOTLIGHT 10 / 12
Recommend
More recommend
