Flipping Coins in the War Room: Skill and Chance in the NFL Draft - - PowerPoint PPT Presentation

Flipping Coins in the War Room: Skill and Chance in the NFL Draft

Cade Massey Yale University MIT Sports Analytics Conference March 4, 2011

Skill and Chance in the Draft

Clearly there is skill involved. But are there differences in skill? Are some teams better than others at picking players?

DET CLE JAC HOU BUF TB NO TEN MIA OAK KC SF NYG CIN DEN NYJ ATL STL CAR SD ARI GB WAS NE BAL MIN CHI PHI SEA DAL PIT IND

5 10 15 20 25 30 35 40

Pro Bowl Seasons by Players Drafted '97-'07

“Indianapolis Colts…not just with Peyton Manning… Dwight Freeney, Edgerrin James and Reggie Wayne were genius picks, too.” “The Cleveland Browns…screwed the Chihuahua. Their run of No. 1 picks from 1999 to 2002 is the single worst stretch of drafting since the Iraqi Republican Guard. Were they using an Ouija board?”

Skeptical, due to research on NFL draft

Massey & Thaler (“Loser’s Curse”)

• What’s the chance that a player is better than the

next player taken at his position?

• E.g., the 3rd cornerback vs. the 4th cornerback

.1 .2 .3 .4 .5 .6 .7 Probability 1 2 3 4 5 6 7 by draft round

Better Than The Next Guy?

Skeptical, due to research in psychology

• People under-appreciate the role of chance
• E.g., belief in the “law of small numbers”
• An important implication: “Fictitious Variation”
• Matthew Rabin (2002)
• Suggests that people over-attribute differences in
• utcomes to differences in skill.
• Provides our main hypothesis: There is less

difference in draft-picking skill than people believe.

Methodology

Use games started

• Results are robust to pro bowl, Massey-Thaler $s,

etc.

Adjusted for draft position

• Compared against long-run average for that pick

Use the player’s first 5 years Two tests:

• Within year, across teams (intra-class correlation)
• Within team, across years (autocorrelation)

2 4 6 8 10 12 14 16

Starts/Season

1 33 65 97 129 161 193 225 257

Draft Pick

Performance evaluated over player's 1st 5 years.

Draft-pick Performance vs. Expectation

Warren Wilson Johnson Klecko Samuel Koppen Kingsbury Nead Banta-Cain Kelley

2 4 6 8 10 12 14 16

Starts/Season

1 33 65 97 129 161 193 225 257

Draft Pick

Performance evaluated over player's 1st 5 years.

Patriots, 2003

Draft-pick Performance vs. Expectation

• 10
• 5

5 10

Games Started vs. Expected

ARI ATL BAL BUF CAR CHI CIN CLE DAL DEN DET GB HOU IND JAC KC

Team

Pick Performance by Team (1995 Draft)

.1 .2 .3 .4 .5 .6 .7 .8 .9 1

Intraclass Correlation

1991 1993 1995 1997 1999 2001 2003 2005

Draft Year

• 2
• 1

1 2

Avg Games Started vs. Expected

This Year Next Year

4th 25% 3rd 25% 2nd 25% Top 25% 4th 25% 3rd 25% 2nd 25% Top 25%

1991-2006 Drafts

r=-0.14 (n.s)

Skill and Chance in the Draft

So it’s all chance on the margin! Is this nihilistic? Quite the contrary.

Decision strategy depends on situation

Skill-based Chance-based

?

Logic of Chance (LOC)

Distinct set of prescriptions for operating in an environment with high uncertainty.

LOC: First, acknowledge role of chance

Roethlisberger on the number

• f times they’ve beaten the

Ravens in close games: “We got lucky.”

LOC: Maximize your # of chances

LOC: Minimize the cost of choosing

vs.

LOC: Value the process (and protect it)

LOC: Seek broad, independent input

LOC: Be slow to place credit, blame

In sum: At the margin, picking players is chance

Teams don’t vary at this… …and there are no persistent differences in drafting ability.

.1 .2 .3 .4 .5 .6 .7 .8 .9 1

Intraclass Correlation

1991 1993 1995 1997 1999 2001 2003 2005

Draft Year

• 2
• 1

1 2

Avg Games Started vs. Expected

This Year Next Year

4th 25% 3rd 25% 2nd 25% Top 25% 4th 25% 3rd 25% 2nd 25% Top 25%

1991-2006 Drafts

Managing chance is where the skill is.