SLIDE 1
Error Bars Considered Harmful
Exploring Alternate Encodings for Mean and Error
Michael Correll Michael Gleicher University of Wisconsin-Madison
Don’t Use Error Bars
They don’t work as advertised Try something else instead!
Error Bars Considered Harmful Exploring Alternate Encodings for Mean - - PowerPoint PPT Presentation
Error Bars Considered Harmful Exploring Alternate Encodings for Mean - - PowerPoint PPT Presentation
Error Bars Considered Harmful Exploring Alternate Encodings for Mean and Error Michael Correll Michael Gleicher University of Wisconsin-Madison Dont Use Error Bars They dont work as advertised Try something else instead! Dont Use
SLIDE 2
SLIDE 3
Don’t Use Error Bars
They don’t work as advertised
SLIDE 4
20 40 60 80 100 Placebo Treatment
SLIDE 5
20 40 60 80 100 Placebo Treatment
SLIDE 6
20 40 60 80 100 Placebo Treatment
SLIDE 7
20 40 60 80 100 Placebo Treatment
p<0.05
SLIDE 8
20 40 60 80 100 Placebo Treatment *
SLIDE 9
Error Bars:
Are ambiguous Are asymmetric Are “all or nothing”
SLIDE 10
Error Bars:
Are ambiguous Are asymmetric Are “all or nothing”
SLIDE 11
InfoVis 2010-2013
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Labeled Unlabeled
SLIDE 12
InfoVis 2010-2013
Standard error 95% t confidence interval Range 1.5 x interquartile range Standard deviation 80% t confidence interval
SLIDE 13
20 40 60 80 100 Placebo Treatment
SLIDE 14
Error Bars:
Are ambiguous Are asymmetric Are “all or nothing”
SLIDE 15
20 40 60 80 100 Placebo Treatment
SLIDE 16
20 40 60 80 100 Placebo Treatment
SLIDE 17
20 40 60 80 100 Placebo Treatment
SLIDE 18
Within-the-bar bias
Newman, George E, and Brian J Scholl. “Bar graphs depicting averages are perceptually misinterpreted: the within-the-bar bias.” Psychonomic bulletin & review 19.4 (2012): 601–7.
10 20 30 40 50 60 70 80 90 100 10 20 30 40 50 60 70 80 90 100
SLIDE 19
Within-the-bar bias
10 20 30 40 50 60 70 80 90 100 10 20 30 40 50 60 70 80 90 100
SLIDE 20
Within-the-bar bias
10 20 30 40 50 60 70 80 90 100 10 20 30 40 50 60 70 80 90 100
SLIDE 21
Error Bars:
Are ambiguous Are asymmetric Are “all or nothing”
SLIDE 22
10 20 30 40 50 60 70 80
SLIDE 23
10 20 30 40 50 60 70 80
SLIDE 24
10 20 30 40 50 60 70 80
SLIDE 25
Don’t Use Error Bars
They don’t work as advertised Try something else instead!
SLIDE 26
A solution?
20 40 60 80 100 Placebo Treatment 20 40 60 80 100 Placebo Treatment
SLIDE 27
Design Requirements
Consistent Symmetric Continuous ?
SLIDE 28
Design Requirements
Consistent Symmetric Continuous ?
SLIDE 29
Design Requirements
Consistent Symmetric Continuous ?
SLIDE 30
Bar Chart
SLIDE 31
Violin Plot
- J. Hintze and R. Nelson. Violin plots: a box plot-density trace
- synergism. The American Statistician, 1998.
95% t-confidence interval
SLIDE 32
Gradient Plot?
95% t-confidence interval
SLIDE 33
Gradient Plot
95% t-confidence interval “100%” t-confidence interval
SLIDE 34
Methods
3 experiments on Amazon Mechanical Turk, 240 participants 3 problem frames (election polling, weather forecasting, financial modeling) No prerequisite of statistical knowledge Participants gave a predictions as either binary forced choice, or on a Likert scale
SLIDE 35
One Sample Judgments
How likely (or how surprising) do you think the red potential outcome is, given the poll?
SLIDE 36
Results
SLIDE 37
“Within the bar” bias
Error bars suffer from this bias… but other encodings don’t
SLIDE 38
Two Sample Judgments
If forced to guess, which city do you predict will get more snow?
SLIDE 39
Overconfidence
Error bars make people unjustifiably confident… but other encodings don’t
SLIDE 40
Costs are low
p-value Effect size
SLIDE 41
Don’t Use Error Bars
They don’t work as advertised Try something else instead!
SLIDE 42
What’s next?
More encodings More testing Real stakes
SLIDE 43
Make your own!
http://graphics.cs.wisc.edu/Vis/ErrorBars/
SLIDE 44
Acknowledgments
This work was supported in part by NSF award IIS-1162037, NIH award R01 AI077376, and ERC Advanced Grant “Expressive.” Thanks to Wei-Chen Chen for web generation code. Visit: http://graphics.cs.wisc.edu/Vis/ErrorBars/ to make your own plots! (and for data tables, stimuli, and sample experiments). Contact: mcorrell@cs.wisc.edu
SLIDE 45
Box Plot
50% t-confidence interval 95% t-confidence interval
SLIDE 46
Müller-Lyer Illusion
- W. Stock and J. Behrens. Box, Line, and Midgap plots: Effects of Display
Characteristics on the Accuracy and Bias of Estimates of Whiskey Length. Journal of Educational and Behavioral Statistics, 1991
SLIDE 47
Müller-Lyer Illusion
- W. Stock and J. Behrens. Box, Line, and Midgap plots: Effects of Display
Characteristics on the Accuracy and Bias of Estimates of Whiskey Length. Journal of Educational and Behavioral Statistics, 1991
SLIDE 48
p=.05?
20 40 60 80 100 Placebo Treatment
SLIDE 49
p<0.01
20 40 60 80 100 Placebo Treatment
SLIDE 50
p=.05
20 40 60 80 100 Placebo Treatment
SLIDE 51
p=.05
20 40 60 80 100 Placebo Treatment
SLIDE 52