SMALL STEPS TO ACCURACY: Pavel Atanasov Jens Witkowski INCREMENTAL - PowerPoint PPT Presentation

SMALL STEPS TO ACCURACY: Pavel Atanasov Jens Witkowski INCREMENTAL BELIEF UPDATERS Lyle Ungar Barbara Mellers ARE BETTER FORECASTERS Philip Tetlock 21 ST ACM CONFERENCE ON ECONOMICS AND COMPUTATION (EC’20) JULY 13-17, 2020 1

BELIEF UPDATING AND SKILL Questions: How do the best forecasters update their beliefs? Methods & Data: Good Judgment Project, 515 forecasters with 80k+ forecasts on 487 real-world geopolitical forecasts. Outcome: Brier score, cross-validated across questions 2

UPDATING AND ACCURACY MEASURES 4 !"#$" &'("$ = 1 1 ) 3 + ,2(1 − 0 156 100% 80% Probability 60% 40% 20% 0% Time Update Measures Frequency: Number of new forecasts, denoted by red dots in example above (3) Magnitude: Mean absolute update size, length of vertical lines (27%) 3

MAGNITUDE & FREQUENCY PREDICT ACCURACY AT DIFFERENT TIMES Worse Standardized Brier Score -0.4 0.0 0.4 Rescaled Brier Score UpdatePattern Large, infrequent Large, frequent Small, infrequent Small, frequent Better All Start Middle End Periods Time within Question 4