SMALL STEPS TO ACCURACY: Pavel Atanasov Jens Witkowski INCREMENTAL - - PowerPoint PPT Presentation

▶

Oct 31, 2023 176 likes •219 views

SMALL STEPS TO ACCURACY: Pavel Atanasov Jens Witkowski INCREMENTAL BELIEF UPDATERS Lyle Ungar Barbara Mellers ARE BETTER FORECASTERS Philip Tetlock 21 ST ACM CONFERENCE ON ECONOMICS AND COMPUTATION (EC20) JULY 13-17, 2020 1 BELIEF

SLIDE 1

SMALL STEPS TO ACCURACY: INCREMENTAL BELIEF UPDATERS ARE BETTER FORECASTERS

21 ST ACM CONFERENCE ON ECONOMICS AND COMPUTATION (EC’20) JULY 13-17, 2020

Pavel Atanasov Jens Witkowski Lyle Ungar Barbara Mellers Philip Tetlock

SLIDE 2

Questions:

How do the best forecasters update their beliefs?

Methods & Data:

Good Judgment Project, 515 forecasters with 80k+ forecasts on 487 real-world geopolitical forecasts. Outcome: Brier score, cross-validated across questions

BELIEF UPDATING AND SKILL

SLIDE 3

0% 20% 40% 60% 80% 100% Time

UPDATING AND ACCURACY MEASURES

Probability

!"#$" &'("$ = 1 + ,2(1 − 0

1)3 4 156

Update Measures Frequency: Number of new forecasts, denoted by red dots in example above (3) Magnitude: Mean absolute update size, length of vertical lines (27%)

SLIDE 4

MAGNITUDE & FREQUENCY PREDICT ACCURACY AT DIFFERENT TIMES

Start Middle End

Time within Question UpdatePattern

Large, infrequent Large, frequent Small, infrequent Small, frequent

All Periods

Rescaled Brier Score

0.4 0.0 0.4