The Golden Rule of Forecasting Kesten C. Green University of South - - PowerPoint PPT Presentation

▶
the golden rule of forecasting
SMART_READER_LITE
LIVE PREVIEW

The Golden Rule of Forecasting Kesten C. Green University of South - - PowerPoint PPT Presentation

Nov 13, 2023 47 likes •330 views

The Golden Rule of Forecasting Kesten C. Green University of South Australia Business School with J. Scott Armstrong Wharton, University of Pennsylvania Andreas Graefe LMU Munich, Germany International Symposium on Forecasting, Seoul 11:40

slide-1
SLIDE 1

The Golden Rule of Forecasting

Kesten C. Green

University of South Australia Business School with

  • J. Scott Armstrong

Wharton, University of Pennsylvania

Andreas Graefe

LMU Munich, Germany

International Symposium on Forecasting, Seoul 11:40AM−12:10PM, Wednesday 26 June 2013, Room 101

Golden Rule – 2013-ISF-v12

slide-2
SLIDE 2

GR is “Be Conservative”

  • Future will be like the past
  • Behavior
  • Long-run relationships
  • Levels
  • Trends
  • Use cumulative broad knowledge
  • “Shoe leather” (e.g. Broad St pump)
  • Involve subject matter experts
  • Use forecasting knowledge
  • Evidence-based methods
  • Start with no-change model

How many of you agree with the GR?

2 GoldenRuleofForecasting.com

slide-3
SLIDE 3

Are Golden Rule violations easily spotted? Yes.

  • A. If the description of the

forecasting methods is adequate.

  • B. If not, the Golden Rule has been

violated.

3 GoldenRuleofForecasting.com

slide-4
SLIDE 4

Causes of Golden Rule Violations Supply of

  • 1. Complex statistical procedures
  • 2. Big data

Demand for complex statistical procedures by

  • 1. Clients
  • 2. Academics

4 GoldenRuleofForecasting.com

slide-5
SLIDE 5

When to be conservative

  • All forecasting problems
  • Especially important if situation is
  • Complex
  • Uncertain

– and

  • Bias is likely…
  • Common with
  • government policies
  • investment decisions

5 GoldenRuleofForecasting.com

slide-6
SLIDE 6

Consider All prior data: Example using Global temperatures

6 GoldenRuleofForecasting.com

slide-7
SLIDE 7

Forecasts from a complex method that ignores knowledge and data: NASA GISS

7

*Hansen ¡et ¡al. ¡(1988) ¡NASA ¡GISS ¡forecasts ¡vs ¡actual ¡(5-­‑year ¡rolling ¡average) ¡from ¡ ¡

http://www.kaltesonne.de/?p=4006 ¡

GoldenRuleofForecasting.com

slide-8
SLIDE 8

Betting on conservatism: TheClimateBet.com Starting at "Tipping Point” according to Gore

UAH ¡ RSS ¡ Temp ¡up ¡on ¡ previous ¡month ¡ 28 ¡ 29 ¡ Temp ¡down ¡on ¡ previous ¡month ¡ 33 ¡ 35 ¡ Armstrong ¡wins ¡ month ¡ 67% ¡ 70% ¡ Gore ¡error ¡larger ¡ by… ¡ 19% ¡ 26% ¡

8 GoldenRuleofForecasting.com

!0.7% !0.5% !0.3% !0.1% 0.1% 0.3% 0.5% 0.7% 1981% 1986% 1991% 1996% 2001% 2006% 2011% 2016% Armstrong% Gore% Monthly%anomaly% !0.3% !0.2% !0.1% 0% 0.1% 0.2% 0.3% 0.4% 0.5% 0.6% 2007% 2008% 2009% 2010% 2011% 2012% 2013% 2014% 2015% 2016% 2017% Armstrong% Gore% Monthly%anomaly% !0.3% !0.2% !0.1% 0% 0.1% 0.2% 0.3% 0.4% 0.5% 0.6% 2007% 2008% 2009% 2010% 2011% 2012% 2013% 2014% 2015% 2016% 2017% Armstrong% Gore% Monthly%anomaly% !0.5% !0.25% 0% 0.25% 0.5% 0.75% 1981% 1986% 1991% 1996% 2001% 2006% 2011% 2016% Armstrong% Gore% Monthly%anomaly% !0.2% !0.1% 0% 0.1% 0.2% 0.3% 0.4% 0.5% 0.6% 0.7% 2007% 2008% 2009% 2010% 2011% 2012% 2013% 2014% 2015% 2016% 2017% Armstrong% Gore% Monthly%anomaly% !0.2% !0.1% 0% 0.1% 0.2% 0.3% 0.4% 0.5% 0.6% 0.7% 2007% 2008% 2009% 2010% 2011% 2012% 2013% 2014% 2015% 2016% 2017% Armstrong% Gore% Monthly%anomaly%
slide-9
SLIDE 9

Conservative problem structuring checklist

  • 1. Obtain and use all knowledge by:
  • a. Obtaining all relevant information and understanding
  • b. Decomposing the problem to best use knowledge
  • c. Use evidence-based forecasting methods validated for the

situation

  • 2. Avoid bias in the selection of methods and data by:
  • a. Specifying multiple hypotheses or concealing the purpose of

the forecast

  • b. Obtaining signed ethics statements before and after

forecasting

  • 3. Provide full disclosure to encourage independent

audits and replications

9 GoldenRuleofForecasting.com

slide-10
SLIDE 10

Bias in Forecasting

Example: Demand forecasts for 24 large rail transportation projects are consistently optimistic, with a median

  • verestimate of 96 percent for traffic

(Flyvbjerg)

GoldenRuleofForecasting.com 10

slide-11
SLIDE 11

Conservative judgmental forecasting checklist

  • 1. Avoid unaided judgment
  • 2. Frame questions about a given issue in various

ways

  • 3. Combine independent forecasts from 5 to 20

heterogeneous experts

  • 4. Obtain reasons for forecasts
  • 5. Ask experts to consider why their forecast might

be wrong and to then revise.

  • 6. Use judgmental bootstrapping
  • 7. Use structured analogies
  • 8. Avoid judgmental adjustments

11 GoldenRuleofForecasting.com

slide-12
SLIDE 12

Unaided judgments are not conservative

Typical belief: “Things are different now” In study of over 27,000 political and economic forecasts made over a 20-year period, 284 experts from different fields expected the status quo to change 65% of the time. However, change from the status quo actually

  • ccurred only 51% of the time.

Expert political judgment (Tetlock 2005)

12 GoldenRuleofForecasting.com

slide-13
SLIDE 13

Global temperature forecasting Example of a complex uncertain problem

“Please forecast the missing years for the series shown

  • n the two charts

and pass to the end

  • f the row for

collection.”

13 GoldenRuleofForecasting.com

slide-14
SLIDE 14

Hadley global mean °C temperature anomalies showing selected half-centuries: Note similarities of A & B (B has been said to be unique and near – or past the tipping point)

!1.0% !0.8% !0.6% !0.4% !0.2% 0.0% 0.2% 0.4% 0.6% 0.8%

14

A ¡ B ¡

25 ¡year ¡ ¡ forecast ¡ period ¡ (origin ¡−. 07°C; ¡ ¡ slope ¡−.003 ¡ p.a.) ¡

GoldenRuleofForecasting.com

slide-15
SLIDE 15

Conservative extrapolation checklist

1. Use all valid and reliable data 2. Decompose by causal forces 3. Damp trend forecasts if the:

  • a. situation is uncertain or unstable
  • b. forecast horizon is longer than historical series
  • c. trend goes outside the range of the previous data
  • d. short and long-term trends are inconsistent
  • e. series are contrary (trend inconsistent with causal forces)

4. Damp seasonal factors.

15 GoldenRuleofForecasting.com

slide-16
SLIDE 16

Simon Applies Contrary Series Rule

Ehrlich “Population Bomb” (1968) Running out of resources => price rises, economic collapse and mass starvation by the 1990s Julian Simon Human ingenuity & free markets => resource prices decline over long term Offers bet in 1980: No change in resource prices Ehrlich & Holdren select 5 metals where prices had been rising with a 10-year horizon All prices down by 1990

16 GoldenRuleofForecasting.com

slide-17
SLIDE 17

Conservative causal model checklist

  • 1. Use prior knowledge to identify predictor

variables along with their directional effects, magnitudes, and reasonable limits.

  • 2. Damp estimated coefficients toward equal

weights

  • 3. Use diverse information, data, and models

(there isn’t a perfect model).

  • 4. Use index models when there are many

important variables and much knowledge about their relationships

17 GoldenRuleofForecasting.com

slide-18
SLIDE 18

Conservatism via combining

Combine across methods & forecasters

18

  • Incorporate more prior knowledge
  • Reduce effects of errors and biases

– Data errors – Computational errors – Model selection errors – Biased forecasts

  • Error reductions, under ideal

conditions, exceed 50%

GoldenRuleofForecasting.com

slide-19
SLIDE 19

Survey of experts; initial finding: Broad agreement on Guidelines

19

  • 34 responses to 24 June 2013.
  • 93% of 27 guidelines supported by

more than 70% of experts

GoldenRuleofForecasting.com

slide-20
SLIDE 20

Survey of experts; initial finding: Controversial Guidelines

20

For 6 of 27 Guidelines, more than ¼ of experts rejected the Guideline

  • 1. Specify multiple hypotheses or conceal the purpose of

the forecast (59%)

  • 2. Use structured analogies (50%)
  • 3. Damp when horizon is longer than history (29%)
  • 4. Decompose by causal forces (28%)
  • 5. Damp when uncertain or unstable 28%)
  • 6. Ask experts to consider why wrong and revise (27%)

GoldenRuleofForecasting.com

slide-21
SLIDE 21

Barriers to conservative forecasting

21

Expensive, due to need to acquire comprehensive prior knowledge (shoe leather)

– Unimpressive easily understood methods – Boring forecasts lack newsworthiness

Impressiveness, and obscurity of complex statistical methods and unaided expert judgment, and drama of forecasts

GoldenRuleofForecasting.com

slide-22
SLIDE 22

Conclusions

22

Golden Rule can be unpopular as a constraint

  • n dramatic forecasts that violate the

politician’s syllogism that “Something must be done; this is something; therefore this must be done.”

  • Complex statistical methods and large

databases harm forecast accuracy by ignoring full knowledge about situation and forecasting methods

  • Potential for substantial gains in forecast

accuracy from adopting the Golden Rule: Forecast conservatively by holding to prior knowledge and eschewing complex un- validated methods

GoldenRuleofForecasting.com

slide-23
SLIDE 23

Possible action steps

What implications for a forecasting problem with which you are familiar? How many of the 27 checklist items might be used? Or consider a public policy issue, such as

“What effect would the withdrawal of government funding have on scientific research?” “Will government mandated messages and images on cigarette packs increase welfare?”

23 GoldenRuleofForecasting.com

slide-24
SLIDE 24

Golden Rule Checklist 1/2: (% error reduction)

24 GoldenRuleofForecasting.com

slide-25
SLIDE 25

Golden Rule Checklist 2/2: (% error reduction)

25 GoldenRuleofForecasting.com

slide-26
SLIDE 26

The checklist is an inexpensive way for inquisitive people to identify forecasts derived in ways that violate the Golden Rule The simplicity of the Guidelines and the use

  • f checklists* gives clients the ability to

assess forecasts in a rapid and inexpensive way. If the method is too complex to understand, give it a failing mark.

* Our paper summarizes the evidence on the value of checklists.

26 GoldenRuleofForecasting.com

slide-27
SLIDE 27

Application to forecasts of dangerous manmade global warming Green & Armstrong used the Checklist to assess the forecasting procedures described in the forecasts used by the IPCC. Took only 10 minutes to complete. Concurred that 25 of the 27 guidelines were relevant. Concluded that none were followed.

27 GoldenRuleofForecasting.com

slide-28
SLIDE 28

For Further information

For copies of…

  • These slides
  • “Golden Rule of Forecasting” working

paper

  • The Golden Rule checklist

GoldenRuleofForecasting.com

28 GoldenRuleofForecasting.com