Forecasting with Judgment Forecasting with Judgment Simone - - PowerPoint PPT Presentation

Forecasting with Judgment Forecasting with Judgment

Simone Manganelli

DG-Research European Central Bank

(Frankfurt am Main, Germany)

Disclaimer: The views expressed in this paper are our own and do not necessarily reflect the views of the ECB or the Eurosystem

Classical Theory of Forecasting Classical Theory of Forecasting

• Forecasts serve to make decisions about the future.
• Forecast errors impose costs on the decision-maker.
• Agents want to minimize expected loss associated to

forecast errors.

• Classical forecast is the minimizer of the sample

equivalent of the expected loss.

• See, e.g., Haavelmo (1944), Granger and Newbold

(1986), Granger and Machina (2005).

Optimal Point Forecast Optimal Point Forecast

] ) [( ) ( min

2 1

• +

T

y E Q

=

• =

T t t T

y T

1 1

ˆ

• ]

[ 2

1

=

• +
• T

y E

FOC:

] [

1 * +

=

T

y E

• T

t t

y

1

} {

=

Want to forecast yT+1 Simplest case:

• i.i.d. normally distributed observations with variance 1
• quadratic loss function

The Problem The Problem

T

• ˆ

=

• T

t t T

y T Q

1 2 1

] ) [( ) ( ˆ min

• is the minimiser of:

]} ) [( ] ) [( ] ) [( { min

2 1 1 2 1 2 1

• +
• +

=

• +
• T

T t t T

y E y T y E ) ( T

• )}

( ] ) [( { min

2 1

• T

T

y E +

• +

Introducing Judgment Risk Analysis of New Estimator Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

Outline Outline

] [ ) (

1

=

• +
• T

y E Q

• ~

ˆ ] ~ [ ) ~ ( ˆ

1 1

• =
• =
• T

T t t T

y y T Q

maker. decision the

• f

guess subjective a is ~ where

Start from a Subjective Guess Start from a Subjective Guess

] ) [( ) ( min

2 1

• +

T

y E Q

The sample equivalent of the FOC is: NOTE: This is a random variable, which may be different from zero just because of statistical error.

Introducing Judgment Introducing Judgment Risk Analysis of New Estimator Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

• Choose a confidence level α
• ηα/2 is the critical value

Then Do Hypothesis Testing Then Do Hypothesis Testing

mean true the is ~ : H

) / 1 , ( ~ ~ ˆ T N yT

• Introducing Judgment

Introducing Judgment Risk Analysis of New Estimator Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

If null is rejected: → First derivatives statistically ≠ 0 → Can be confident to decrease the true objective function → Only up to the point where new H0 cannot be rejected

*

ˆ

T

2 /

• )

~ ( ˆ

T

Q

• )

~ ( ˆ

T

Q

• )

ˆ ( ˆ

* T T

Q

• Graphical Illustration

Graphical Illustration

Distribution of FOC under H0

Introducing Judgment Introducing Judgment Risk Analysis of New Estimator Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

1 1

ˆ y =

• First derivatives not statistically

different from zero

2 / 1

ˆ

• 2

/ 1

ˆ

• +
• ~

] ) [(

2

• y

E

2 1

) (

• y

100*(1-α)% confidence interval

) ( 2

1

y

• Introducing Judgment

Introducing Judgment Risk Analysis of New Estimator Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

Intuition Intuition

New estimator:

• <
• +

<

• >
• =

2 / 2 / 2 / 2 / 2 / *

~ ˆ if ˆ | ~ ˆ | if ~ ~ ˆ if ˆ

• T

T T T T T

y y y y y

Introducing Judgment Introducing Judgment Risk Analysis of New Estimator Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

Interpretation Interpretation

• α is the probability of committing type I

errors, i.e. of rejecting the null when it is true.

• Choose low α when confident in subjective

guess or if the cost of type I errors is high.

• Classical paradigm sets α =1:

– Always FOC equal to zero; – No room for subjective guess; – It commits type I errors with probability 1.

Introducing Judgment Introducing Judgment Risk Analysis of New Estimator Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

Problems with Pretest Estimators Problems with Pretest Estimators

• ~

ˆ : =

T

H

Test the following null hypothesis, for given confidence level α: Then:

• If do not reject, keep the subjective guess
• If reject, take the maximum likelihood estimator

This is wrong!

Introducing Judgment Introducing Judgment Risk Analysis of New Estimator Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

1 1

ˆ y =

• First derivatives not statistically

different from zero

2 / 1

ˆ

• 2

/ 1

ˆ

• +
• ~

] ) [(

2

• y

E

2 1

) (

• y

(1-α)% confidence interval

) ( 2

1

y

• Introducing Judgment

Introducing Judgment Risk Analysis of New Estimator Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

Introducing Judgment Risk Analysis of New Estimator Risk Analysis of New Estimator Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

Estimate the mean, given a single observation y1 (Magnus 02) ) 1 , ( . . . ~

• N

d i i yt Let

• <
• +

<

• >
• =

2 / 1 2 / 1 2 / 1 2 / 1 2 / 1 * 1

~ if | ~ | if ~ ~ if

• y

y y y y Compare the risk properties of two estimators: 1) Subjective classical estimator (coincides with Magnus 02)

• >
• <
• =

2 / 1 1 2 / 1

| ~ | if | ~ | if ~ ˆ

• y

y y

P

2) Pretest estimator

Introducing Judgment Risk Analysis of New Estimator Risk Analysis of New Estimator Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

Risk Associated to Risk Associated to f f( (y y) )

1 2 3 4 0.5 1 1.5 2 2.5 3 3.5 4

OLS (_=1) Subjective Guess (_=0) Pretest (_=0.10) Subjective Classical (_=0.10)

~ , ] ) ) ( [(

2

=

• y

f E

Introducing Judgment Risk Analysis of New Estimator Risk Analysis of New Estimator Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

Monte Carlo Simulation Monte Carlo Simulation

• Random draws from standard normal
• Sample sizes = 5, 20, 60, 120, 240, 1000
• Quadratic loss function
• Two estimators: classical, new (α=0.10)
• Evaluated expected loss with MC simulation
• Repeat 5000 times and average

Introducing Judgment Risk Analysis of New Estimator Risk Analysis of New Estimator Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

0.1 0.2 0.3 0.4 0.5 0.6 0.7 5 20 60 120 240 1000

Mean 0.05 0.1 0.5 1

Introducing Judgment Risk Analysis of New Estimator Risk Analysis of New Estimator Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

Implications Implications

• Good guess (gut feelings) are as important as good

econometric models.

• Organization of forecasting process:

– Subjective guess based on maximum likelihood estimates can never be rejected by construction; – Clear separation b/w:

Who provides guess, based on judgment; Who tests the guess, based on econometric models.

• Shared responsibility for the quality of the forecasts:

– High confidence in bad judgment results in bad forecast.

Introducing Judgment Risk Analysis of New Estimator Risk Analysis of New Estimator Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

Introducing Judgment Risk Analysis of New Estimator Example 1: Asset Allocation Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

Generalization Generalization

) , ( ) ( ˆ

• N

U T

d T

• ]

1 , [ ~ ) 1 ( ˆ ) (

*

• +
• T

T

*

) ( : null Under the

• =

T

H

)) ( ( ˆ ˆ )) ( ( ˆ )) ( ( ˆ

* 1 * ' *

• T

T T T T T

U U T z

• k

T T T T k T T k T T

z U z z

, * * ] 1 , [ , * , *

)) ( ( ˆ s.t. )) ( ( ˆ max arg ˆ )) ( ( ˆ if ˆ )) ( ( ˆ if

• =

=

• >

=

• Introducing Judgment

Risk Analysis of New Estimator Example 1: Asset Allocation Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

Mean-Variance Asset Allocation Mean-Variance Asset Allocation

• Mean-variance optimizers tend to produce portfolio

allocations with little or negative investment value.

• “[They] overuse statistically estimated information

and magnify the impact of estimation errors. It is not simply a matter of garbage in, garbage out, but, rather, a molehill of garbage in, a mountain of garbage out” (Michaud 1998)

Introducing Judgment Risk Analysis of New Estimator Example 1: Asset Allocation Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

Set up Set up

• N+1 assets
• yt,i return of asset i
• θi weight of asset i (they sum to 1)
• +

=

=

1 1 ,

) (

N i i t i t

y y

• portfolio return

Introducing Judgment Risk Analysis of New Estimator Example 1: Asset Allocation Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

Otpimization Otpimization

} )] ( [ )] ( [ { )] ( [ } ] [ ' { ] [ ' )] ( [ )] ( [ ] ; [

2 2

• t

t t t t t t t

y E y E y E y V y E y V y E y U

• =
• =
• =

} ] ) ( [ ) ( { ) ( ] ; [ ˆ

2 1 1 1 2 1 1 1

• =
• =
• =
• =

T t t T t t T t t t

y T y T y T y U

• Introducing Judgment

Risk Analysis of New Estimator Example 1: Asset Allocation Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

Implementation Details Implementation Details

• Monthly log returns from DJIA
• From January 1987 to July 2005
• λ = 4 (coefficient of risk aversion)
• Rolling windows (M=60 and 120 as in

DeMiguel et al. 2007)

• Benchmark portfolio: equally weighted
• Report out of sample differences in average

utilities between optimal and benchmark portfolios

Introducing Judgment Risk Analysis of New Estimator Example 1: Asset Allocation Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

)} ( ] ) ( ) [( ) ( { ) ( ) (

1 1 2

• t

T M t t t t t

y y M T y y U

• +

=

• =
• +

=

• T

M t t t t M T

U U M T Z

1 * 1 1

)] ~ ( )) ˆ ( ( [ ) (

• Out of Sample Evaluation

Out of Sample Evaluation

Test for statistical significance using Giacomini and White (2006)

Introducing Judgment Risk Analysis of New Estimator Example 1: Asset Allocation Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

0.92** 0.01 0.02 α=0.01 1.48* 4.19*** 0.82* 0.09 0.48* α=0.10 2.61 6.47* 1.82

• 8.40***

1.00

• 0.30

α=1 N=30 N=16 N=4 N=30 N=16 N=4 M=120 M=60

Results Results

Introducing Judgment Risk Analysis of New Estimator Example 1: Asset Allocation Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

Introducing Judgment Risk Analysis of New Estimator Example 1: Asset Allocation Example 2: Forecasting US GDP Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

Implementation Issues Implementation Issues

Difficult to formulate guess on abstract model parameters. Assume the decision-maker can formulate a guess on the variable to be forecast (U.S. GDP, in this example). This guess can be mapped into a guess for model parameters as follows:

1 1

~ ) ( ˆ s.t. ) ( ˆ max arg ~

+ +

= =

T T T

y y U

• Subjective guess

Model’s forecast

Introducing Judgment Risk Analysis of New Estimator Example 1: Asset Allocation Example 2: Forecasting US GDP Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

Implementation Details Implementation Details

t i i t i t

y y

• +

+ =

=

• 4

1

The model, AR(4):

=

• =

T t t t T

y y T U

1 2 1

)] ( ˆ [ ) ( ˆ

• The objective function (quadratic):

The data:

• Quarterly U.S. real GDP growth rates
• FRED data base, seasonally adjusted
• From Q1 1983 to Q3 2005

Introducing Judgment Risk Analysis of New Estimator Example 1: Asset Allocation Example 2: Forecasting US GDP Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

Results Results

3.77% 5% 3% 3% 3.19% 0.01

• 0.04

0.05 0.05 0.04

• 0.12
• 0.03
• 0.18
• 0.18
• 0.16

0.34 0.30 0.37 0.37 0.36 0.29 0.42 0.21 0.21 0.23 1.99 2.73 1.54 1.54 1.65 % 3 ~

1 = + T

y

• )

ˆ (

*

T % 5 ~

1 = + T

y

T

• ˆ
• ~

1

• 2
• 3
• 4
• )

( ˆ

1 + T

y

• ~

) ˆ (

*

T

Introducing Judgment Risk Analysis of New Estimator Example 1: Asset Allocation Example 2: Forecasting US GDP Example 2: Forecasting US GDP Relationship with Bayesian Conclusion

Introducing Judgment Risk Analysis of New Estimator Example 1: Asset Allocation Example 2: Forecasting US GDP Relationship with Bayesian Relationship with Bayesian Conclusion

Non sample information is summarised by:

• priors, in Bayesian econometrics
• subjective guess and confidence associated to it, in our case

Special cases:

1. No uncertainty about optimal value of θ0:

– Bayesian: prior is degenerate distribution with total mass on θ0; – Classical: subjective guess=θ0, α=0 (i.e. never reject the null)

2. No information, besides the sample:

– Bayesian: diffuse prior – Classical : α=1 (i.e. set FOC equal to zero)

In other intermediate cases, no clear mapping b/w the two. When non sample info if formulated:

• via prior distributions, be Bayesian
• via subjective guess and confidence, use subjective classical estimator

In general, the choice is dictated by the decision maker, through the format in which s/he provides the non sample info.

Relationship with Bayesian Econometrics Relationship with Bayesian Econometrics

Conclusion Conclusion

• Ignoring non sample information and estimation error

are connected problems in the classical theory of forecasting.

• Forecasts should maximize the objective function in a

stochastic sense, not deterministic.

• Start from subjective guess and construct estimator

(instead of the other way round)

• Test if FOC are statistically different from zero:

– If not, subjective guess is retained as forecast – If yes, objective function is increased as long as FOC are ~= 0