Combining Forecasts Forty Years Later Kenneth F. Wallis Emeritus - - PowerPoint PPT Presentation

6th Eurostat Colloquium on Modern Tools for Business Cycle Analysis Luxembourg, 26-29 September 2010

Combining Forecasts – Forty Years Later

Kenneth F. Wallis

Emeritus Professor of Econometrics, University of Warwick http://go.warwick.ac.uk/kfwallis This paper is dedicated to the memory of Clive Granger. It will appear in a special issue of Applied Financial Economics in his memory.

1

• 1. Introduction

A major theme of “Combining forecasts – twenty years later” (Granger, 1989) is the relevance of information sets, discussion of which “is an essential feature of all forecast situations, including combining.” This paper returns to two of the topics in that article that are of much current interest, some forty years after Bates and Granger’s seminal article on “The combination of forecasts”:

• the impact on the original point forecast combination result of forecasters having

different information sets, with each forecaster using their information efficiently. (In the

• riginal result, the efficiency gain rests on the component forecasts being inefficient.)
• properties of different methods of combining (pooling) density forecasts, where the

quantile-based methods proposed in “Twenty years later” have not been taken up, for reasons I have discussed in a previous paper (Wallis, 2005).

2

• 2. Point forecasts with different information sets

Each one of N forecasters has an information set with a public and a private component: public: current and past values of variables t

z

private: … variables

, 1,...,

jt

x j N =

; all components are mutually independent.

The universal information set t

U consists of all information available to all forecasters; if t U

were known, the optimum (l.l.s.) one-step-ahead forecast of target variable y would be

( )

1 1

E ( ) ( )

N t t t t j jt j

E y U B z B x α β

+ =

= = +∑

.

(1) Individual forecasts:

( ) ( ) , 1,...,

jt t j jt

E B z B x j N α β = + =

(2) Combined forecast:

1 1

1 1 ( ) ( )

N N t jt t j jt j j

E E B z B x N N α β

= =

= = +

∑ ∑

(3) “… aggregating forecasts is not the same as aggregating information sets.

t

E is based on

all the information, but is not equal to

t

E , as the information is not being used efficiently.”

3

However, if yet another forecast is constructed, using only the public information, namely

( )

t t

E B z α =

(4) “then it is seen immediately that the optimal forecast

t

E can be achieved by

1

( 1)

N t jt t j

E E N E

=

= − −

∑

.” (5) (Equations and quotations taken from “Twenty years later”) The inefficiency-of-mean-forecasts rests on a comparison of expressions for

t

E and

t

E .

I then slightly modify Granger’s setting to obtain explicit expressions for the mean square error (MSE) of the various forecasts, with inequalities that deliver the inefficiency result. These inequalities match those given by Kim, Lim and Shaw (2001) in their independent rediscovery of the inefficiency-of-mean-forecasts result (their term). Forecasters in their model receive a public and a private signal, each equal to the target variable plus random

• noise. In my setting the source of random variation is the innovation in the DGP, as usual.

4

However Kim, Lim and Shaw do not consider the possibility of constructing a fully efficient forecast, since “neither … common nor private information is separately observable”. Instead they consider the exploitation of a sequence of signals and associated forecast revisions in a fixed-event forecasting context. Crowe (2010) adopts this framework to analyse the monthly forecasts for current-year and next-year economic growth collected by Consensus Economics in 38 countries. In fixed-event forecasting the public information includes previously-published mean forecasts of the same target variable, and hence the successive revisions to those mean forecasts. Crowe finds that use of this information could have achieved efficiency gains of some 5% RMSE in out-of-sample mean current-year forecasts for 2007 and 2008.

5

• 3. Linear and logarithmic combination of density forecasts

Consider N individual density forecasts

( ), 1,..., ,

j

f y j N =

at some time, horizon, etc. Linear combination:

1

( ) ( )

N C j j j

f y w f y

=

= ∑

with weights

0, 1,..., , 1

j j

w j N w ≥ = Σ = .

If

( )

2

( ) , , 1,..., ,

j j j

f y j N μ σ = = N

then

( )

C

f y is a mixture of normals (Pearson, 1894).

For any distributional forms, the combined density has mean and variance:

1 N C j j j

w μ μ

=

= ∑

and

( )

2 2 2 1 1 N N C j j j j C j j

w w σ σ μ μ

= =

= + −

∑ ∑

. In the case of equal weights,

1

j

w N =

, the variance equation can be interpreted as: aggregate uncertainty = average individual uncertainty + disagreement.

6

Logarithmic combination: This is usually written in geometric form:

( ) ( ) ( )

j j

w j G w j

f y f y f y dy = ∏

∏ ∫

If

( )

2

( ) , , 1,..., ,

j j j

f y j N μ σ = = N

then

( )

2

( ) , ,

G G G

f y μ σ = N

where

2 2 1 N j G j j G j

w μ μ σ σ

=

= ∑

and

2 2 1

1 1

N j j G j

w σ σ

=

= ∑

. In the case of equal weights:

2 G

σ

is the harmonic mean of the individual variances; this is less than their arithmetic mean, which is less than the finite mixture variance

2 C

σ

;

G

μ is the linear combination of the

j

μ with inverse variance weights.

7

Example We add a logarithmic combination to some of the competing forecasts that appear in the simulation study of Mitchell and Wallis (2010). The data-generating process is the Gaussian second-order autoregression:

( )

1 1 2 2 2

, ~ 0,

t t t t t

Y Y Y

ε

φ φ ε ε σ

− −

= + + N

( ) ( )

2 2 1 1 2 2 1 1 2 1 1 2 2

1 , , 1

y ε

ρ φ φ ρ φ ρ φ σ φ ρ φ ρ σ = − = + = − − .

The universal information set comprises the observations

1 t

y − and

2 t

y − , the model and its

parameter values, so the optimal or “ideal” density forecast of

t

y is

( )

2 1 1 2 2

( ) ,

• t

t t

f y y y

ε

φ φ σ

− −

= + N

. Two variant density forecasts, and their linear and logarithmic combinations (with equal weights), are constructed as follows:

8

AR1 forecaster:

( )

2 1 1 1 1

( ) ,

t t

f y y ρ σ

−

= N ,

( )

2 2 2 1 1

1

y

σ ρ σ = −

AR2 (same, with data delay):

( )

2 2 2 2 2

( ) ,

t t

f y y ρ σ

−

= N ,

( )

2 2 2 2 2

1

y

σ ρ σ = −

Linear combination:

( ) ( )

2 2 1 1 1 2 2 2

( ) 0.5 , 0.5 ,

C t t t

f y y y ρ σ ρ σ

− −

= + N N

Logarithmic combination:

( )

2

( ) ,

G t G G

f y μ σ = N

, with

2 2 2 1 1 1 2 2 2 2 1 2

,

t t G

y y σ ρ σ ρ μ σ σ

− −

+ = +

2 2 2 1 2 2 2 1 2

2 .

G

σ σ σ σ σ = +

The AR1 and AR2 forecasts are the correct conditional distributions with respect to their specific information sets. So we expect to find that they satisfy probabilistic calibration, but not complete calibration; that is, they have PITs that are

(0,1) U

but are not independent. The two combined forecasts satisfy neither of these calibration concepts.

9

Table 1. Simulation design and density forecast variances (

2

1

ε

σ = in all cases) Parameter Autocorrel’n Density forecast variance Case

1

φ

2

φ

1

ρ

2

ρ

2 1

σ

2 2

σ

2 C

σ

2 G

σ (1) 1.5 –0.6 0.94 0.81 1.56 4.52 3.40 2.32 (2) 0.15 0.2 0.19 0.23 1.04 1.02 1.05 1.03 (3) 0.95 0.95 10.26 1 7.94 1.82 (4) –0.5 0.3 –0.71 0.66 1.10 1.27 1.34 1.18

• 2

G

σ

is less than

2 C

σ

, and intermediate between

2 1

σ and

2 2

σ , in all four cases

• 2

C

σ

exceeds both

2 1

σ and

2 2

σ in two cases

10

Figure 1. PIT histograms for the autoregressive example Rows: cases (1)-(4). Columns: forecasts, respectively AR1, AR2, linear pool, log pool

11

Table 2. Tests of complete calibration and forecast performance: rejection percentages at nominal 5% level

Case (1) Case (2) Case (3) Case (4) Forecast Bk KLIC Bk KLIC Bk KLIC Bk KLIC AR1 100 98 17 25 99 100 62 46 AR2 100 100 30 15 5.6 n.a. 97 93 Lin combn 100 100 14 10 100 100 62 55 Log combn 100 100 12 7 99 84 37 25

Note: Bk is the likelihood ratio test of Berkowitz (2001); KLIC is a test of KLIC differences vs. the ideal forecast (Mitchell and Hall, 2005)

• although AR1 and AR2 are probabilistically calibrated, no forecast is completely calibrated

(except AR2 in case (3))

12

Comparing linear and log combination in our example, log does less badly than linear on probabilistic calibration, and is rejected less frequently by tests of complete calibration. The combinations in our example use equal weights. There is an emerging literature on practical pooling with estimated weights based on the average log score. A Norges Bank study comparing linear and log pooling of forecasting models for inflation and growth finds the log pool to be better probabilistically calibrated. A popular choice of distribution for interval and density forecasts is the normal distribution. Combinations of such forecasts are more convenient analytically and easier to communicate if they have the same functional form, which log combination delivers. The result that a combination of probabilistically calibrated density forecasts is not itself probabilistically calibrated parallels the inefficiency of mean point forecasts discussed

• above. The same result also holds for combinations of event probability forecasts.

“… aggregating forecasts is not the same as aggregating information sets.”

6th Eurostat Colloquium on Modern Tools for Business Cycle Analysis Luxembourg, 26-29 September 2010

Combining Forecasts – Forty Years Later

Kenneth F. Wallis

Emeritus Professor of Econometrics, University of Warwick http://go.warwick.ac.uk/kfwallis This paper is dedicated to the memory of Clive Granger. It will appear in a special issue of Applied Financial Economics in his memory.