Empirical Simultaneous Confidence Regions for Path-Forecasts scar - - PowerPoint PPT Presentation

Empirical Simultaneous Confidence Regions for Path-Forecasts

Òscar Jordà1, Malte Knüppel2, Massimiliano Marcellino3

1 University of California, Davis 2 Deutsche Bundesbank 3 European University Institute, Università Bocconi and CEPR

August 2010

Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 1 / 18

Motivation

Confidence bands are used by many forecasting institutions which publish multi-period-ahead forecasts (pathcasts) These confidence bands contain intervals where the realizations are going to lie in with a certain probability for each single period However, path forecasts are often more important than forecast for a single period

“Deflation is a decline in the general level of prices... It has to be persistent — and last for an extended period of time, ...” (Bini Smaghi, ECB Board member, 2009) “The entire path for inflation and the real economy, both before and after the two years, will be taken into account when setting interest rates.“ (Jarle Bergo, Deputy Governor, Norges Bank, 2004) Pricing path-dependent “exotic” options

In such cases, confidence bands should be calculated for path forecasts, not for single h-step-ahead forecast

Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 2 / 18

Motivation

Confidence regions for impulse responses developed by Jorda (2009, REStat) Methodology applied to path forecasts by Jorda & Marcellino (2010, JAE) Jorda & Marcellino study case of known forecasting model, derive model-based confidence regions In most central banks (Fed, ECB, Bank of England, Sveriges Riksbank, Deutsche Bundesbank...) it is standard practice to determine forecast uncertainty based on past forecast errors Aim of this work: Investigate properties of confidence regions for path forecasts if forecasting model is unknown, but sample of forecast errors is available

Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 3 / 18

Introduction - Path Forecast Uncertainty

Fan charts convey information about forecast uncertainty several periods ahead But confidence intervals are marginal bands for single periods, do not contain information about probability of paths Bundesbank Bank of England

Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 4 / 18

Introduction - An Example

Suppose yt = ρyt−1 + εt, εt ∼ N (0, 1) Then 1- and 2-step-ahead forecast errors are distributed as ut+1 ut+2

• ∼ N
• ,

1 ρ ρ 1 + ρ2

• Forecast errors have a joint distribution, thus forecast paths can be

ranked according to their p-values

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

• 4
• 3
• 2
• 1

1 2 3 4 forecast horizon y(t) upper marginal lower marginal path 1 path 2 path 3 path 4 path 5 path 6 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

• 4
• 3
• 2
• 1

1 2 3 4 forecast horizon y(t) upper marginal lower marginal path 1 path 2 path 3 path 4 path 5 path 6

ρ = 0.9, 95% marginal confidence bands

path 1: p-value 0.06 path 2: p-value 0.00 path 3: p-value 0.08 paths 4,5,6: p-value 0.05 bands: p-value 0.10

Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 5 / 18

Introduction - Example Cont’d

p-values correspond to multiple testing procedure with H0 : ut+1 = 0 ut+2 = 0, resulting in a confidence ellipse. Looking at the forecast error distribution from above (ρ = 0.9):

• 2

2

• 3
• 2
• 1

1 2 3 u(t+1) u(t+2) 95% conf. ellipse 2 s.e. box Scheffé box

Drawing confidence bands produces rectangular regions ⇒ Confidence ellipse cannot be represented in fan charts. But rectangular region can at least be chosen such that confidence bands have correct “Wald coverage”

Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 6 / 18

Introduction - Scheffé Bands

Correct "Wald coverage" means: We want 100·α% of all paths to have a lower p-value than the confidence bands Scheffé bands proposed by Jorda (2009) have correct Wald coverage (or FDR control = false discovery rate control), are easy to construct, have interesting statistical interpretation. But what about their coverage in the conventional sense? That is, what is the percentage of paths lying within the Scheffé bands given α? Equivalent question: How good is their FWER control ( = family-wise error control)? Marginal bands fail at FWER control, dramatically so when H is large Another alternative: Bonferroni bands, based on inequality Pr (A1 ∪ A2 ∪ . . . ∪ AH) ≤ ∑H

i=1 Pr (Ai), equivalent to marginal bands

with critical value α

H

Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 7 / 18

Introduction - Example Cont’d

Forecast errors ut+1 ut+2

• ∼ N
• ,

1 ρ ρ 1 + ρ2

• Forecast path given by ˆ

Y (H) Set α = 0.05 Marginal bands: ˆ Y (H) ± 1.96

• 1
• 1 + ρ2
• Bonferroni bands: ˆ

Y (H) ± 2.24

• 1
• 1 + ρ2
• Scheffé bands: ˆ

Y (H) ± 1 ρ 1

5.99 2

1 1

• Note: Only Scheffé bands use entire covariance matrix, other bands
• nly use diagonal elements

Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 8 / 18

Introduction - Example Cont’d

Confidence bands for H = 2 and H = 12, α = 0.05, ρ = 0.9

0.5 1 1.5 2

• 4
• 3
• 2
• 1

1 2 3 4 forecast horizon y(t) marginal Bonferroni Scheffé 2 4 6 8 10 12

• 10
• 5

5 10 forecast horizon y(t) marginal Bonferroni Scheffé

Marginal bands are always narrower than Bonferroni bands Scheffé bands here are narrower than marginal bands at the beginning, wider than Bonferroni bands at the end

Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 9 / 18

Introduction - Example Cont’d

Coverage of bands, H = 2, α = 0.32

0.2 0.4 0.6 0.8 1 0.45 0.5 0.55 0.6 0.65 0.7 0.75 ρ FWER control nominal marginal Bonferroni Scheffé 0.2 0.4 0.6 0.8 1 0.4 0.5 0.6 0.7 0.8 0.9 ρ FDR control nominal marginal Bonferroni Scheffé

Typical result: Bonferroni bands are too conservative, marginal bands have too small coverage Scheffé bands have exact Wald coverage and too small coverage with respect to FWER control

Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 10 / 18

Empirical Confidence Regions

Using past forecast errors to estimate variance-covariance matrix (vcv matrix) reasonable if

forecasting model is misspecified forecasting model is not available

Both reasons highly relevant in macroeconomics, probably therefore fan charts of most central banks based on past forecast errors Potential problem of Scheffé bands: Larger estimation uncertainty, because entire vcv matrix has to be estimated Marginal and Bonferroni bands only require estimation of diagonal elements of vcv matrix Coverage of bands investigated with MC simulations Set-up similar to Jorda & Marcellino, but with misspecified models

Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 11 / 18

Simulation Design

DGP is Stock-Watson (2001, JEP) VAR(4) with variables unemployment, inflation, and federal funds rate, normally distributed shocks Coverage investigated for model-based and forecast-error-based (i.e. empirical) confidence bands Misspecifications of forecasting models:

dynamic (1 lag instead of 4)

• mitted variable (unemployment missing)

break in coefficients of DGP break in vcv matrix of DGP both breaks mentioned in DGP (all breaks assumed to occur in 1985)

Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 12 / 18

Results

Marginal bands are too narrow Bonferroni bands have good FWER control, but are often too narrow wrt to FDR control Scheffé bands have good FDR and reasonable FWER control Empirical Scheffé bands work well even if number of empirical forecast errors q is small (unless H is very large) For example, with H = 8 and q = 40, coverage is still not too far from nominal level For most misspecifications, model-based and empirical bands have similar coverage But break in vcv matrix leads to very poor results of all model-based bands, while empirical Scheffé bands retain good coverage

Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 13 / 18

Application

We look at the Greenbook forecasts for US growth (annualized quarterly growth rates of GNP/GDP) and inflation (annualized quarterly growth rates of GNP/GDP deflator) Choosing H = 5 gives 119 path forecasts from 1974q2 to 2003q4 In addition to final release data, real-time data are used to calculate forecast errors Normality tests of forecast errors do not reject in most cases We construct empirical bands based on a rolling window of forecast errors for 40 path forecasts This yields bands for 75 path forecasts with the first path starting in 1985q2 For these bands and path forecasts, we calculate the coverage rates

Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 14 / 18

Application - Growth

Growth, FWER control coverage in % 50 68 95 Marg Bonf Schef Marg Bonf Schef Marg Bonf Schef vintage 1 19 67 25 32 73 44 77 91 80 vintage 5 16 68 35 33 76 49 77 96 91 vintage 10 11 69 32 33 77 49 81 97 95 final release 9 64 27 27 72 47 73 100 85 Growth, FDR control coverage in % 50 68 95 Marg Bonf Schef Marg Bonf Schef Marg Bonf Schef vintage 1 31 76 49 43 84 67 85 97 95 vintage 5 29 72 49 43 85 67 85 97 93 vintage 10 24 73 53 47 80 76 84 97 97 final release 32 81 47 40 83 67 85 99 99

Empirical Scheffé bands have very good coverage wrt FDR control Bonferroni bands perform best at FWER control Confirmation of results from MC simulations is at least partly surprising, given likely structural changes of forecast error distribution in sample

Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 15 / 18

Application - Inflation

Inflation, FWER control coverage in % 50 68 95 Marg Bonf Schef Marg Bonf Schef Marg Bonf Schef vintage 1 12 76 44 33 80 60 85 96 92 vintage 5 15 76 44 39 80 57 84 89 88 vintage 10 24 65 45 39 77 63 80 92 87 final release 16 71 47 31 79 60 88 92 93 Inflation, FDR control coverage in % 50 68 95 Marg Bonf Schef Marg Bonf Schef Marg Bonf Schef vintage 1 4 84 61 28 89 80 91 96 96 vintage 5 12 81 59 40 87 77 89 92 91 vintage 10 25 72 61 51 79 72 81 89 89 final release 3 65 67 16 73 75 75 88 91

Empirical Scheffé bands in general have best coverage wrt FDR control Bonferroni bands perform best at FWER control and α = 0.05, similar deviations from nominal coverage as Scheffé bands for α = 0.32, α = 0.50 Again, confirmation of results from MC simulations is at least partly surprising

Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 16 / 18

Application - Inflation Cont’d

One-sided bands for α = 0.32 when all forecast errors and final release data are used

0.00 1.00 2.00 3.00 h=0 h=1 h=2 h=3 h=4 Marg Bonf Schef

Scheffé bands increase markedly with h while other bands do not This indicates that inflation forecast errors have large correlations ignored by marginal and Bonferroni bands Hence, inflation path forecasts are more uncertain than flat shape of marginal and Bonferroni bands suggests

Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 17 / 18

Summary

Path forecasts are often more important than single-period forecasts Uncertainty bands of path forecasts are related to multiple testing procedure, should have good FDR control Scheffé bands have good FDR and reasonable FWER control In practice, empirical bands are often more relevant than model-based bands Empirical Scheffé bands have good FDR and reasonable FWER control, even if sample of forecast errors is relatively small (unless H is very large) Empirical Scheffé bands are superior to model-based bands in the presence of strong model misspecification We find good FDR and reasonable FWER control of empirical Scheffé bands for Greenbook growth and inflation forecasts

Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 18 / 18