An Empirical Investigation of Direct and Iterated Multistep - - PowerPoint PPT Presentation

An Empirical Investigation of Direct and Iterated Multistep Conditional Forecasts

Michael W. McCracken Joseph T. McGillicuddy

Federal Reserve Bank Federal Reserve Bank

• f St. Louis
• f St. Louis1

Deutsche Bundesbank September 9, 2017

1The views expressed herein are solely those of the authors and do not necessarily

reflect the views of the Federal Reserve Bank of St. Louis or the Federal Reserve System.

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 1 / 27

Introduction

Central banks use conditional forecasts to assess hypothetical policies. Christoffel et al. (ECB, 2008) Large banks have to construct conditional forecasts as part of stress testing exercises. Sarychev (BoE, 2014) Recent surge in conditional forecasting in academic literature:

Giannone et al. (NY Fed, 2014): Big VARs Baumeister and Kilian (BoC, 2013): Oil Aastveit et al. (NB, 2014): Break tests Clark and McCracken (Feds, 2017): Inference

All of these are VARs or near-VARs using IMS approach

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 2 / 27

Does DMS vs. IMS Matter for Forecasting?

For unconditional point forecasts, theory says “Yes.”

Bhansali (1997) Schorfheide (2005)

For unconditional point forecasts, empirics say “Yes.”

Marcellino, Stock, and Watson (MSW; 2006)

For impulse response functions, empirics say “Yes.”

Jorda (2008) McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 3 / 27

What We Do

We provide empirical evidence on whether the DMS vs IMS battle matters for conditional point forecasts.

IMS is MSE optimal from OLS VAR (Waggoner and Zha, 1999) DMS is just OLS BLP (Goldberger, 1962)

We do so by shamelessly emulating what MSW did.

2000 bivariate systems used to construct MSEs 150 trivariate “monetary” systems used to construct MSEs

Scenarios are ex-post realized actuals so that scenarios actually occur and MSE is relevant.

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 4 / 27

Overview of Results

Whether our results coincide with MSW is sample dependent

Over common sample, our results align with theirs: IMS generally

favored with some benefits to DMS for nominals

For Great Moderation sample, DMS heavily favored for nominals and

about the same as IMS for reals/financials

Robustness

Bivariate and trivariate horizons: h = 3, 6, 12, 24 lag selection: fixed at 4 or 12, or AIC/BIC real-time vintage data McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 5 / 27

What We Don’t Do

Consider bayesian estimation. Consider large VARs. Consider univariate ARDLs (Guerrieri and Welch 2014) or alternative DMS in Jorda and Marcellino (2008).

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 6 / 27

DMS vs. IMS examples

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 7 / 27

Examples of DMS vs. IMS

VAR(1) taking the form yt xt

• =

b c yt−1 xt−1

• +

et vt

• ,

with i.i.d. N(0,1) errors with contemporaneous correlation ρ. Use pseudo-true parameters used for forecasting One-step conditional forecast of yt+1 given xt+1 DMS model is trivial: yt = γxt + ηt

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 8 / 27

Correct Specification

IMS forecast is ˆ yc

t,1 = bxt + ρ(xt+1 − cxt)

E(eIMS

t,1 )2 = 1 − ρ2.

DMS forecast is ˆ yc

t,1 = (bc + ρ(1 − c2))xt+1

E(eDMS

t,1

)2 = 1 − ρ2 + (b − ρc)2.

Minimum-MSE IMS better.

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 9 / 27

VAR Misspecification of Conditional Mean

Equation for x misspecified as xt = αxt−2 + ηt IMS forecast is ˆ yc

t,1 = bxt + ρ √ 1+c2 (xt+1 − c2xt)

E(eIMS

t,1 )2 = 1 + ρ2 − 2ρ2 √ 1+c2

DMS forecast is ˆ yc

t,1 = (bc + ρ(1 − c2))xt+1

E(eDMS

t,1

)2 = 1 − ρ2 + (b − ρc)2.

DMS better if b = ρ2 and c = ρ.

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 10 / 27

VAR Misspecification of Residual Variance

Conditional mean ok. Residual correlation changes from ρ0 to ρ1 between T and T + 1. IMS forecast is ˆ yc

T ,1 = bxT + ρ0(xT +1 − cxT )

E(eIMS

T ,1 )2 = 1 + ρ2 0 − 2ρ0ρ1

DMS forecast is ˆ yc

T ,1 = (bc + ρ0(1 − c2))xT +1

E(eDMS

T ,1 )2 = 1 + b2 + ρ2 0(1 − c2) − 2bcρ1 − 2ρ0ρ1(1 − c2).

DMS better if b = 0 and c ∈ (0, 1), ρ0 ∈ (−1, 0), and ρ1 ∈ (0, 1).

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 11 / 27

Methodology

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 12 / 27

VAR-based IMS Conditional Forecasts

Zt = (Yt, Xt) in levels or log-levels. Let zt = (yt, xt) denote the stationary transformation: level or differences. OLS estimate of VAR for zt = (yt, xt): zt = C + A(L)zt−1 + et used to produce h-step conditional forecasts of the form ˆ yc

t,h = ˆ

yu

t,h + ∑1≤i≤h ˆ

γi,t(xt+i − ˆ xu

t,i)

for constants ˆ γi,t that are known functions of ˆ Ai,t, and ˆ Σt. ˆ Y c

t,h =

   ˆ yc

t,h

if Yt is I(0) Yt + ∑h

i=1 ˆ

yc

t,i

if Yt is I(1) Yt + h∆Yt + ∑h

i=1 ∑i j=1 ˆ

yc

t,j

if Yt is I(2)   .

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 13 / 27

ARDL-based DMS Conditional Forecasts

Define y (h)

t

=    Yt if Yt is I(0) Yt − Yt−h if Yt is I(1) Yt − Yt−h − h∆Yt if Yt is I(2)    . OLS estimate of ARDL for zt = (yt, xt): y (h)

t

= α + ∑p−1

j=0 βjyt−h−j + ∑p−1 j=0 δjxt−h−j + (∑1≤i≤h γixt−h+i) + εt

used to produce h-step conditional forecasts of the form ˆ yc(h)

t,h

= ˆ αt + ∑p−1

j=0 ˆ

βj,tyt−j + ∑p−1

j=0 ˆ

δj,txt−j + (∑1≤i≤h ˆ γi,txt+i) ˆ Y c

t,h computed relative to order of integration of Y .

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 14 / 27

Choices

Lag selection: 4, 12, or updated AIC/BIC Horizons: 3, 6, 12, 24 Paths: Future x known throughout forecast horizon In-sample: Start in 1959:01 or 1984:01 Out-of-sample: 1979:01+h — 2002:12 or 2002:12+h — 2016:12

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 15 / 27

Bivariate Data

Bivariate results use June 2017 vintage of 121 monthly series taken from FRED-MD (McCracken and Ng, 2016).

Series mapped into the 5 groups MSW have:

a) Income, output, sales, and capacity utilization b) Employment and unemployment c) Construction, inventories, and orders d) Interest rates and asset prices e) Nominal prices, wages, and money

MSW have 170 series: differences mostly come from groups (a) and (c)

above

The 2,000 bivariate systems are obtained at random by selecting pairs (y, x) from distinct groups (200 from each of the 10 possible group pairs). From these, 4,000 forecasts made for each lag-selection method, horizon, and forecasting approach (IMS or DMS).

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 16 / 27

Trivariate Data

For trivariate results we use 6 real, 5 nominal, and 5 financial series

Real Nominal 1 real personal income 1

• avg. hourly mfg. earnings

2 IP 2 PPI 3 employment 3

• il price

4 unemployment rate 4 CPI 5

• avg. weekly mfg. hours

5 PCEPI 6 real personal consumption Financial 1 M1 money stock 2 fed funds rate 3 10-year treasury 4 trade-weighted US dollar index 5 S&P 500

Most results from June 2017 vintage but some from real-time data The 150 trivariate systems provide 450 forecasts for each lag-selection method, horizon, and forecasting approach (IMS or DMS).

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 17 / 27

Empirical Results

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 18 / 27

Bivariate Results: Conditional on Full Path, Sample Start 1959:01, Out-of-Sample Period 1979:01+h to 2002:12

(A) All variables (B) Pairs excl. PWM (C) PWM variables Forecast Horizon: 3 12 24 3 12 24 3 12 24 AR(4) Mean 1.00 1.04 1.11 1.00 1.05 1.12 0.98 0.92 0.96 Median 1.00 1.02 1.07 1.01 1.04 1.09 0.98 0.91 0.94 IMS better 3.2% 7.5% 11.2% 3.5% 8.6% 12.8% 0.8% 0.0% 1.0% DMS better 5.7% 8.2% 5.9% 5.2% 4.3% 2.8% 10.5% 27.3% 19.6% AR(12) Mean 1.02 1.08 1.18 1.02 1.08 1.18 1.01 1.03 1.09 Median 1.01 1.06 1.13 1.01 1.06 1.13 1.01 1.03 1.06 IMS better 5.3% 10.4% 15.3% 4.8% 11.5% 16.7% 6.4% 6.4% 4.5% DMS better 0.9% 1.3% 2.0% 0.8% 0.9% 1.3% 1.6% 3.3% 5.3% BIC Mean 0.95 0.96 1.04 0.98 0.99 1.09 0.84 0.72 0.76 Median 0.99 0.98 1.04 0.99 0.99 1.06 0.83 0.70 0.73 IMS better 3.7% 6.3% 9.6% 4.5% 7.0% 11.6% 0.5% 0.1% 0.8% DMS better 17.7% 16.1% 11.3% 12.6% 8.3% 4.4% 46.3% 53.5% 41.0%

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 19 / 27

Bivariate Results: Conditional on Full Path, Sample Start 1959:01, Out-of-Sample Period 2002:12+h to 2016:12

(A) All variables (B) Pairs excl. PWM (C) PWM variables Forecast Horizon: 3 12 24 3 12 24 3 12 24 AR(4) Mean 1.00 1.00 1.06 1.01 1.00 1.04 0.98 0.88 0.88 Median 1.00 0.99 1.00 1.00 1.00 1.01 0.98 0.85 0.80 IMS better 6.3% 3.4% 5.7% 7.7% 3.7% 5.9% 0.4% 0.4% 1.6% DMS better 4.8% 7.6% 9.6% 5.4% 5.5% 7.3% 5.5% 18.9% 23.0% AR(12) Mean 1.01 1.01 1.07 1.01 1.03 1.07 0.99 0.90 0.90 Median 1.01 1.01 1.03 1.01 1.03 1.05 0.99 0.90 0.84 IMS better 4.6% 5.2% 8.3% 5.3% 5.5% 6.4% 2.8% 2.0% 4.6% DMS better 1.9% 3.0% 5.1% 1.9% 3.3% 5.3% 2.1% 2.4% 7.1% BIC Mean 0.96 0.94 0.99 0.97 0.95 1.01 0.88 0.70 0.68 Median 0.99 0.97 0.98 0.99 0.97 0.98 0.88 0.62 0.54 IMS better 4.2% 3.0% 5.3% 4.3% 2.2% 4.9% 0.6% 0.4% 1.6% DMS better 10.2% 9.0% 10.4% 11.5% 5.6% 7.0% 13.1% 25.4% 28.9%

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 20 / 27

Bivariate Results: Conditional on Full Path, Sample Start 1984:01, Out-of-Sample Period 2002:12+h to 2016:12

(A) All variables (B) Pairs excl. PWM (C) PWM variables Forecast Horizon: 3 12 24 3 12 24 3 12 24 AR(4) Mean 1.00 0.98 1.01 1.00 1.00 1.04 0.98 0.82 0.78 Median 1.00 0.97 0.95 1.00 0.99 0.99 0.98 0.80 0.73 IMS better 2.5% 2.3% 4.3% 2.9% 2.5% 5.5% 0.1% 1.0% 1.3% DMS better 1.5% 3.8% 7.0% 1.8% 2.2% 4.0% 2.0% 12.3% 21.6% AR(12) Mean 1.01 1.00 1.03 1.02 1.03 1.06 0.99 0.83 0.82 Median 1.01 1.01 0.99 1.02 1.03 1.03 0.99 0.85 0.77 IMS better 4.0% 4.1% 5.5% 4.5% 4.8% 6.8% 0.8% 0.3% 1.5% DMS better 0.6% 1.6% 3.6% 1.0% 1.4% 3.1% 0.0% 3.4% 8.0% BIC Mean 0.97 0.92 0.94 0.98 0.98 1.01 0.89 0.59 0.55 Median 0.99 0.95 0.93 0.99 0.97 0.97 0.85 0.53 0.43 IMS better 2.3% 2.7% 3.5% 2.6% 3.0% 4.1% 0.8% 0.9% 1.9% DMS better 5.5% 8.4% 10.3% 5.6% 2.0% 3.4% 9.8% 34.9% 39.9%

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 21 / 27

Trivariate Results: Conditional on Full Path, Sample Start 1959:01, Out-of-Sample Period 1979:01+h to 2002:12

(A) Real (B) Nominal (C) Financial Forecast Horizon: 3 12 24 3 12 24 3 12 24 AR(4) Mean 1.03 1.12 1.24 1.00 0.97 1.01 1.02 1.06 1.08 Median 1.02 1.09 1.19 1.00 0.95 0.98 1.01 1.06 1.04 IMS better 13.0% 11.7% 19.3% 0.0% 0.0% 1.0% 5.7% 12.7% 19.3% DMS better 0.0% 0.0% 0.0% 1.7% 6.7% 3.7% 0.0% 4.7% 7.0% AR(12) Mean 1.04 1.22 1.48 1.01 1.05 1.12 1.01 1.08 1.15 Median 1.03 1.22 1.42 1.01 1.06 1.06 1.01 1.08 1.08 IMS better 9.7% 32.0% 29.0% 4.7% 4.7% 2.0% 11.7% 9.0% 20.0% DMS better 0.0% 1.3% 0.0% 0.0% 0.3% 1.3% 0.0% 0.0% 1.0% BIC Mean 1.00 1.07 1.15 0.76 0.63 0.64 0.98 0.96 0.99 Median 1.00 1.06 1.09 0.77 0.62 0.64 1.01 1.02 1.00 IMS better 4.7% 6.7% 11.7% 0.0% 0.0% 0.0% 5.0% 6.7% 17.7% DMS better 2.0% 1.7% 0.3% 77.7% 75.7% 55.0% 11.3% 18.3% 12.3%

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 22 / 27

Trivariate Results: Conditional on Full Path, Sample Start 1959:01, Out-of-Sample Period 2002:12+h to 2016:12

(A) Real (B) Nominal (C) Financial Forecast Horizon: 3 12 24 3 12 24 3 12 24 AR(4) Mean 1.00 1.03 1.09 1.00 0.85 0.86 1.02 1.08 1.18 Median 1.01 1.03 1.09 0.99 0.85 0.85 1.01 1.02 1.05 IMS better 5.7% 10.7% 10.3% 1.0% 0.0% 0.0% 13.3% 8.7% 9.7% DMS better 1.3% 0.3% 1.7% 0.0% 17.7% 17.7% 0.0% 1.3% 1.0% AR(12) Mean 1.03 1.10 1.10 1.00 0.91 0.88 1.03 1.08 1.23 Median 1.02 1.09 1.10 1.00 0.88 0.85 1.01 1.04 1.09 IMS better 3.3% 6.0% 11.3% 0.7% 0.3% 1.7% 2.0% 0.7% 5.3% DMS better 1.3% 0.0% 2.3% 2.0% 2.3% 6.0% 0.3% 0.0% 0.3% BIC Mean 0.95 1.05 1.10 0.82 0.55 0.52 0.99 0.99 1.09 Median 0.97 1.03 1.09 0.83 0.54 0.51 1.00 0.96 1.05 IMS better 2.0% 6.0% 10.7% 0.0% 0.0% 0.0% 7.7% 9.0% 8.3% DMS better 1.7% 0.3% 2.3% 11.3% 64.3% 64.7% 0.7% 4.0% 5.7%

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 23 / 27

Trivariate Results: Conditional on Full Path, Sample Start 1984:01, Out-of-Sample Period 2002:12+h to 2016:12

(A) Real (B) Nominal (C) Financial Forecast Horizon: 3 12 24 3 12 24 3 12 24 AR(4) Mean 1.00 0.94 0.97 1.00 0.76 0.69 1.01 1.03 1.03 Median 1.00 0.93 0.93 1.00 0.78 0.70 1.00 0.96 1.00 IMS better 6.3% 1.7% 5.0% 0.0% 0.0% 0.0% 2.0% 2.7% 5.7% DMS better 0.7% 0.0% 1.7% 0.0% 23.0% 31.0% 0.0% 0.3% 4.7% AR(12) Mean 1.03 1.06 1.01 1.00 0.74 0.61 1.02 1.08 1.12 Median 1.03 1.04 0.98 1.00 0.70 0.56 1.02 1.06 1.07 IMS better 3.3% 1.0% 5.3% 0.0% 0.0% 0.0% 5.7% 7.3% 4.3% DMS better 1.0% 0.0% 0.0% 0.0% 3.3% 8.7% 0.0% 0.0% 2.0% BIC Mean 0.96 0.99 1.02 0.84 0.43 0.33 0.99 0.91 0.95 Median 0.97 0.99 1.00 0.85 0.45 0.33 1.00 0.92 0.94 IMS better 0.3% 4.7% 2.3% 0.0% 0.0% 0.0% 3.7% 0.3% 1.0% DMS better 0.3% 0.3% 0.0% 21.0% 69.3% 74.0% 1.7% 5.0% 14.0%

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 24 / 27

Trivariate Results: Unconditional, Sample Start 1959:01, Out-of-Sample Period 1979:01+h to 2002:12

(A) Real (B) Nominal (C) Financial Forecast Horizon: 3 12 24 3 12 24 3 12 24 AR(4) Mean 1.03 1.03 1.11 1.00 0.92 0.90 1.02 1.05 1.08 Median 1.02 1.02 1.10 1.00 0.92 0.90 1.01 1.04 1.06 IMS better 13.3% 10.0% 3.3% 0.0% 0.0% 0.0% 6.7% 18.7% 22.0% DMS better 0.0% 0.0% 0.0% 2.0% 21.3% 30.0% 0.0% 10.0% 2.7% AR(12) Mean 1.04 1.21 1.34 1.01 1.04 0.98 1.01 1.08 1.15 Median 1.03 1.19 1.28 1.01 1.05 1.00 1.01 1.10 1.10 IMS better 8.0% 24.7% 10.7% 4.0% 0.0% 1.3% 8.7% 7.3% 14.7% DMS better 0.0% 0.0% 0.0% 0.0% 0.7% 6.0% 0.0% 0.0% 0.0% BIC Mean 1.01 1.03 1.07 0.77 0.61 0.57 0.98 0.96 0.97 Median 1.01 1.03 1.05 0.77 0.60 0.61 1.00 1.02 1.02 IMS better 6.0% 23.3% 0.7% 0.0% 0.0% 0.0% 3.3% 2.7% 12.7% DMS better 2.0% 2.7% 0.0% 76.7% 87.3% 78.7% 10.0% 20.0% 8.0%

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 25 / 27

Trivariate Results: Unconditional, Sample Start 1984:01, Out-of-Sample Period 2002:12+h to 2016:12

(A) Real (B) Nominal (C) Financial Forecast Horizon: 3 12 24 3 12 24 3 12 24 AR(4) Mean 1.00 0.95 0.99 1.00 0.77 0.68 1.01 0.98 0.94 Median 1.01 0.93 0.97 1.00 0.76 0.69 1.01 0.99 0.93 IMS better 2.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.7% 0.0% 3.3% DMS better 0.0% 0.0% 2.0% 0.0% 40.0% 44.7% 0.0% 6.7% 16.7% AR(12) Mean 1.03 1.05 0.94 1.00 0.69 0.52 1.02 0.92 0.85 Median 1.03 1.03 0.95 1.00 0.65 0.45 1.02 0.90 0.82 IMS better 3.3% 0.7% 4.0% 0.7% 0.0% 0.0% 6.0% 1.3% 0.0% DMS better 0.7% 0.0% 1.3% 0.0% 0.7% 13.3% 0.0% 0.7% 5.3% BIC Mean 0.97 1.02 1.05 0.87 0.46 0.37 0.99 0.89 0.86 Median 0.97 1.04 1.03 0.88 0.46 0.37 0.99 0.94 0.86 IMS better 0.7% 13.3% 4.7% 0.0% 0.0% 0.0% 1.3% 0.0% 0.7% DMS better 0.7% 1.3% 0.0% 17.3% 87.3% 90.7% 0.0% 23.3% 24.0%

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 26 / 27

Conclusion

In the context of conditional forecasts, we compare the relative accuracy of point forecasts from VAR-based IMS models to that of ARDL-based DMS models Somewhat to our surprise, DMS methods do quite well relative to “optimal” IMS methods

Despite the fact the models are obviously flawed DMS seems to have improved relative to IMS over the Great

Moderation

Large gains to forecasting nominals using DMS methods On average, it is basically a push between DMS and IMS for other

variables

McCracken and McGillicuddy Conditional DMS vs. IMS September 9, 2017 27 / 27