How far can we forecast? Statistical tests of the predictive content - PowerPoint PPT Presentation

intro test statistics MC sims empirical results How far can we forecast? Statistical tests of the predictive content Jörg Breitung and Malte Knüppel University of Cologne and Deutsche Bundesbank 9. September 2017 Workshop on Forecasting Deutsche Bundesbank, Frankfurt

intro test statistics MC sims empirical results The null hypothesis • Assume that Y t is stationary and ergodic • Let � Y t + h | t denote the forecast based on the information set I t • The forecast is uninformative if var ( Y t + h − � H 0 : ) = var ( Y t + h − µ ) Y t + h | t � �� � � �� � u t + h e t + h | t • Since E ( e 2 t + h | t ) = E [( Y t + h − µ ) − ( � Y t + h | t − µ )] 2 ⇒ sufficient (but not necessary) condition for an uninformative forecast is � Y t + h | t = µ • For rational forecasts with E ( e t + h | � Y t + h | t ) = 0 it follows that E ( Y t + h − µ )( � Y t + h | t − µ ) = E ( e t + h | t + � Y t + h | t − µ )( � Y t + h | t − µ ) Y t + h | t − µ ) 2 = E ( � ⇒ H 0 is equivalent to cov ( Y t + h , � Y t + h | t ) = 0.

intro test statistics MC sims empirical results • Maximum forecast horizon There exists some h ∗ such that var ( e t + h | t ) ≥ var ( u t + h ) for h > h ∗ H 0 : h ∗ is called the maximum forecast horizon • Sequential test of H 0 for h = 1 , 2 , . . . , h max . Stop when H 0 is not rejected for first time. Previous horizon is ˆ h ∗ . • Non-stationary variables: � h ∆ � Y t + h | t = Y t + Y t + s | t s = 1 � h � h (∆ Y t + s − ∆ � e t + h | t = ∆ e t + s | t = Y t + s | t ) s = 1 s = 1 ⇒ Non-predictability of Y t + h equivalent to non-predictability of ∆ Y t + s for s = 1 , . . . , h

intro test statistics MC sims empirical results Earlier work a) Theil’s (1958) inequality coefficient: �� n t = 1 ( Y t + h − � Y t + h | t ) 2 U 2 ( h ) = �� n t = 1 ( Y t + h − Y 0 t + h ) 2 where Y 0 t + h denotes some “naive forecast” (typically “no-change forecast”) ⇒ forecast uninformative if U 2 ( h ) = 1 b) Nelson (1976) or Granger-Newbold (1986) measure: R 2 ( h ) = 1 − var ( e t + h | t ) var ( Y t + h ) c) Diebold-Kilian (2001) forecastability measure: Q ( L , h , k ) = 1 − E [ L ( e t + h | t )] where k > h E [ L ( e t + k | t )]

intro test statistics MC sims empirical results • Note that for (i) stationary variables and (ii) MSE as the loss function: k →∞ Q ( MSE , h , k ) = R 2 ( h ) lim • Our approach is based on R 2 ( h ) (resp. MSE DIFF) • We propose tests for the limiting horizon h ∗ beyond which forecasts become uninformative • Empirical work suggests that economic forecasts of macroeconomic key variables (output growth, inflation) are informative 2-6 quarters ahead (or even less) • Our empirical application based on survey forecasts from Consensus Economics indicates a maximum forecast horizon of typically less than one year

intro test statistics MC sims empirical results Maximum forecast horizons in quarters US EA JP DE UK IT CA FR median GDP q-o-q DM-type test 2 2 1 1 3 1 1 2 1 . 5 encompassing test 2 2 1 2 3 5 1 4 2 CPI y-o-y DM-type test 3 5 3 2 2 3 4 3 3 encompassing test 3 3 4 3 3 3 4 3 3 PrivCons q-o-q DM-type test 3 1 0 0 − 1 1 0 2 0 . 5 encompassing test 3 3 0 3 3 3 1 5 3 d(3m rate) DM-type test 1 0 3 2 2 1 2 2 encompassing test 2 0 2 6 3 1 2 2 Note: Regions considered are USA, Euro area, Japan, Germany, UK, Italy, Canada, and France. Variables are growth rates of real GDP, CPI, and real private consumption, and 1st differences of interest rates. h = 0 refers to the nowcast.

intro test statistics MC sims empirical results Notation and assumptions • Forecast results from replacing θ by some estimator � θ such that � � θ Y t + h | t = Y t + h | t • The forecast evaluation may be based on three different schemes: {− T + 1 , − T + 2 , . . . , t } recursive: { t − T + 1 , t − T + 2 , . . . , t } rolling: fixed: { t − T + 1 , . . . , 0 } • We assume that we only observe actual values and forecasts but do not know (i) the forecasting model and (ii) the data used for estimating the model • Forecast errors: e t + h | t = Y t + h − Y θ t + h | t e t + h | t = Y t + h − � � Y t + h | t

intro test statistics MC sims empirical results Assumption 1: (Time series process for Y t ) Let Y t = µ + u t with u t = φ ( L ) ε t , φ ( L ) = 1 + φ 1 L + φ 2 L 2 + · · · is a lag polynomial with all roots outside the unit circle, � ∞ i = 1 | φ i | < ∞ and ε t is an i.i.d. white noise process with E ( ε t ) = 0 and E ( ε 2 t ) = σ 2 ε . Furthermore E | ε t | 2 + δ < ∞ for some δ > 0 . Assumption 2: (Properties of the forecast) (i) Under H 0 : u t + h = Y t + h − µ is independent of the past estimation error � θ t − θ , � θ t − 1 − θ, . . . . (ii) The parameters are estimated consistently with � √ t � θ 0 − θ = O p ( T − 1 / 2 ) , � � θ t − � a ) b ) θ 0 = O p T t + h | t /∂θ and D h ( θ ) = n − 1 � n (iii) Let D t + h ( θ ) = ∂ Y θ t = 1 D t + h ( θ ) � n 1 2 with 0 < D 2 < ∞ p ( D t + h ( θ ) − D h ) 2 → D n t = 1 E | D t + h ( θ ) u t + h | 2 + δ < ∞ for some δ > 0 and all t .

intro test statistics MC sims empirical results Diebold-Mariano-type test Comparing model forecast with unconditional mean Y h : � � 2 δ h e 2 loss diff. t = � t + h | t − Y t + h − Y h DM (1995) statistic: n � 1 δ h d h = √ n t , ω δ � t = 1 ω 2 δ denotes the estimated long-run variance of δ h where � t . Theorem 1: Asymptotic distribution of the DM statistic: If T → ∞ , n → ∞ , n / T → 0 we have � n � d h = √ n | u | → | z | d + O p 2 , 2 � ω u T where z ∼ N ( 0 , 1 ) . ⇒ non-standard as under H 0 the forecasts are nested

intro test statistics MC sims empirical results Modified DM statistic Theorem 1 suggests the 2 adjusted DM statistics d → |N ( 0 , 1 ) | 2 d h � n d h = 1 � d δ h → χ 2 t 1 ω 2 � u t = 1 ω 2 where � u is a consistent estimator for the long-run variance of u t = y t − y . • 5% critical values are 0 . 0627 and 0 . 0039, resp. ⇒ large size distortions • Under H 1 : 2 d h = O p ( √ n ) and � d h = O p ( n ) . Nevertheless the local power is identical. • If the model-based forecast is biased, the tests become conservative

intro test statistics MC sims empirical results Actual sizes for various n / T combinations ( α = 0 . 05) n = 25 n = 50 n = 100 n = 200 � � � � T 2 d 1 d 1 2 d 1 d 1 2 d 1 d 1 2 d 1 d 1 50 0.089 0.094 0.066 0.070 0.044 0.047 0.027 0.029 100 0.105 0.110 0.089 0.093 0.065 0.069 0.043 0.046 200 0.114 0.121 0.105 0.111 0.088 0.093 0.065 0.069 500 0.116 0.123 0.115 0.122 0.106 0.112 0.094 0.099 1000 0.110 0.117 0.116 0.123 0.114 0.121 0.108 0.114 ∞ 0.049 0.049 0.049 0.049 0.050 0.050 0.051 0.051 Note: For T = ∞ the test statistics are computed using the true parameter values. Results are based on 100,000 replications.

intro test statistics MC sims empirical results Encompassing test • Modifications based on the following decomposition: � � 2 � n � n � � 2 δ h Y t + h − Y h − ( � t = Y t + h | t − Y h ) − Y t + h − Y h t = 1 t = 1 n n � � Y t + h | t − Y h ) 2 ( � ( Y t + h − Y h )( � = − 2 Y t + h | t − Y h ) t = 1 t = 1 � �� � always positive • First term does not contribute to power • Reject H 0 if correlation between Y t + h and � Y t + h | t is large • LM-type test statistic: n � 1 ξ h ̺ h = √ n � t ω ξ t = 1 t = ( Y t + h − Y h )( � Y t + h | t − � where ξ h Y h )

intro test statistics MC sims empirical results ω 2 and � ξ denotes the corresponding long-run variance k � ω 2 γ ξ w k γ ξ � ξ = � 0 + 2 j � j j = 1 � n j = 1 γ ξ ξ t ξ t − j . � n t = j + 1 • Note that this test is asymptotically equivalent to the Mincer-Zarnowitz regression: Y t + h = β 0 , h + β 1 , h � Y t + h | t + u t + h but with H 0 : β 1 , h = 0 instead of β 1 , h = 1 • Encompassing test: Y t + h = λ � Y t + h | t + ( 1 − λ ) Y h + v t + h Y t + h − Y h = λ ( � Y t + h | t − Y h ) + v t + h with H 0 : λ = 0

intro test statistics MC sims empirical results Theorem 2: Under Assumption 1–2, a recursive forecasting scheme with h > h ∗ , T → ∞ , n → ∞ and n / T → 0 we have d ̺ h → N ( 0 , 1 ) • This might look trivial but is not. In the proof we show that Y t + h | t − Y h ≈ ( � � θ 0 − θ ) D t + h ( θ ) p and thus the regressor tends to zero as � θ → θ • Asymptotically, the test is equivalent to the regression Y t + h = β ∗ 0 , h + β ∗ 1 , h D t + h ( � θ ) + η t + h

intro test statistics MC sims empirical results Local power Assume that the target value is generated as � c � √ n Y t + 1 = µ + X t + u t + 1 such that √ regression forecast error: u t + 1 + O p ( n / T ) u t + 1 − u 1 + ( c / √ n )( X t − X ) unconditional forecast error Theorem 4: Under the sequence of alternatives β = c / √ n , X t ∼ iid ( 0 , σ 2 x ) , √ Assumptions 1 – 2 and n / T → 0 it follows that � d → z 2 1 − 2 λ z 2 − λ 2 d 1 d → sign ( c ) z 2 + λ � ̺ 1 where λ 2 = c 2 σ 2 x /σ 2 u is the signal-to-noise ratio and z 1 and z 2 are independent N ( 0 , 1 ) ⇒ tests are NOT asymptotically equivalent