David de Antonio Liedo Licensed under the EUPL - - PowerPoint PPT Presentation

RESEARCH & DEVELOPMENT

STATISTICS (NBB)

Licensed under the EUPL (http://ec.europa.eu/idabc/eupl). The last updated version of the software can be downloaded here http://www.cros-portal.eu/content/jdemetra

Jean Palate David de Antonio Liedo

LINK NKIN ING G TECHNO NOLOG OGY Y IN A RE REAL AL-TIM IME E FO FORE RECA CAST STING ING ENVIRO IRONMENT ENT

Monitoring the macro economy in real time and detect etectin ing tur turning ning po poin ints ts requires certain skills and intuition

T

echn chnology logy can can help …

Red Bull Racing Chief Technical Officer Adrian Newey Source: Mark Thompson/Getty Images AsiaPac Sebastian Vettel driving for Red Bull Racing in 2010. Photographer: Andrew Hoskins at British Grand Prix

• Humans have limited

ed capaci acity ty to process information and interprete it.

• Confir

nfirma matio tion bias s is pervasive in macroeconomic forecasting.

2011Q3 2,000,000 2,050,000 2,100,000 2,150,000 2,200,000 2,250,000 2,300,000 2,350,000 2,400,000 2,450,000 2000Q1 2000Q4 2001Q3 2002Q2 2003Q1 2003Q4 2004Q3 2005Q2 2006Q1 2006Q4 2007Q3 2008Q2 2009Q1 2009Q4 2010Q3 2011Q2 2012Q1 2012Q4 2013Q3 2014Q2

EA12 GDP

Chain linked volumes (2010), million euro

LINK NKIN ING G TECHNO NOLOG OGY Y IN A RE REAL AL-TIM IME E FO FORE RECA CAST STING ING ENVIRO IRONMENT ENT

Monitoring the macro economy in real time and detect etectin ing tur turning ning po poin ints ts requires certain skills and intuition

T

echno chnology logy can can help …

• Humans have limited

ed capaci acity ty to process information and interprete it.

• Confir

nfirma matio tion bias s is pervasive in macroeconomic forecasting.

2011Q3 2,000,000 2,050,000 2,100,000 2,150,000 2,200,000 2,250,000 2,300,000 2,350,000 2,400,000 2,450,000 2000Q1 2000Q4 2001Q3 2002Q2 2003Q1 2003Q4 2004Q3 2005Q2 2006Q1 2006Q4 2007Q3 2008Q2 2009Q1 2009Q4 2010Q3 2011Q2 2012Q1 2012Q4 2013Q3 2014Q2

EA12 GDP

Chain linked volumes (2010), million euro

LINK NKIN ING G TECHNO NOLOG OGY Y IN A RE REAL AL-TIM IME E FO FORE RECA CAST STING ING ENVIRO IRONMENT ENT

Monitoring the macro economy in real time and detect etectin ing tur turning ning po poin ints ts requires certain skills and intuition

T

echno chnology logy can can help …

“Recovery gaining ground“ 25 February 2014; Winter Forecasts “Euro area’s economic recovery gradually taking hold, albeit at a slow and uneven pace “ 28 February 2014; Draghi “The euro area is turning the corner from recession to recovery” 21 January 2014; World Economic Outlook “Economic activity is projected to continue to recover as confidence improves further” May 2014; “Euro Area” in OECD Economic Outlook, Volume 2014 Issue 1, OECD Publishing “Growth becoming broader-based” 5 May 2014; Sprint Forecasts “The recovery is losing momentum “ 22 September 2014; Draghi

However, some doubts start to appear

THE FO FORE RECA CAST STING ING RA RACE

Since January 2014, international organizations’ offici ficial l commun unica icatio tions ns have been in line with the widesp espread ead believe eve that the recession is over

“Lack of evidence of sustained improvement of economic

activity “ 11 June 2014 ; EABC Dating Committee “The Demise of Wishful Thinkers“ (3 October 2014; Philippe Weil, Chair of the EABC Dating Committee; Conference in honor of André Sapir)

KEY QUESTIO STIONS  Which models produce better forecasts?  When?

Are model (a) forecasts always better than those of model (b)

• r only under certain information

assumptions?

KEY QUESTIO STIONS  Which models produce better forecasts?  When?

Are model (a) forecasts always better than those of model (b)

• r only under certain information

assumptions?

 Re-definition of the “forecast horizon” concept

Features

• Simulates forecasting scenarios using real-time data

availability (users can define the release calendar in a simple manner)

• Check whether a new model yields statistically significant

gains in forecasting accuracy with respect to alternatives

• Robust quantification of forecast accuracy as a function of

the information available (i.e. “forecast horizon”).

• Many measures of forecast accuracy and possibility to

perform analysis by subsamples

C Getty Images

Photo: Urban Events

You are the pilot

• Think about the most suitable forecasting model
• Estimate the model and assess its in-sample fit
• Before using your model out-of-sample , use
• ur “simulator” to become aware of the risks

A Real-Time Forecasting Evaluation Library

This is work in progress

Features

• Simulates forecasting scenarios using real-time data

availability (users can define the release calendar in a simple manner)

• Check whether a new model yields statistically significant

gains in forecasting accuracy with respect to alternatives

• Robust quantification of forecast accuracy as a function of

the information available (i.e. “forecast horizon”).

• Many measures of forecast accuracy and possibility to

perform analysis by subsamples

You are the pilot

• Think about the most suitable forecasting model
• Estimate the model and assess its in-sample fit
• Before using your model out-of-sample , use
• ur “simulator” to become aware of the risks
• Good luck!

A Real-Time Forecasting Evaluation Library

(α=5%) This is work in progress

• 1. WHAT IS JDEMETRA (JD) +

A Real-Time Forecasting Evaluation Library

• 1. WHAT IS JDEMETRA (JD) +
• 2. FORECAST EVALUATION

A Real-Time Forecasting Evaluation Library

• 1. WHAT IS JDEMETRA (JD) +
• 2. FORECAST EVALUATION
• 4. NEXT STEPS

AN EXAMPLE Defining the calendar Recursive estimation Real-Time simulation

A Real-Time Forecasting Evaluation Library

JDEMETRA+ is Pure Java software

• Mainly (>95%) based on libraries written by Research & Development (NBB)
• Complete control
• High-performance (compared to Matlab…)
• No economic cost for the user: Open Access software licensed under the EUPL

(http://ec.europa.eu/idabc/eupl)

• It has been designed for extension (today you will see the proof)

JDEMETRA+ provides many useful services

• Primary goal remains seasonal adjustment (TRAMO-SEATS and X12).
• Externalities: temporal disaggregation (Chow-Lin, Fernandez, Litterman),

benchmarking (Denton, Cholette), Outliers detections, chain linking, etc…

• On-going: Multivariate models (SUTSE, DFM, BVAR)
• Dynamic access to different sources: Excel, Txt, SAS, Databases…
• Rich graphical components
• Storage of current work through workspace…
• Graphical interface based on NetBeans

International Cooperation

• Maintenance partly ensured by the Bundesbank
• Support of the SA Center of Excellence (INSEE, ONS, ISTAT, EUROSTAT…)

 Related Literature: Evaluating forecasts on the basis of pseudo out-of- sample exercises is standard practice. Tricky to have realistic simulations:

Some examples for euro area GDP

Real-time publication schedule Real-time data

(instead of revised) Camacho M. and G. Pérez-Quirós (2010)

YES YES

De Antonio Liedo (2014) «Nowcasting Belgium»

YES YES

Angelini, Camba-Mendez, Giannone, Reichlin and Rünstler (2011)

stylized NO

Banbura and Modugno (2014)

stylized NO

Kuzin, Marcelino and Schumacher (2011)

stylized NO

 Related Literature: Evaluating forecasts on the basis of pseudo out-of- sample exercises is standard practice. Tricky to have realistic simulations:

Some examples for euro area GDP

Real-time publication schedule Real-time data

(instead of revised) Camacho M. and G. Pérez-Quirós (2010)

YES YES

De Antonio Liedo (2014) «Nowcasting Belgium»

YES YES

Angelini, Camba-Mendez, Giannone, Reichlin and Rünstler (2011)

stylized NO

Banbura and Modugno (2014)

stylized NO

Kuzin, Marcelino and Schumacher (2011)

stylized NO

 We propose an efficient framework to simulate the publication calendar, and to some limited extent, the real-time data.  Forecast accuracy testing and visualization

Example: How to perform a real-time forecasting simulation ?

1) Just introduce the publication delay for each series ... 2) Decide when to update your forecasts

(e.g. in this example, the days when GDP flash, employment and industrial production are released)

3) next, specify your model: SUTSE, DFM, BVAR …

πt

= Λπβt + ξt π

yt

= Z αt − Λ βt + ξt

7 × 1 7 × 1 1 × 1 7 × 1 1 × 1 7 × 1 6 × 1 6 × 1 1 × 1 6 × 1

βt αt βt αt = T

11 1

T

12 1

T21

1

T22

1

βt−1 αt−1 + ⋯ + T

11 𝑞

T

12 𝑞

T21

𝑞

T22

𝑞

βt−𝑞 αt−𝑞 + uβ,t uα,t State Equation Measurement Equation In this example, a dynamic factor model in state-space form

à la Banbura and Modugno (2014) or Camacho and Pérez-Quirós (2010) Charles, Maggi, Palate and De Antonio (2015)

βt αt = T

11 1

T

12 1

T21

1

T22

1

βt−1 αt−1 + ⋯ + T

11 𝑞

T

12 𝑞

T21

𝑞

T22

𝑞

βt−𝑞 αt−𝑞 + uβ,t uα,t

Vector autoregressive process of order 𝑞 Usual identification assumptions

Idiosyncratic terms ξt

is iid ~ N 0, R with diagonal covariance

Idiosyncratic terms ξt

uncorrelated with the factor innovations uβ,t

uα,t

Unrestricted covariance of factor innovations uβ,t uα,t

3) next, specify your model: SUTSE, DFM, BVAR …

In this example, a dynamic factor model in state-space form

à la Banbura and Modugno (2014) or Camacho and Pérez-Quirós (2010)

4) Define evaluation sample and dates at which model parameters must be re-estimated

For univariate models, recursive estimation every month, while multivariate models may be re-estimated once or twice per year, depending on the application

5) Visualize results (prototype in Excel)

(prototype in Excel)

5) Visualize results

Diebold, F.X. and R.S. Mariano (1995) Diebold, F.X. (2013), “Comparing Predictive Accuracy, Twenty Years Later: A Personal Perspective on the Use and Abuse of Diebold-Mariano Tests” Harvey, et al. (1998). “Tests for forecast encompassing”. Hyndman, R. J. and Koehler A. B. (2006). "Another look at measures of forecast accuracy."

6) Quantitative Results

RMSE as a function

• f the “forecast horizon”

6) Quantitative Results

6) Quantitative Results

RMSE as a function

• f the “forecast horizon”

6) Quantitative Results

RMSE as a function

• f the “forecast horizon”

Next Steps

 Improve visualization of results  Better integration with the rest of the JD+ environment  Compare the forecasts of alternative univariate modeling strategies for seasonal adjustment.  Alternative multivariate models: Bayesian VAR, SUTSE

Next Steps

 Improve visualization of results  Better integration with the rest of the JD+ environment  Compare the forecasts of alternative univariate modeling strategies for seasonal adjustment.  Alternative multivariate models: Bayesian VAR, SUTSE  Challenge: link “forecasting performance” to “model el selecti ction

• n strategy

rategy” (dangerous + not feasible in standard evaluation exercises e.g. Diebold, F.X., 2013 )