Calibrating Agent-Based Models with Machine Learning Surrogates - - PowerPoint PPT Presentation

▶
calibrating agent based models with machine learning
SMART_READER_LITE
LIVE PREVIEW

Calibrating Agent-Based Models with Machine Learning Surrogates - - PowerPoint PPT Presentation

Apr 17, 2023 24 likes •615 views

Calibrating Agent-Based Models with Machine Learning Surrogates Francesco LAMPERTI 1 , 2 , Antoine MANDEL 2 , Andrea ROVENTINI 1 , Amir SANI 2 1 Institute of Economics and LEM, Scuola Superiore SantAnna (Pisa) 2 Universit e Paris 1 Path

slide-1
SLIDE 1

Calibrating Agent-Based Models with Machine Learning Surrogates

Francesco LAMPERTI 1,2, Antoine MANDEL 2, Andrea ROVENTINI 1, Amir SANI 2

1Institute of Economics and LEM, Scuola Superiore Sant’Anna (Pisa) 2Universit´

e Paris 1 Path´ eon-Sorbonne, Centre d’Economie de la Sorbonne and CNRS, Paris School of Economics

Macroeconomic Agent-Based Modeling Surrogates Workshop

February 9th, 2016

Research supported by Horizons 2020 FET, DOLFINS project.

slide-2
SLIDE 2

What is the Problem? What is our Approach? Example Evaluation Times Empirical Results Empirical Results

slide-3
SLIDE 3

What is the Problem?

slide-4
SLIDE 4

(many) Macroeconomic Agent-Based Models (ABMs) are

◮ computationally expensive ◮ defined by dozens of parameters ◮ hard to calibrate, estimate, test, explore ∗

∗Due to complex parameter-specific behaviours.

slide-5
SLIDE 5

What are we proposing?

slide-6
SLIDE 6

Machine Learning Surrogates!

slide-7
SLIDE 7

Faster

◮ Agent-Based Model “Calibration” (e.g. Inference) ◮ Scenario Stress Testing (e.g. Monte Carlo) ◮ Policy Exercises (e.g. Exploration)

slide-8
SLIDE 8

Faster

◮ Agent-Based Model “Calibration” (e.g. Inference) ◮ Scenario Stress Testing (e.g. Monte Carlo) ◮ Policy Exercises (e.g. Exploration)

slide-9
SLIDE 9

Faster

◮ Agent-Based Model “Calibration” (e.g. Inference) ◮ Scenario Stress Testing (e.g. Monte Carlo) ◮ Policy Exercises (e.g. Exploration)

slide-10
SLIDE 10

Surrogate(ABM + “Calibration” Qality Test)

slide-11
SLIDE 11

Surrogate

  • Brock & Hommes ABM + Kolmogorov–Smirnov

Two Sample Test

slide-12
SLIDE 12

Brock & Hommes (BH) Very simple ABM with 10 parameters

slide-13
SLIDE 13

Brock & Hommes This model is NOT representative of complex Macroeconomic Agent-Based Models

slide-14
SLIDE 14

Brock & Hommes Smooth Learning Manifold

slide-15
SLIDE 15

but

slide-16
SLIDE 16

13.4 quadrillion combinations

slide-17
SLIDE 17

Tiny positive region

slide-18
SLIDE 18

Enormous parameter space!

slide-19
SLIDE 19
slide-20
SLIDE 20

The Curse of Dimensionality!

slide-21
SLIDE 21

How do we measure calibration quality?

slide-22
SLIDE 22

Kolmogorov–Smirnov Two Sample Test (KS)

slide-23
SLIDE 23

Dn,n′ = sup

x

  • FXSP500,n(x) − FXBH,n′(x)
slide-24
SLIDE 24
slide-25
SLIDE 25

Evaluation Times

slide-26
SLIDE 26

Single Evaluation BH + KS = 0.30seconds

slide-27
SLIDE 27

Policy Constraint

Security prices are not negative

slide-28
SLIDE 28

Policy Constraint

Only run parameters yi that pass the following constraint (filter), S(T1 ∗ P + B1) + (1 − S) ∗ (T2 ∗ P + B2) > 0, where S is the share type, P is the initial deviation from the fundamental price, B1 and B2 are the bias of agent 1 and 2, and T1 and T2 are the trend of agent 1 and 2.

slide-29
SLIDE 29

Simulation X BH(yi) = X BH

slide-30
SLIDE 30

Simulation Objective yi that produce 250 Log Returns

slide-31
SLIDE 31

An obvious Constraint

slide-32
SLIDE 32

Length Filter ABM simulations should produce 250 Log Returns

slide-33
SLIDE 33

Length Filter len(X BH) == 250

slide-34
SLIDE 34

Length Filter (Density) 3,000 per 100,000†

†Random Latin Hypercube Sampling[4]

slide-35
SLIDE 35

2-Sample KS Test KS(X SP500, X BH) = {DX SP500,X BH, p-value}

slide-36
SLIDE 36

KS Threshold p-value >0.05 DXSP500,XBH <0.20

slide-37
SLIDE 37

KS Threshold (Density) 55 per 1,000,000‡

‡Random Latin Hypercube Sampling[4]

slide-38
SLIDE 38

Dataset Highly Imbalanced!

slide-39
SLIDE 39

Evaluation Time

slide-40
SLIDE 40

Evaluation Time 1,000,000 samples ≈ 3.5 days

slide-41
SLIDE 41

Evaluation Time 100 passing tests ≈ 20.4 days

slide-42
SLIDE 42

What about a Machine Learning Surrogate?

slide-43
SLIDE 43

Surrogate Modeling

◮ Function Approximation [3] ◮ Meta-Modeling [1] ◮ “Response Surface” Methodology [2, 5] ◮ Experimental Design ◮ Model Emulation ◮ “Model of a Model”

slide-44
SLIDE 44

Surrogate Solution

  • Draw 1, 000, 000 parameters using RLH
  • Policy Constraint ≈ 500, 000
  • Time to compute yPC

i

≈ 0.25 secs

  • BH(yPC

i

) = X PC

i

: 2, 500 min

  • Length Constraint ≈ 3, 000 yPC,len

i

: 5 min

  • KS(X SP500, X PC,len

i

) = {DXSP500,XPC,len

i

, p-valuePC,len

i

}: 800 min

  • Threshold Constraint ≈ 55 yPC,len,Thresholded

i

: 1 min

slide-45
SLIDE 45

Surrogate Solution

  • Draw 1, 000, 000 parameters using RLH
  • Policy Constraint ≈ 500, 000
  • Time to compute yPC

i

≈ 0.25 secs

  • BH(yPC

i

) = X PC

i

: 2, 500 min

  • Length Constraint ≈ 3, 000 yPC,len

i

: 5 min

  • KS(X SP500, X PC,len

i

) = {DXSP500,XPC,len

i

, p-valuePC,len

i

}: 800 min

  • Threshold Constraint ≈ 55 yPC,len,Thresholded

i

: 1 min

slide-46
SLIDE 46

Surrogate Solution

  • Draw 1, 000, 000 parameters using RLH
  • Policy Constraint ≈ 500, 000
  • Time to compute yPC

i

≈ 0.25 secs

  • BH(yPC

i

) = X PC

i

: 2, 500 min

  • Length Constraint ≈ 3, 000 yPC,len

i

: 5 min

  • KS(X SP500, X PC,len

i

) = {DXSP500,XPC,len

i

, p-valuePC,len

i

}: 800 min

  • Threshold Constraint ≈ 55 yPC,len,Thresholded

i

: 1 min

slide-47
SLIDE 47

Surrogate Solution

  • Draw 1, 000, 000 parameters using RLH
  • Policy Constraint ≈ 500, 000
  • Time to compute yPC

i

≈ 0.25 secs

  • BH(yPC

i

) = X PC

i

: 2, 500 min

  • Length Constraint ≈ 3, 000 yPC,len

i

: 5 min

  • KS(X SP500, X PC,len

i

) = {DXSP500,XPC,len

i

, p-valuePC,len

i

}: 800 min

  • Threshold Constraint ≈ 55 yPC,len,Thresholded

i

: 1 min

slide-48
SLIDE 48

Surrogate Solution

  • Draw 1, 000, 000 parameters using RLH
  • Policy Constraint ≈ 500, 000
  • Time to compute yPC

i

≈ 0.25 secs

  • BH(yPC

i

) = X PC

i

: 2, 500 min

  • Length Constraint ≈ 3, 000 yPC,len

i

: 5 min

  • KS(X SP500, X PC,len

i

) = {DXSP500,XPC,len

i

, p-valuePC,len

i

}: 4, 000 min

  • Threshold Constraint ≈ 55 yPC,len,Thresholded

i

: 1 min

slide-49
SLIDE 49

Total Time BH+KS: 6, 5081

4min

slide-50
SLIDE 50

Out of Sample 100, 000, 000 out of sample yi§ ≈ 2 min

§using RLH

slide-51
SLIDE 51

BH+KS 1, 000 × 6, 5081

4min = 6, 508, 250 min

slide-52
SLIDE 52

Machine Learning Surrogate

  • (naive) Budgeted Model Search¶: 60 min

¶htps://github.com/hyperopt/hyperopt-sklearn

slide-53
SLIDE 53

Machine Learning Surrogate Filter OOS using Learned Model ≈ 12 min

slide-54
SLIDE 54

Speedup (OOS Only) BH+KS: 1, 000 × 6, 5081

4min = 6, 508, 250 min

Machine Learning Surrogate: 72 min ≈ 90, 3921

3× Speedup!

slide-55
SLIDE 55

Advantage Reusable Machine Learning Surrogate Model

slide-56
SLIDE 56
slide-57
SLIDE 57
slide-58
SLIDE 58

Thank you!

slide-59
SLIDE 59

References

[1] Robert W Blanning. The construction and implementation of metamodels. simulation, 24(6):177–184, 1975. [2] George EP Box and KB Wilson. On the experimental atainment of optimum

  • conditions. Journal of the Royal Statistical Society. Series B (Methodological),

13(1):1–45, 1951. [3] Donald R Jones. A taxonomy of global optimization methods based on response surfaces. Journal of global optimization, 21(4):345–383, 2001. [4] Michael D McKay, Richard J Beckman, and William J Conover. Comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics, 21(2):239–245, 1979. [5] Raymond H Myers, Douglas C Montgomery, and Christine M Anderson-Cook. Response surface methodology: process and product optimization using designed experiments, volume 705. John Wiley & Sons, 2009. [6] The Art of Sofware. Derivation of Bias-Variance Decomposition, September

  • 2012. http://artofsoftware.org/2012/09/13/

derivation-of-bias-variance-decomposition/.