SLIDE 1
Modeling and Control of Dynamic Systems Validation
Darya Krushevskaya Konstantin Tretyakov
SLIDE 2 Introduction
Model evaluation
In accordance with intended use of the model Investigate particular
Experiment Experiment Select model structure Select model structure
Investigate particular property
structure structure Estimate model Estimate model Validate model Accepted Not accepted
SLIDE 3
Data
Test or validation set
Not used during training
Cross-validation
Partitioning of the data into subsets Partitioning of the data into subsets
SLIDE 4 Validation
- 1. Evaluation of the residuals
- Tests for correlation
- 2. Estimation of the average generalization
error error
- 3. Visualization of the model’s ability to predict
- Graphical comparison
SLIDE 5
Tests for Correlations I
Residuals should be uncorrelated with all linear and nonliniar combinations of past data
Complete test is unrealistic Consider only few tests Consider only few tests
SLIDE 6
Correlation Tests
SLIDE 7 Tests for Correlations II
Calculate correlation functions (τ) If the data are indeed uncorrelated, the values (τ) are asymptotically normal with distribution :
) 1 , (
This suggests a simple statistical test (|(τ)| < 1.96/N) for
) 1 , (
20 , 20 [− ∈ τ
SLIDE 8
NNARX demo
SLIDE 9
NNARX demo
SLIDE 10
NNARX demo
SLIDE 11
NNARX demo
SLIDE 12
NNARX demo
SLIDE 13
NNARX demo
SLIDE 14
NNARX demo
SLIDE 15
NNARX demo
SLIDE 16
Estimation of the average generalization error
SLIDE 17
Visualization of the Predictions
Shows variation in accuracy of the prediction Can show overfitting and possible systematic errors
SLIDE 18
Visualization of the Predictions
Underparametrized model
SLIDE 19
Visualization of the Predictions
Overparametrized model
SLIDE 20 Prediction intervals
Estimating reliability of predictions for a specific input
- Variance of the prediction error of regression
M S ∈
Variance of the prediction error of regression vector φ(t):
SLIDE 21
NNATX model evaluation
A 95% confidence interval is drawn
SLIDE 22
K-step ahead predictions
In case of fast sampling Check that ŷ(t|)=y(t1) K-step ahead prediction
) 1 ( ) ( − ≈ t y t y
K-step ahead prediction
SLIDE 23
K-step prediction demo
SLIDE 24
Summary
Model validation
Correlation functions Estimation average generalization error Visualization of predictions Visualization of predictions
SLIDE 25 Variance
M S ∈
The covariance matrix:
SLIDE 26
The Noise variance
The noise variance:
