lasso regularization paths for narmax models via
play

Lasso Regularization Paths for NARMAX Models via Coordinate Descent - PowerPoint PPT Presentation

Mar 23, 2023 •194 likes •482 views

Lasso Regularization Paths for NARMAX Models via Coordinate Descent Ant onio H. Ribeiro, Luis A. Aguirre Universidade Federal de Minas Gerais (UFMG), Brazil American Control Conference, June 29, 2018 Milwaukee, U.S. A. H. Ribeiro, L. A.

  1. Lasso Regularization Paths for NARMAX Models via Coordinate Descent Antˆ onio H. Ribeiro, Luis A. Aguirre Universidade Federal de Minas Gerais (UFMG), Brazil American Control Conference, June 29, 2018 Milwaukee, U.S. A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 1 / 18

  2. Problem Statement Figure: The system identification problem. A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 2 / 18

  3. Prediction Error Methods Framework Cost Function observed � � ���� 2 � � � V ( θ ) = y [ k ] − ˆ y θ [ k ] . � � � �� � k predicted A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 3 / 18

  4. Linear-in-the-Parameters Model Linear-in-the-parameter models: basis functions � �� � � y θ [ k ] = ˆ θ i · x i ( y [ k − 1] , u [ k − 1]) , i Ordinary least-squares formulation: � y θ [ k ] � 2 ⇒ min θ � y − X θ � 2 min � y [ k ] − ˆ 2 θ k A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 4 / 18

  5. L 1 penalty The Lasso θ � y − X θ � 2 min 2 + λ � θ � 1 , Figure: Lasso interpretation (Tibshirani, 1996). Tibshirani, R. (1996). Regression shrinkage and selection via the LASSO. Journal of the Royal Statistical Society. Series B (Methodological) , pages 267–288. A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 5 / 18

  6. Literature Review Solving Lasso Problem Quadratic Programming; Tibshirani, R. (1996). Regression shrinkage and selection via the LASSO. Journal of the Royal Statistical Society. Series B (Methodological) , pages 267–288. A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 6 / 18

  7. Literature Review Solving Lasso Problem Quadratic Programming; LARS (Least Angle Regression) algorithm; Efron, B., Hastie, T., Johnstone, I., and Tibshirani, R. (2004). Least angle regression. The Annals of Statistics , 32(2):407–499. A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 6 / 18

  8. Literature Review Solving Lasso Problem Quadratic Programming; LARS (Least Angle Regression) algorithm; Coordinate Descent ; Friedman, J., Hastie, T., H¨ ofling, H., and Tibshirani, R. (2007). Pathwise coordinate optimization. The Annals of Applied Statistics , 1(2):302–332. Friedman, J., Hastie, T., and Tibshirani, R. (2009). Glmnet: Lasso and elastic-net regularized generalized linear models. R package version , 1(4). Friedman, J., Hastie, T., and Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of statistical software , 33(1):1. A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 6 / 18

  9. Coordinate Descent Algorithm One-at-a-time coordinate optimization: θ j ← arg θ j min � y − X θ � 2 2 + λ � θ � 1 , Figure: Soft threshold operator A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 7 / 18

  10. Coordinate Descent Algorithm One-at-a-time coordinate optimization: r 1 �� � � �� � � T x j ; λ θ j ← � x j � 2 S ( y − X θ ) + x j θ j , Figure: Soft threshold operator A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 7 / 18

  11. Coordinate Descent Algorithm Optimization Problem θ � y − X θ � 2 min 2 + λ � θ � 1 , Repeat: � � 1 1 θ j ← ( r + x j θ j ) T x j ; λ � x j � 2 S 2 Update r = ( y − X θ ) 3 Next j . A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 8 / 18

  12. Coordinate Descent Algorithm Optimization Problem θ � y − X θ � 2 min 2 + λ � θ � 1 , Repeat: � � 1 1 θ j ← ( r + x j θ j ) T x j ; λ → O ( N ) � x j � 2 S 2 Update r = ( y − X θ ) → O ( N ) 3 Next j . A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 8 / 18

  13. NARMAX model Assuming that: r [ k ] = y [ k ] − ˆ y θ [ k ] p � ˆ y θ [ k ] = θ i · x i ( y [ k − 1] , u [ k − 1] , r [ k − 1] ) . � �� � � �� � i =1 measured values noise term Estimated parameter: ˆ θ = arg θ min � y − X ( y , u , r ) θ � 2 2 . A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 9 / 18

  14. Extended Least Squares Optimization Problem θ � y − X ( y , u , r ) θ � 2 min 2 , Repeat: � � ( i +1) ← arg θ min 2 ˆ � � � y − X ( y , u , r ( i ) ) θ θ 1 � r ( i +1) ← y − X ( y , u , r ( i ) ) θ ( i +1) 2 ˆ 3 i ← i + 1. A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 10 / 18

  15. Coordinate Descent Algorithm (Revisited) Optimization Problem θ � y − X ( y , u , r ) θ � 2 min 2 + λ � θ � 1 , Repeat: Update x j if it depends on r 1 � � 2 θ + 1 ( r + x j θ j ) T x j ; λ j ← � x j � 2 S 3 Update r = ( y − X θ ) 4 Next j . A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 11 / 18

  16. Coordinate Descent Algorithm (Revisited) Optimization Problem θ � y − X ( y , u , r ) θ � 2 min 2 + λ � θ � 1 , Repeat: Update x j if it depends on r → O ( N ) 1 � � 2 θ + 1 ( r + x j θ j ) T x j ; λ j ← � x j � 2 S → O ( N ) 3 Update r = ( y − X θ ) → O ( N ) 4 Next j . A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 11 / 18

  17. Example I The dataset was generated from the linear system: y [ k ] = 0 . 5 y [ k − 1] − 0 . 5 u [ k − 1] + 0 . 5 v [ k − 1] + v [ k ] . A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 12 / 18

  18. Example I The dataset was generated from the linear system: y [ k ] = 0 . 5 y [ k − 1] − 0 . 5 u [ k − 1] + 0 . 5 v [ k − 1] + v [ k ] . We try to fit the following linear model to the training data (30 regressors): 10 10 10 � � � y [ k ] = θ i y [ k − i ] + θ ( i +10) u [ k − i ] + θ ( i +20) r [ k − i ] . i =1 i =1 i =1 A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 12 / 18

  19. Example I 0.8 y [ k − 1] 0.4 v [ k − 1] 0.0 θ -0.4 u [ k − 1] -0.8 10 − 1 10 − 2 10 − 3 10 − 4 λ Figure: Estimated parameter vector θ as a function of λ . Estimated system: y [ k ] = 0 . 48 y [ k − 1] − 0 . 50 u [ k − 1] + 0 . 44 v [ k − 1]. A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 13 / 18

  20. Example II The dataset was generated from the nonlinear system (Chen, et. al., 1990): (0 . 8 − 0 . 5exp( − y [ k − 1] 2 ) y [ k − 1] + u [ k − 1] − y [ k ] = (0 . 3 + 0 . 9exp( − y [ k − 1] 2 ) y [ k − 2] + 0 . 2 u [ k − 2] + 0 . 1 u [ k − 1] u [ k − 2] + 0 . 1 v [ k − 1] + 0 . 3 v [ k − 2] + v [ k ] , And, we fit a polynomial model with degree 2 and 44 regressors to it. S. Chen, S. A. Billings, and P. M. Grant (1990). Non-linear system identification using neural networks International Journal of Control , vol. 51, no. 6, pp. 1191–1214, 1990. A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 14 / 18

  21. Example II 1.2 0.8 0.4 0.0 θ -0.4 -0.8 -1.2 10 0 10 − 1 10 − 2 10 − 3 10 − 4 λ Figure: Estimated parameter vector θ as a function of λ . For this optimal λ the mean absolute error in the validation set is 1.03 and the model includes the regressors y [ k − 1], u [ k − 1], y [ k − 3], y [ k − 2], u [ k − 2], r [ k − 1], r [ k − 2], y [ k − 1] y [ k − 2], u [ k − 1] u [ k − 2], y [ k − 3] r [ k − 1], y [ k − 2] u [ k − 2]. A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 15 / 18

  22. Related Work H. Wang, G. Li, and C.-L. Tsai (2007). Regression Coefficient and Autoregressive Order Shrinkage and Selection Via the Lasso. Journal of the Royal Statistical Society. Series B (Statistical Methodology) , vol. 69, no. 1, pp. 63–78, 2007. Y. J. Yoon, C. Park, and T. Lee (2013). Penalized regression models with autoregressive error terms. Journal of Statistical Computation and Simulation , vol. 83, no. 9, pp. 1756–1772, Sep. 2013. A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 16 / 18

  23. Conclusion 1 Timmings; A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 17 / 18

  24. Conclusion 1 Timmings; 2 Convergence; A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 17 / 18

  25. Conclusion 1 Timmings; 2 Convergence; 3 Scaling; A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 17 / 18

  26. Conclusion 1 Timmings; 2 Convergence; 3 Scaling; 4 Elastic net ; A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 17 / 18

  27. Acknoledgments The implementation is available at: https://github.com/antonior92/NarmaxLasso.jl A. H. Ribeiro, L. A. Aguirre (UFMG) Lasso Regularization Paths for NARMAX ACC 2018 18 / 18

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Why LASSO, EN, and General Regularization CLOT: Invariance-Based Scale-Invariance: . . .

Why LASSO, EN, and General Regularization CLOT: Invariance-Based Scale-Invariance: . . .

Need for Regularization Currently Used . . . Why: Remaining . . . Probabilistic . . . Why LASSO, EN, and General Regularization CLOT: Invariance-Based Scale-Invariance: . . . Shift-Invariance: . . . Explanation Why LASSO Beyond EN and

601 views • 59 slides
Complexity Analysis of the Lasso Regularization Path Julien Mairal and Bin Yu Inria, UC Berkeley

Complexity Analysis of the Lasso Regularization Path Julien Mairal and Bin Yu Inria, UC Berkeley

Complexity Analysis of the Lasso Regularization Path Julien Mairal and Bin Yu Inria, UC Berkeley San Diego, SIAM Optimization, May 2014 Julien Mairal, Inria Complexity Analysis of the Lasso Regularization Path 1/15 What this work is about

802 views • 22 slides
Why LASSO, Ridge Need for Strictly . . . Regression, and EN: General Analysis of the . . . Why

Why LASSO, Ridge Need for Strictly . . . Regression, and EN: General Analysis of the . . . Why

Need for Regularization Which Regularizations . . . Need for Degrees of . . . Need for And- and . . . Why LASSO, Ridge Need for Strictly . . . Regression, and EN: General Analysis of the . . . Why LASSO Explanation Based on Soft Why

779 views • 34 slides
RIDGE and LASSO regularization for regression Feature selection - Some algorithms perform

RIDGE and LASSO regularization for regression Feature selection - Some algorithms perform

RIDGE and LASSO regularization for regression Feature selection - Some algorithms perform naturally feature selection - for example Decision Trees, Boosting - Other algorithms have difficulty with correlated features - for example Naive Bayes,

801 views • 14 slides
Regularization: Ridge Regression and the LASSO Statistics 305: Autumn Quarter 2006/2007

Regularization: Ridge Regression and the LASSO Statistics 305: Autumn Quarter 2006/2007

Agenda Regularization: Ridge Regression and the LASSO Statistics 305: Autumn Quarter 2006/2007 Wednesday, November 29, 2006 Statistics 305: Autumn Quarter 2006/2007 Regularization: Ridge Regression and the LASSO Agenda Agenda 1 The

910 views • 56 slides
Using the lasso in Stata for inference in high-dimensional models David M. Drukker Executive

Using the lasso in Stata for inference in high-dimensional models David M. Drukker Executive

Using the lasso in Stata for inference in high-dimensional models David M. Drukker Executive Director of Econometrics Stata Spanish Stata User Group Meeting 17 Octubre 2019 Outline What are high-dimensional models? 1 What is the lasso? 2

1.04k views • 49 slides
Using the lasso in Stata for inference in high-dimensional models David M. Drukker Executive

Using the lasso in Stata for inference in high-dimensional models David M. Drukker Executive

Using the lasso in Stata for inference in high-dimensional models David M. Drukker Executive Director of Econometrics Stata London Stata Conference 5-6 September 2019 Outline What are high-dimensional models? 1 What is the lasso? 2 Using

895 views • 41 slides
Regularization Paths Boosting fits a regularization path toward a max-margin classifier.

Regularization Paths Boosting fits a regularization path toward a max-margin classifier.

June 2006 Trevor Hastie, Stanford Statistics 1 June 2006 Trevor Hastie, Stanford Statistics 2 Theme Regularization Paths Boosting fits a regularization path toward a max-margin classifier. Svmpath does as well. Trevor Hastie In

112 views • 9 slides
On Model Selection Consistency Of Lasso Yewon Kim 12/08/2015 Introduction Model selection is a

On Model Selection Consistency Of Lasso Yewon Kim 12/08/2015 Introduction Model selection is a

On Model Selection Consistency Of Lasso Yewon Kim 12/08/2015 Introduction Model selection is a commonly used method to find sparsity or parsimony of statistical models, but usually involves a computationally heavy combinatorial search. Lasso

454 views • 19 slides
Regularization Overview Regularization Overview Problems & Multicollinearity We will

Regularization Overview Regularization Overview Problems & Multicollinearity We will

Regularization Overview Regularization Overview Problems & Multicollinearity We will discuss three popular methods for obtaining better estimates of the linear model coefficients Regularization Techniques Principal

305 views • 12 slides
Discriminative Regularization for Latent Variable Models with Applications to Electrocardiography

Discriminative Regularization for Latent Variable Models with Applications to Electrocardiography

Discriminative Regularization for Latent Variable Models with Applications to Electrocardiography Poster #53 Andrew C. Miller with Ziad Obermeyer, John P . Cunningham, and Sendhil Mullainathan <latexit

674 views • 9 slides
A practical tour of optimization algorithms for the Lasso Alexandre Gramfort

A practical tour of optimization algorithms for the Lasso Alexandre Gramfort

A practical tour of optimization algorithms for the Lasso Alexandre Gramfort alexandre.gramfort@inria.fr Inria, Parietal Team Universit Paris-Saclay Huawei - Apr. 2017 Outline What is the Lasso Lasso with an orthogonal design

560 views • 29 slides
MLCC 2017 Regularization Networks I: Linear Models Lorenzo Rosasco UNIGE-MIT-IIT June 27, 2017

MLCC 2017 Regularization Networks I: Linear Models Lorenzo Rosasco UNIGE-MIT-IIT June 27, 2017

MLCC 2017 Regularization Networks I: Linear Models Lorenzo Rosasco UNIGE-MIT-IIT June 27, 2017 About this class We introduce a class of learning algorithms based on Tikhonov regularization We study computational aspects of these

743 views • 46 slides
Area under lattice paths associated with certain urn models. Alois Panholzer, Markus Kuba

Area under lattice paths associated with certain urn models. Alois Panholzer, Markus Kuba

P olya-Eggenberger urn models Diminishing urn models: Area Analysis Further discussion Area under lattice paths associated with certain urn models. Alois Panholzer, Markus Kuba Institute of Discrete Mathematics and Geometry Vienna

766 views • 76 slides
Why Geometric Progression LASSO Method in Selecting the LASSO How Is Selected: . . . Natural

Why Geometric Progression LASSO Method in Selecting the LASSO How Is Selected: . . . Natural

Need for Regression Need for Linear Regression The Least Squares . . . Need to Go Beyond . . . Why Geometric Progression LASSO Method in Selecting the LASSO How Is Selected: . . . Natural Uniqueness . . . Parameter: A Theoretical

610 views • 31 slides
Regularization Regularization is a general approach to add a complexity parameter to a

Regularization Regularization is a general approach to add a complexity parameter to a

Regularization Regularization is a general approach to add a complexity parameter to a learning algorithm. Requires that the model parameters be continuous. (i.e., Regression OK, IAML: Regularization and Ridge Regression Decision trees

204 views • 3 slides
Econometric Evaluation of Social Programs Part I: Identification James J. Heckman and Edward J.

Econometric Evaluation of Social Programs Part I: Identification James J. Heckman and Edward J.

Problems Econometric Evaluation of Social Programs Part I: Identification James J. Heckman and Edward J. Vytlacil Econ 312, Spring 2019 Heckman and Vytlacil Identification Problems Problems Identification Problems: Determining Models From

386 views • 7 slides
Enough? Interactive Selection of Bonds between Pairs of Tangible Molecules Patrick Maier, Marcus

Enough? Interactive Selection of Bonds between Pairs of Tangible Molecules Patrick Maier, Marcus

What Do You Do When Two Hands Are Not Enough? Interactive Selection of Bonds between Pairs of Tangible Molecules Patrick Maier, Marcus Tnnis, Gudrun Klinker / Alexander Raith, Markus Drees, Fritz Khn Fachgebiet Augmented Reality / Fachgebiet

654 views • 17 slides
Truncated Moreaus sweeping process Florent Nacry - Institut Elie Cartan de Lorraine joint works

Truncated Moreaus sweeping process Florent Nacry - Institut Elie Cartan de Lorraine joint works

Truncated Moreaus sweeping process Florent Nacry - Institut Elie Cartan de Lorraine joint works with Lionel Thibault - Institut Montpellirain Alexander Grothendieck Journes annuelles du GdR MOA, Universit de Pau et des Pays de lAdour,

527 views • 52 slides
Modularity (1): Childhood Activity Modularity Abstract Data Types (ADTs) EECS3311 A & E:

Modularity (1): Childhood Activity Modularity Abstract Data Types (ADTs) EECS3311 A & E:

Modularity (1): Childhood Activity Modularity Abstract Data Types (ADTs) EECS3311 A & E: Software Design Fall 2020 C HEN -W EI W ANG (I NTERFACE ) S PECIFICATION (A SSEMBLY ) A RCHITECTURE Sources: https://commons.wikimedia.org and

417 views • 5 slides
Numerical Analysis of initial data identification of parabolic problems Dmitriy Leykekhman

Numerical Analysis of initial data identification of parabolic problems Dmitriy Leykekhman

Numerical Analysis of initial data identification of parabolic problems Dmitriy Leykekhman University of Connecticut Joint work with Boris Vexler (TU M unchen) and Daniel Walter, (RICAM) New trends in PDE constrained optimization RICAM

1.1k views • 80 slides
De-anonymizing D4D Datasets Kumar Sharad 1 George Danezis 2 1 University of Cambridge 2 Microsoft

De-anonymizing D4D Datasets Kumar Sharad 1 George Danezis 2 1 University of Cambridge 2 Microsoft

De-anonymizing D4D Datasets Kumar Sharad 1 George Danezis 2 1 University of Cambridge 2 Microsoft Research July 12, 2013 6th Workshop on Hot Topics in Privacy Enhancing Technologies 2013, Bloomington, Indiana, USA Can Personally Identifiable

668 views • 18 slides
The CAPCM (Welch, Chapter 10-A) Ivo Welch Maintained Assumptions Perfect Markets 1. No

The CAPCM (Welch, Chapter 10-A) Ivo Welch Maintained Assumptions Perfect Markets 1. No

The CAPCM (Welch, Chapter 10-A) Ivo Welch Maintained Assumptions Perfect Markets 1. No differences in opinion. 2. No taxes. 3. No transaction costs. 4. No big sellers/buyersinfinitely many clones that can buy or sell. With risk and

691 views • 57 slides
Risk management for hedge funds AQF 2005 Nicolas Papageorgiou Outline VaR and drawbacks

Risk management for hedge funds AQF 2005 Nicolas Papageorgiou Outline VaR and drawbacks

Risk management for hedge funds AQF 2005 Nicolas Papageorgiou Outline VaR and drawbacks Nonlinearities Credit risk and liquidity Can risk management be a source of alpha Consider fund with E(r)= 10% and volatility=70%

376 views • 14 slides

More recommend