Least Squares Support Vector Regression with Applications to - PowerPoint PPT Presentation

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Least Squares Support Vector Regression with Applications to Large-Scale Data: a Statistical Approach Kris De Brabanter Public Defense April, 27 2011 Promotor: Prof. dr. ir. B. De Moor Co-Promotor: Prof. dr. ir. J. Suykens 1/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Outline Goal & Overview 1 Introduction 2 Parametric vs. nonparametric regression Nonparametric regression estimates: an overview Fixed-Size Least Squares Support Vector Machines 3 Fixed Size LS-SVM formulation Selection of Support Vectors Practical identification problem Robust Nonparametric Methods 4 Problems with outliers Robust nonparametric regression Correlated Errors 5 Problems with correlation in nonparametric regression Removing correlation effects Confidence Intervals 6 Conclusions 7 2/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Outline Goal & Overview 1 Introduction 2 Parametric vs. nonparametric regression Nonparametric regression estimates: an overview Fixed-Size Least Squares Support Vector Machines 3 Fixed Size LS-SVM formulation Selection of Support Vectors Practical identification problem Robust Nonparametric Methods 4 Problems with outliers Robust nonparametric regression Correlated Errors 5 Problems with correlation in nonparametric regression Removing correlation effects Confidence Intervals 6 Conclusions 7 3/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Goal of the Thesis Goal of the Thesis Study the properties of Least Squares Support Vector Machines for regression with an emphasis on statistical aspects and develop a framework for large scale data 4/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Overview Model Selection Large Scale (Chapter 3) Data Sets (Chapter 4) Model Building (Chapter 2) Robustness (Chapter 5) Least Squares Introduction Support Vector (Chapter 1) Machines Correlated Errors (Chapter 6) Conclusions & Further Research Confidence (Chapter 9) Intervals Applications & (Chapter 7) Case Studies (Chapter 8) 5/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Outline Goal & Overview 1 Introduction 2 Parametric vs. nonparametric regression Nonparametric regression estimates: an overview Fixed-Size Least Squares Support Vector Machines 3 Fixed Size LS-SVM formulation Selection of Support Vectors Practical identification problem Robust Nonparametric Methods 4 Problems with outliers Robust nonparametric regression Correlated Errors 5 Problems with correlation in nonparametric regression Removing correlation effects Confidence Intervals 6 Conclusions 7 6/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions A simple example 30 20 10 0 −10 −20 −30 −5 0 5 7/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions A simple example 30 20 10 0 −10 −20 −30 −5 0 5 7/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions A simple example 30 20 Y = aX + b 10 0 −10 −20 −30 −5 0 5 7/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions A simple example 30 100 20 50 Y = aX + b 10 0 0 −50 −10 −100 −20 −30 −150 −5 0 5 0 10 20 30 40 50 60 7/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions A simple example 30 100 20 50 Y = aX + b 10 0 0 −50 −10 −100 −20 −30 −150 −5 0 5 0 10 20 30 40 50 60 7/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions A simple example 30 100 20 50 Y = ? Y = aX + b 10 0 0 −50 −10 −100 −20 −30 −150 −5 0 5 0 10 20 30 40 50 60 7/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions A simple example 30 100 20 50 Y = ? Y = aX + b 10 0 0 −50 −10 −100 −20 −30 −150 −5 0 5 0 10 20 30 40 50 60 PARAMETRIC FORM IS NOT ALWAYS EASY TO FIND 7/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Construction of a nonparametric estimate: NW smoother 1.5 1 0.5 0 −0.5 −1 0 0.2 0.4 0.6 0.8 1 8/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Construction of a nonparametric estimate: NW smoother 1.5 1 0.5 0 −0.5 −1 0 0.2 0.4 0.6 0.8 1 8/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Construction of a nonparametric estimate: NW smoother 1.5 1 0.5 0 −0.5 −1 0 0.2 0.4 0.6 0.8 1 x 8/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Construction of a nonparametric estimate: NW smoother 1.5 1 0.5 0 −0.5 −1 0 0.2 0.4 0.6 0.8 1 x 8/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Construction of a nonparametric estimate: NW smoother 1.5 m n ( x ) ˆ 1 0.5 0 −0.5 −1 0 0.2 0.4 0.6 0.8 1 x 8/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Construction of a nonparametric estimate: NW smoother 1.5 m n ( x ) ˆ 1 0.5 0 −0.5 −1 0 0.2 0.4 0.6 0.8 1 x n K ( x − X i ) � h m n ( x ) = ˆ Y i j =1 K ( x − X j � n ) i =1 h 8/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Other nonparametric regression estimates Local constant regression (Nadaraya, 1964; Watson, 1964) Regression trees (Breiman et al. , 1984) Wavelets (Daubechies, 1992) Nearest Neighbors (Devroye et al. , 1994) Local linear regression (Fan & Gijbels, 1996) Support vector machines (Vapnik, 1995) Splines (Wahba, 1990; Eubank, 1999) Partitioning estimates (Gy¨ orfi et al. , 2002) Least squares support vector machines (Suykens et al. , 2002) . . . 9/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Least squares support vector machines Primal formulation (LS-SVM formulation for regression) 2 w T w + γ w,b,e J P ( w, e ) = 1 � n k =1 e 2 min k 2 w T ϕ ( X k ) + b + e k = Y k , k = 1 , . . . , n. s.t. 10/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Least squares support vector machines Primal formulation (LS-SVM formulation for regression) 2 w T w + γ � n w,b,e J P ( w, e ) = 1 k =1 e 2 min 2 k w T ϕ ( X k ) + b + e k = Y k , k = 1 , . . . , n. s.t. input space ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ 10/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Least squares support vector machines Primal formulation (LS-SVM formulation for regression) 2 w T w + γ � n w,b,e J P ( w, e ) = 1 k =1 e 2 min 2 k w T ϕ ( X k ) + b + e k = Y k , k = 1 , . . . , n. s.t. input space ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ 10/39

Goal & Overview Introduction FS-LSSVM Robust Nonparametric Methods Correlated Errors Confidence Intervals Conclusions Least squares support vector machines Primal formulation (LS-SVM formulation for regression) 2 w T w + γ � n w,b,e J P ( w, e ) = 1 k =1 e 2 min 2 k w T ϕ ( X k ) + b + e k = Y k , k = 1 , . . . , n. s.t. input space feature space ⋆ ϕ ( · ) ⋆ ϕ ( ) ⋆ ⋆ ϕ ( ) ⋆ ⋆ ⋆ ϕ ( ) ϕ ( ) ⋆ ϕ ( ) ⋆ ⋆ ϕ ( ) ⋆ ⋆ ϕ ( ) ⋆ ϕ ( ) ⋆ ϕ ( ) ⋆ ϕ ( ) ⋆ ⋆ ⋆ ⋆ ϕ ( ) ϕ ( ) ⋆ ⋆ ⋆ ϕ ( ) ⋆ ϕ ( ) ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ 10/39