Linear Models for Regression Henrik I Christensen Robotics & - PowerPoint PPT Presentation

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Linear Models for Regression Henrik I Christensen Robotics & Intelligent Machines @ GT Georgia Institute of Technology, Atlanta, GA 30332-0280 hic@cc.gatech.edu Henrik I Christensen (RIM@GT) Linear Regression 1 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Outline Introduction 1 Preliminaries 2 Linear Basis Function Models 3 Baysian Linear Regression 4 Baysian Model Comparison 5 Summary 6 Henrik I Christensen (RIM@GT) Linear Regression 2 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Introduction The objective of regression is to enable prediction of a value t based on modelling over a dataset X . Consider a set of D observations over a space How can we generate estimates for the future? Battery time? Time to completion? Position of doors? Henrik I Christensen (RIM@GT) Linear Regression 3 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Introduction (2) Example from Chapter 1 1 t 0 −1 0 1 x m y ( x , w ) = w 0 + w 1 x + w 2 x 2 + . . . + w m x m = � w i x i i =0 Henrik I Christensen (RIM@GT) Linear Regression 4 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Introduction (3) In general the functions could be beyond simple polynomials The “components” are termed basis functions , i.e. m w T � � y ( x , w ) = w i φ i ( x ) = � φ ( x ) i =0 Henrik I Christensen (RIM@GT) Linear Regression 5 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Outline Introduction 1 Preliminaries 2 Linear Basis Function Models 3 Baysian Linear Regression 4 Baysian Model Comparison 5 Summary 6 Henrik I Christensen (RIM@GT) Linear Regression 6 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Loss Function For optimization we need a penalty / loss function L ( t , y ( x )) Expected loss is then � � E [ L ] = L ( t , y ( x )) p ( x , t ) dxdt For the squared loss function we have � � { y ( x ) − t } 2 p ( x , t ) dxdt E [ L ] = Goal: choose y(x) to minimize expected loss (E[L]) Henrik I Christensen (RIM@GT) Linear Regression 7 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Loss Function Derivation of the extrema δ E [ L ] � δ y ( x ) = 2 { y ( x ) − t } p ( x , t ) dt = 0 Implies that � tp ( x , t ) dt � y ( x ) = = tp ( t | x ) dt = E [ t | x ] p ( x ) Henrik I Christensen (RIM@GT) Linear Regression 8 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Loss Function - Interpretation t y ( x ) y ( x 0 ) p ( t | x 0 ) x 0 x Henrik I Christensen (RIM@GT) Linear Regression 9 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Alternative Consider a small rewrite { y ( x ) − t } 2 = { y ( x ) − E [ t | x ] + E [ t | x ] − t } 2 The expected loss is then � � { y ( x ) − E [ t | x ] } 2 p ( x ) dx + { E [ t | x ] − t } 2 p ( x ) dx E [ L ] = Henrik I Christensen (RIM@GT) Linear Regression 10 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Outline Introduction 1 Preliminaries 2 Linear Basis Function Models 3 Baysian Linear Regression 4 Baysian Model Comparison 5 Summary 6 Henrik I Christensen (RIM@GT) Linear Regression 11 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Polynomial Basis Functions 1 Basic Definition: 0.5 φ i ( x ) = x i 0 Global functions −0.5 Small change in x affects all of them −1 −1 0 1 Henrik I Christensen (RIM@GT) Linear Regression 12 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Gaussian Basis Functions Basic Definition: 1 φ i ( x ) = e − ( x − µ i )2 2 s 2 0.75 A way to Gaussian mixtures, 0.5 local impact 0.25 Not required to have probabilistic interpretation. 0 µ control position and s −1 0 1 control scale Henrik I Christensen (RIM@GT) Linear Regression 13 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Sigmoid Basis Functions Basic Definition: 1 � x − µ i � φ i ( x ) = σ 0.75 s 0.5 where 1 0.25 σ ( a ) = 1 + e − a 0 −1 0 1 µ controls location and s controls slope Henrik I Christensen (RIM@GT) Linear Regression 14 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Maximum Likelihood & Least Squares Assume observation from a deterministic function contaminated by Gaussian Noise p ( ǫ | β ) = N ( ǫ | 0 , β − 1 ) t = y ( x , w ) + ǫ the problem at hand is then p ( t | x , w , β ) = N ( t | y ( x , w ) , β − 1 ) From a series of observations we have the likelihood N � N ( t i | w T φ ( x i ) , β − 1 ) p ( t | X | w , β ) = i =1 Henrik I Christensen (RIM@GT) Linear Regression 15 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Maximum Likelihood & Least Squares (2) This results in ln p ( t | w , β ) = N 2 ln β − N 2 ln(2 π ) − β E D ( w ) where N E D ( w ) = 1 � { t i − w T φ ( x i ) } 2 2 i =1 is the sum of squared errors Henrik I Christensen (RIM@GT) Linear Regression 16 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Maximum Likelihood & Least Squares (3) Computing the extrema yields: � − 1 � Φ T Φ Φ T t w ML = where   φ 0 ( x 1 ) φ 1 ( x 1 ) · · · φ M − 1 ( x 1 ) φ 0 ( x 1 ) φ 1 ( x 2 ) · · · φ M − 1 ( x 2 )   Φ =  . . .  ... . . .   . . .   φ 0 ( x N ) φ 1 ( x N ) · · · φ M − 1 ( x N ) Henrik I Christensen (RIM@GT) Linear Regression 17 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Line Estimation Least square minimization: Line equation: y = ax + b i ( y i − ax i − b ) 2 Error in fit: � Solution: � ¯ � ¯ � � � � y 2 x 2 ¯ a x = y ¯ x ¯ 1 b Minimizes vertical errors. Non-robust! Henrik I Christensen (RIM@GT) Linear Regression 18 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References LSQ on Lasers Line model: r i cos( φ i − θ ) = ρ Error model: d i = r i cos( φ i − θ ) − ρ i ( r i cos( φ i − θ ) − ρ ) 2 Optimize: argmin ( ρ,θ ) � Error model derived in Deriche et al. (1992) Well suited for “clean-up” of Hough lines Henrik I Christensen (RIM@GT) Linear Regression 19 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Total Least Squares Line equation: ax + by + c = 0 i ( ax i + by i + c ) 2 where a 2 + b 2 = 1. Error in fit: � Solution: � ¯ � � a � a x 2 − ¯ � � x ¯ x xy − ¯ ¯ x ¯ y = µ ¯ y 2 − ¯ xy − ¯ ¯ x ¯ y y ¯ y b b where µ is a scale factor. c = − a ¯ x − b ¯ y Henrik I Christensen (RIM@GT) Linear Regression 20 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Line Representations The line representation is crucial Often a redundant model is adopted Line parameters vs end-points Important for fusion of segments. End-points are less stable Henrik I Christensen (RIM@GT) Linear Regression 21 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Sequential Adaptation In some cases one at a time estimation is more suitable Also known as gradient descent w ( τ ) − η ∇ E n w ( τ +1) = w ( τ ) − η ( t n − w ( τ ) T φ ( x n )) φ ( x n ) = Knows as least-mean square (LMS). An issue is how to choose η ? Henrik I Christensen (RIM@GT) Linear Regression 22 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Regularized Least Squares As seen in lecture 2 sometime control of parameters might be useful. Consider the error function: E D ( w ) + λ E W ( w ) which generates N 1 { t i − w t φ ( x i ) } 2 + λ � 2 w T w 2 i =1 which is minimized by � − 1 � λ I + Φ T Φ Φ T t w = Henrik I Christensen (RIM@GT) Linear Regression 23 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Outline Introduction 1 Preliminaries 2 Linear Basis Function Models 3 Baysian Linear Regression 4 Baysian Model Comparison 5 Summary 6 Henrik I Christensen (RIM@GT) Linear Regression 24 / 39

Introduction Preliminaries Linear Models Bayes Regress Model Comparison Summary References Bayesian Linear Regression Define a conjugate prior over w p ( w ) = N ( w | m 0 , S 0 ) given the likelihood function and regular from Bayesian analysis we can derive p ( w | t ) = N ( w | m N , S N ) where � � S − 1 0 m 0 + β Φ T t m N = S N S − 1 S − 1 + β Φ T Φ = 0 N Henrik I Christensen (RIM@GT) Linear Regression 25 / 39