SLIDE 1
Linear Regression
- Prof. Andreas Krause
Basic Supervised Learning Pipeline
2
Training Data “spam” “ham” “spam”
Learning method
Model
Predic- tion
Introduction to Machine Learning Linear Regression Prof. Andreas - - PowerPoint PPT Presentation
Introduction to Machine Learning Linear Regression Prof. Andreas - - PowerPoint PPT Presentation
Introduction to Machine Learning Linear Regression Prof. Andreas Krause Learning and Adaptive Systems (las.ethz.ch) Basic Supervised Learning Pipeline Training Data Test Data spam ? Learning Predic- ham Model method ?
SLIDE 2
SLIDE 3
Regression
Instance of supervised learning Goal: Predict real valued labels (possibly vectors) Examples:
3
X Y Flight route Delay (minutes) Real estate objects Price Customer & ad features Click-through probability
SLIDE 4
Running example: Diabetes
[Efron et al ‘04]
Features X:
Age Sex Body mass index Average blood pressure Six blood serum measurements (S1-S6)
Label (target) Y
quantitative measure of disease progression
4
SLIDE 5
Regression
Goal: learn real valued mapping
5
f : Rd → R + + + + + + + + + x y
SLIDE 6
Important choices in regression
6
What types of functions f should we consider? Examples How should we measure goodness of fit? + + + + + + + + + x f(x) + + ++ + + + + + + + + + + + + x f(x)
SLIDE 7
Example: linear regression
7
+ + + + + + + + + x y
SLIDE 8
Homogeneous representation
8
SLIDE 9
Quantifying goodness of fit
9
+ ++ + + + + + + x y D = {(x1, y1), . . . , (xn, yn)} xi ∈ Rd yi ∈ R
SLIDE 10
Least-squares linear regression optimization
[Legendre 1805, Gauss 1809]
Given data set How do we find the optimal weight vector?
10
w∗ = arg min
w n
X
i=1
(yi − wT xi)2 D = {(x1, y1), . . . , (xn, yn)} ˆ w
<latexit sha1_base64="fDIMbQ9YlPRf7DMzabXpUaK0cNU=">AB+XicbVDLSsNAFJ34rPUVdelmsAiuSlIFXbgouHFZwT6gCWUynbRDJ5Mwc1MpIX/ixoUibv0Td/6NkzYLbT0wcDjnXu6ZEySCa3Ccb2tfWNza7uyU93d2z84tI+OzpOFWVtGotY9QKimeCStYGDYL1EMRIFgnWDyV3hd6dMaR7LR5glzI/ISPKQUwJGti2NyaQeRGBcRBmT3k+sGtO3ZkDrxK3JDVUojWwv7xhTNOISaCaN13nQT8jCjgVLC86qWaJYROyIj1DZUkYtrP5slzfG6UIQ5jZ4EPFd/b2Qk0noWBWayiKiXvUL8z+unEN74GZdJCkzSxaEwFRhiXNSAh1wxCmJmCKGKm6yYjokiFExZVOCu/zlVdJp1N3LeuPhqta8LeuoFN0hi6Qi65RE92jFmojiqboGb2iNyuzXqx362MxumaVOyfoD6zPH1P/lBc=</latexit> SLIDE 11
Method 1: Closed form solution
The problem can be solved in closed form: Hereby:
11
w∗ = (XT X)−1XT y w∗ = arg min
w n
X
i=1
(yi − wT xi)2 ˆ w
<latexit sha1_base64="fDIMbQ9YlPRf7DMzabXpUaK0cNU=">AB+XicbVDLSsNAFJ34rPUVdelmsAiuSlIFXbgouHFZwT6gCWUynbRDJ5Mwc1MpIX/ixoUibv0Td/6NkzYLbT0wcDjnXu6ZEySCa3Ccb2tfWNza7uyU93d2z84tI+OzpOFWVtGotY9QKimeCStYGDYL1EMRIFgnWDyV3hd6dMaR7LR5glzI/ISPKQUwJGti2NyaQeRGBcRBmT3k+sGtO3ZkDrxK3JDVUojWwv7xhTNOISaCaN13nQT8jCjgVLC86qWaJYROyIj1DZUkYtrP5slzfG6UIQ5jZ4EPFd/b2Qk0noWBWayiKiXvUL8z+unEN74GZdJCkzSxaEwFRhiXNSAh1wxCmJmCKGKm6yYjokiFExZVOCu/zlVdJp1N3LeuPhqta8LeuoFN0hi6Qi65RE92jFmojiqboGb2iNyuzXqx362MxumaVOyfoD6zPH1P/lBc=</latexit>ˆ w
<latexit sha1_base64="fDIMbQ9YlPRf7DMzabXpUaK0cNU=">AB+XicbVDLSsNAFJ34rPUVdelmsAiuSlIFXbgouHFZwT6gCWUynbRDJ5Mwc1MpIX/ixoUibv0Td/6NkzYLbT0wcDjnXu6ZEySCa3Ccb2tfWNza7uyU93d2z84tI+OzpOFWVtGotY9QKimeCStYGDYL1EMRIFgnWDyV3hd6dMaR7LR5glzI/ISPKQUwJGti2NyaQeRGBcRBmT3k+sGtO3ZkDrxK3JDVUojWwv7xhTNOISaCaN13nQT8jCjgVLC86qWaJYROyIj1DZUkYtrP5slzfG6UIQ5jZ4EPFd/b2Qk0noWBWayiKiXvUL8z+unEN74GZdJCkzSxaEwFRhiXNSAh1wxCmJmCKGKm6yYjokiFExZVOCu/zlVdJp1N3LeuPhqta8LeuoFN0hi6Qi65RE92jFmojiqboGb2iNyuzXqx362MxumaVOyfoD6zPH1P/lBc=</latexit> SLIDE 12
How to solve? Example: Scikit Learn
12
SLIDE 13
Demo
13
Body mass index Disease progression
SLIDE 14
Method 2: Optimization
The objective function is convex!
14
ˆ R(w) = X
i
(yi − wT xi)2
SLIDE 15
Gradient Descent
Start at an arbitrary For t=1,2,... do Hereby, is called learning rate
15
w0 ∈ Rd ηt wt+1 = wt ηtr ˆ R(wt)
SLIDE 16
Convergence of gradient descent
Under mild assumptions, if step size sufficiently small, gradient descent converges to a stationary point (gradient = 0) For convex objectives, it therefore finds the
- ptimal solution!
In the case of the squared loss, constant stepsize ½ converges linearly
16
SLIDE 17
Computing the gradient
17
SLIDE 18
Demo: Gradient descent
18
SLIDE 19
Choosing a stepsize
What happens if we choose a poor stepsize?
19
SLIDE 20
Adaptive step size
Can update the step size adaptively. For example: 1) Via line search (optimizing step size every step) 2) „Bold driver“ heuristic
If function decreases, increase step size: If function increases, decrease step size:
20
SLIDE 21
Demo: Gradient Descent for Linear Regression
21
SLIDE 22
Gradient descent vs closed form
Why would one ever consider performing gradient descent, when it is possible to find closed form solution? Computational complexity May not need an optimal solution Many problems don‘t admit closed form solution
22
SLIDE 23
Other loss functions
So far: Measure goodness of fit via squared error Many other loss functions possible (and sensible!)
23