9.54 class 8 Supervised learning Optimization, regularization, - - PowerPoint PPT Presentation

9.54, fall semester 2014

9.54 class 8

Shimon Ullman + Tomaso Poggio

Danny Harari + Daneil Zysman + Darren Seibert

Supervised learning Optimization, regularization, kernels

The Regularization Kingdom

• Loss functions and empirical risk minimization
• Basic regularization algorithms

Math

Given a Training Set

f(x) ∼ y

Find

S = (x1, y1), . . . , (xn, yn)

We need a way to measure errors Loss function

V (f(x), y)

• 0 − 1-loss V (f(x), y) = ✓(−yf(x)) (✓ is the step function)
• square loss (L2) V (f(x), y) = (f(x) − y)2 = (1 − yf(x))2
• absolute value (L1) V (f(x), y) = |f(x) − y|
• Vapnik’s ✏-insensitive loss V (f(x), y) = (|f(x) − y| − ✏)+
• hinge loss V (f(x), y) = (1 − yf(x))+
• logistic loss V (f(x), y) = log(1 − e−yf(x)) logistic regression
• exponential loss V (f(x), y) = e−yf(x)

IS[f] = 1

n

Pn

i=1 V (f(xi), yi)

Given a loss function V (f(x), y) We can define the Empirical Error

``Learning processes do not take place in vacuum.’’

Cucker and Smale, AMS 2001

We need to fix a Hypotheses Space

H ⊂ F = {f | f : X → Y } F H

• Linear model f(x) = Pp

j=1 xjwj

• Generalized linear models f(x) = Pp

j=1 Φ(x)jwj

• Reproducing kernel Hilbert spaces f(x) = P

j1 Φ(x)jwj = P i1 K(x, xi)αi

K(x, x0) is a symmetric positive definite function called reproducing kernel

parametric non-parametric

F H H ⊂ F = {f | f : X → Y }

• Linear model f(x) = Pp

j=1 xjwj

• Generalized linear models f(x) = Pp

j=1 Φ(x)jwj

• Reproducing kernel Hilbert spaces f(x) = P

j1 Φ(x)jwj = P i1 K(x, xi)αi

K(x, x0) is a symmetric positive definite function called reproducing kernel

parametric semi-parametric

F H H ⊂ F = {f | f : X → Y }

• Linear model f(x) = Pp

j=1 xjwj

• Generalized linear models f(x) = Pp

j=1 Φ(x)jwj

• Reproducing kernel Hilbert spaces f(x) = P

j1 Φ(x)jwj = P i1 K(x, xi)αi

K(x, x0) is a symmetric positive definite function called reproducing kernel

parametric semi-parametric non-parametric

F H H ⊂ F = {f | f : X → Y }

Empirical Risk Minimization (ERM)

min

f∈H IS[f] = min f∈H

1 n

n

X

i=1

V (f(xi), yi)

Empirical Risk Minimization (ERM)

min

f∈H IS[f] = min f∈H

1 n

n

X

i=1

V (f(xi), yi)

Empirical Risk Minimization (ERM) Which is a good solution?

min

f∈H ES[f] = min f∈H

1 n

n

X

i=1

V (f(xi), yi)

Training set

house.) (living area of

Learning algorithm h predicted y x

(predicted price)

• f house)

S = (x1, y1), . . . , (xn, yn)

The training set is sampled identically and independently (i.i.d) from a fixed unknown probability distribution p(x, y) = p(x)p(y|x)

Ill posed problems often arise if one tries to infer general laws from few data the hypothesis space is too large there are not enough data

In general ERM leads to ill-posed solutions because the solution may be too complex it may be not unique it may change radically when leaving one sample out

Learning is an ill-posed problem

Jacques Hadamard

Regularization Theory provides results and techniques to restore well-posedness, that is stability (hence generalization)

• Beyond drawings & intuitions (...) there is a deep, rigorous

mathematical foundation of regularized learning algorithms (Cucker and Smale, Vapnik and Chervonenkis, ).

• Theory of learning is a synthesis of different fields, e.g. Computer Science

(Algorithms, Complexity) and Mathematics (Optimization, Probability, Statistics).

• Central to the Theory of Machine Learning is the problem of

understanding condition under which ERM can solve

inf E(f), E(f) = E(x,y) V (y, f(x))

Algorithms: The Regularization Kingdom

• loss functions and empirical risk minimization
• basic regularization algorithms

(Tikhonov) Regularization

min

f∈H{ 1

n

n

X

i=1

V (yi, f(xi)) + λR(f))} → f λ

S

regularization parameter regularizer

• The regularizer describes the complexity of the solution

R(f2) is bigger than R(f1) f1 f2

• The regularization parameter determines the trade-off

between complexity and empirical risk

min

f∈H

1 n

n

X

i=1

(yi − f(xi))2

Stability and (Tikhonov) Regularization

Consider f(x) = wT x = Pp

j=1 wjxj, and R(f) = wT w,

min

f∈H

( 1 n

n

X

i=1

(yi f(xi))2 + λkfk2 ) Math

wT = Y XT (XXT + λI)−1

wT = Y XT (XXT )−1

From Linear to Semi-parametric Models

If instead of a linear model we have a generalized linear model we simply have to consider

Xn =    Φ(x1)1 . . . . . . . . . Φ(x1)p . . . . . . . . . . . . . . . Φ(xn)1 . . . . . . . . . Φ(xn)p    f(x) =

p

X

j=1

xjwj | {z }

linear model

= ⇒ f(x) =

p

X

j=1

Φ(x)jwj | {z }

generalized linear model

From Parametric to Nonparametric Models

Then Some simple linear algebra shows that

We can compute Cn or wn depending whether n ≤ p.

The above result is the most basic form of the Representer Theorem. How about nonparametric models?

Math

wT = Y XT (XXT )−1 = Y (XT X)−1XT = CXT since XT (XXT )−1 = (XT X)−1XT f(x) = wT x = CXT x =

n

X

i

cixT

i x

From Linear to Nonparametric Models

f(x) =

p

X

j=1

wj

nxj = n

X

i=1

xT

i x

|{z}

Pp

j=1 xj i xj

ci

Note that We can now consider a truly non parametric model

f(x) = X

j≥1

wjΦ(x)j =

n

X

i=1

K(x, xi) | {z } ci

X

j≥1

Φ(xi)jΦ(x)j

Math

From Linear to Nonparametric Models

We have

Cn = (XnXT

n

| {z } +λnI)−1Yn (XnXT

n )i,j = xT i xj

Cn = ( Kn |{z} +λnI)−1Yn (Kn)i,j = K(xi, xj)

We can now consider a truly non parametric model

f(x) = X

j≥1

wjΦ(x)j =

n

X

i=1

K(x, xi) | {z } ci

X

j≥1

Φ(xi)jΦ(x)j

Math

Kernels

• Linear kernel

K(x, x0) = xT x0

• Gaussian kernel

K(x, x0) = e kxx0k2

σ2

, σ > 0

• Polynomial kernel

K(x, x0) = (xT x0 + 1)d, d ∈ N

• Inner Product kernel/Features

K(x, x0) =

p

X

j=1

Φ(x)jΦ(x0)j Φ : X → Rp.

Reproducing Kernel Hilbert Spaces

Note: An RKHS is equivalently defined as a Hilbert space where the evaluation functionals are continuous.

Given K, 9! Hilbert space of functions (H, h·, ·i) such that,

• Kx := K(x, ·) 2 H, for all x 2 X, and
• f(x) = hf, Kxi, for all x 2 X, f 2 H.

The norm of a function f(x) = Pn

i=1 K(x, xi)ci is given by

kfk2 =

n

X

i,j=1

K(xj, xi)cicj and is a natural complexity measure.

Math

Extensions: Other Loss Functions

• V (f(x), y) = (f(x) − y)2, RLS
• V (f(x), y) = (|f(x) − y| − ✏)+ SVM regression
• V (f(x), y) = (1 − yf(x))+ SVM classification
• V (f(x), y) = log(1 − e−yf(x)) logistic regression
• V (f(x), y) = e−yf(x) boosting

For most loss functions the solution of Tikhonov regularization is of the form

f(x) =

n

X

i=1

K(x, xi)ci.

Extensions: Other Loss Functions (cont)

By changing the loss function we change the way we compute the coefficients in expansion

f(x) =

n

X

i=1

K(x, xi)ci.

• Regularization avoids overfitting, ensures stability of the

solution and generalization

• There are many different instance of regularization beyond

Tikhonov, e.g. early stopping...

min

f

IS[f] | {z }

data fit term

+λ R(f) | {z }

complexity/smoothness term

• Regularization ensures

stability of the solution and generalization

• There are different instance of regularization beyond

Tikhonov, e.g. early stopping

Conclusions

• Regularization Theory provides results and techniques to

avoid overfitting (stability is key to generalization)

• Regularization provide a core set of concepts and

techniques to solve a variety of problems

• Most algorithms can be seen as a form of regularization

Hebbian mechanisms can be used for biological supervised learning (Knudsen, 1990)