9.54 Review Supervised learning + Invariant Learning Shimon Ullman - - PowerPoint PPT Presentation

9.54, fall semester 2014

9.54 Review

Shimon Ullman + Tomaso Poggio

Danny Harari + Daneil Zysman + Darren Seibert

Supervised learning + Invariant Learning

Thus Y − MX = 0 More in general look for M such that

min||Y − MX||2

if M = wT

rV (w) = 2(Y wT X)XT = 0 yields Y XT = wT XXT and wT = Y XT (XXT )−1

rV (w) = 2(Y wT X)XT + 2λwT = 0 yields Y XT = wT XXT + λwT and wT = Y XT (XXT + λI)−1

Now look for w such that

minw||Y − wT X||2 + λ||w||2

Then Some simple linear algebra shows that

We can compute Cn or wn depending whether n ≤ p.

The above result is the most basic form of the Representer Theorem.

Example: representer theorem in the linear case

Math

wT = Y XT (XXT )−1 = Y (XT X)−1XT = CXT since XT (XXT )−1 = (XT X)−1XT f(x) = wT x = CXT x =

n

X

i

cixT

i x

M-theory: exploring a new hypothesis

The main computational goal of the feedforward ventral stream hierarchy — and of vision — is to compute a representation for each incoming image which is invariant to transformations previously experienced in the visual environment.

Remarks:

• A theorem shows that invariant representations may reduce by orders of magnitude

the sample complexity of a classifier at the top of the hierarchy

• Empirical evidence also supports the claim
• Intuitions include the decrease in appearance that invariance afford

Transformation example: affine group

The action of a group transformation on an image is defined as:

g

gI(! x) = I(g−1! x)

gI(! x) = I(A−1! x − ! b), A ∈GL(2), ! b ∈R2

In the case of affine group:

I

Transformation example: affine group

The action of a group transformation on an image is defined as: In the case of affine group:

...

Our basic machine: a HW module

(dot products and histograms for an image in a receptive field window)

• The cumulative histogram (empirical cdf) can be be computed as
• This maps directly into a set of simple cells with threshold
• …and a complex cell indexed by n and k summating the simple cells

µn

k(I) =

1 | G | σ( I,git k + nΔ)

i=1 |G|

∑

nΔ

Dendrites of a complex cells as simple cells…

Active properties in the dendrites of the complex cell

...

Our basic machine: a HW module

• The (empirical) first moment — e.g. the average — is
• The (empirical) second moment is
• …and the infinite order moment is

m1 = 1 | G | I,git k

i=1 |G|

∑

m2 = 1 | G | ( I,git k

i=1 |G|

∑

)2 m∞ = maxi I,git k

Dendrites of a complex cells as simple cells…

Active properties in the dendrites of the complex cell