[PPT] - 0.5 1 1.5 2 2.5 3 Maximum # of dihotomies Gro wth PowerPoint Presentation

SLIDE 1 Review

f

Le ture 5

Di hotomies
Gro

wth fun tion

mH(N)= max

x1,··· ,xN∈X |H(x1, · · · , xN)|

Break

p

int

PSfrag repla ements 0.5 1 1.5 2 2.5 3 3.5 4 0.5 1 1.5 2 2.5 3

Maximum

#

f

di hotomies

x1 x2 x3

SLIDE 2 Lea rning F rom Data Y aser S. Abu-Mostafa Califo rnia Institute

f

T e hnology Le ture 6: Theo ry

f

Generalization Sp

nso

red b y Calte h's Provost O e, E&AS Division, and IST

Thursda

y , Ap ril 19, 2012

SLIDE 3 Outline

Pro
f

that mH(N) is p

lynomial
Pro
f

that mH(N) an repla e M

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 6 2/18

SLIDE 4 Bounding mH(N) T

sho

w:

mH(N)

is p

lynomial

W e sho w:

mH(N) ≤ · · · ≤ · · · ≤

a p

lynomial

Key quantit y:

B(N, k)

: Maximum numb er

f

di hotomies

n N

p

ints,

with b reak p

int k

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 6 3/18

SLIDE 5 Re ursive b

und
n B(N, k)

#

f

ro ws

x1 x2 . . . xN−1 xN +1 +1 . . . +1 +1 −1 +1 . . . +1 −1 S1 α

. . . . . . . . . . . . . . .

+1 −1 . . . −1 −1 −1 +1 . . . −1 +1 +1 −1 . . . +1 +1 −1 −1 . . . +1 +1 S+

2

β

. . . . . . . . . . . . . . .

+1 −1 . . . +1 +1 S2 −1 −1 . . . −1 +1 +1 −1 . . . +1 −1 −1 −1 . . . +1 −1 S−

2

β

. . . . . . . . . . . . . . .

+1 −1 . . . +1 −1 −1 −1 . . . −1 −1

Consider the follo wing table:

B(N, k) = α + 2β

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 6 4/18

SLIDE 6 Estimating α and β #

f

ro ws

x1 x2 . . . xN−1 xN +1 +1 . . . +1 +1 −1 +1 . . . +1 −1 S1 α

. . . . . . . . . . . . . . .

+1 −1 . . . −1 −1 −1 +1 . . . −1 +1 +1 −1 . . . +1 +1 −1 −1 . . . +1 +1 S+

2

β

. . . . . . . . . . . . . . .

+1 −1 . . . +1 +1 S2 −1 −1 . . . −1 +1 +1 −1 . . . +1 −1 −1 −1 . . . +1 −1 S−

2

β

. . . . . . . . . . . . . . .

+1 −1 . . . +1 −1 −1 −1 . . . −1 −1

F

us
n x1, x2, · · · , xN−1
lumns:

α + β ≤ B(N − 1, k)

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 6 5/18

SLIDE 7 Estimating β b y itself #

f

ro ws

x1 x2 . . . xN−1 xN +1 +1 . . . +1 +1 −1 +1 . . . +1 −1 S1 α

. . . . . . . . . . . . . . .

+1 −1 . . . −1 −1 −1 +1 . . . −1 +1 +1 −1 . . . +1 +1 −1 −1 . . . +1 +1 S+

2

β

. . . . . . . . . . . . . . .

+1 −1 . . . +1 +1 S2 −1 −1 . . . −1 +1 +1 −1 . . . +1 −1 −1 −1 . . . +1 −1 S−

2

β

. . . . . . . . . . . . . . .

+1 −1 . . . +1 −1 −1 −1 . . . −1 −1

No w, fo us

n

the S2 = S+

2 ∪ S− 2

ro ws:

β ≤ B(N − 1, k − 1)

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 6 6/18

SLIDE 8 Putting it together #

f

ro ws

x1 x2 . . . xN−1 xN +1 +1 . . . +1 +1 −1 +1 . . . +1 −1 S1 α

. . . . . . . . . . . . . . .

+1 −1 . . . −1 −1 −1 +1 . . . −1 +1 +1 −1 . . . +1 +1 −1 −1 . . . +1 +1 S+

2

β

. . . . . . . . . . . . . . .

+1 −1 . . . +1 +1 S2 −1 −1 . . . −1 +1 +1 −1 . . . +1 −1 −1 −1 . . . +1 −1 S−

2

β

. . . . . . . . . . . . . . .

+1 −1 . . . +1 −1 −1 −1 . . . −1 −1

B(N, k) = α + 2β α + β ≤ B(N − 1, k) β ≤ B(N − 1, k − 1) B(N, k) ≤ B(N − 1, k) + B(N − 1, k − 1)

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 6 7/18

SLIDE 9 Numeri al

mputation
f B(N, k)

b

und

B(N, k) ≤ B(N − 1, k) + B(N − 1, k − 1)

bottom top

k 1 2 3 4 5 6 . . 1 1 2 2 2 2 2 . . 2 1 3 4 4 4 4 . . 3 1 4 7 8 8 8 . . N 4 1 5 11 . . . . . . . . 5 1 6 : . 6 1 7 : . : : : : .

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 6 8/18

SLIDE 10 Analyti solution fo r B(N, k) b

und

B(N, k) ≤ B(N − 1, k) + B(N − 1, k − 1)

bottom top

k 1 2 3 4 5 6 . . 1 1 2 2 2 2 2 . . 2 1 3 4 4 4 4 . . 3 1 4 7 8 8 8 . . N 4 1 5 11 . . . . . . . . 5 1 6 : . 6 1 7 : . : : : : .

Theo rem:

B(N, k) ≤

k−1

i=0

N i

1.

Bounda ry

nditions:

easy

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 6 9/18

SLIDE 11 2. The indu tion step

bottom top

N

k−1 k

N−1

k−1

i=0

N i

=

k−1

i=0

N − 1 i

+

k−2

i=0

N − 1 i

?

= 1+

k−1

i=1

N − 1 i

+

k−1

i=1

N − 1 i − 1

= 1 +

k−1

i=1

N − 1 i

+

N − 1 i − 1

= 1 +

k−1

i=1

N i

=

k−1

i=0

N i

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 6 10/18

SLIDE 12 It is p

lynomial!

F

r

a given H , the b reak p

int k

is xed

mH(N) ≤

k−1

i=0

N i

maximum

p

w

er is Nk−1

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 6 11/18

SLIDE 13 Three examples

k−1

i=0

N i

H

is p

sitive

ra ys: (b reak p

int k = 2)

mH(N) = N + 1 ≤ N + 1

H

is p

sitive

intervals: (b reak p

int k = 3)

mH(N) = 1

2N 2 + 1 2N + 1

≤

1 2N 2 + 1 2N + 1

H

is 2D p er eptrons: (b reak p

int k = 4)

mH(N) = ? ≤

1 6N 3 + 5 6N + 1

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 6 12/18

SLIDE 14 Outline

Pro
f

that mH(N) is p

lynomial
Pro
f

that mH(N) an repla e M

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 6 13/18

SLIDE 15 What w e w ant Instead

f:

P P [ |E in(g) − E

ut(g)| > ǫ ] ≤ 2

M e−2ǫ2N

W e w ant: P P [ |E in(g) − E

ut(g)| > ǫ ] ≤ 2 mH(N) e−2ǫ2N

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 6 14/18

SLIDE 16 Pi to rial p ro

f
Ho

w do es mH(N) relate to

verlaps?
What

to do ab

ut E
ut

?

Putting

it together

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 6 15/18

SLIDE 17

(a) (b) (c)

. data sets space of Hoeffding Inequality Union Bound VC Bound

D

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 6 16/18

SLIDE 18 What to do ab

ut E
ut

Ein(h)

Hi

E E

(h)

ut

in(h)

Hi

E (h)

ut

in(h)

E’

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 6 17/18

SLIDE 19 Putting it together Not quite: P P [ |E in(g) − E

ut(g)| > ǫ ] ≤ 2 mH( N ) e− 2 ǫ2N

but rather: P P [ |E in(g) − E

ut(g)| > ǫ ] ≤ 4 mH(2N) e− 1

8 ǫ2N

The V apnik-Chervonenkis Inequalit y

A

M L

Creato r: Y aser Abu-Mostafa

LFD

Le ture 6 18/18