Noisy ta rgets Review of Le ture 4 Erro r measures • y ou - User-sp e i�ed e y = f ( x ) − → y ∼ P ( y | x ) in truder • � � h ( x ) , f ( x ) UNKNOWN TARGET DISTRIBUTION x P y ( | ) target function f: X Y plus noise 8 +1 > < - In-sample: f UNKNOWN > − 1 : TRAINING EXAMPLES INPUT e in ( h ) = 1 ( , ), ... , ( , ) DISTRIBUTION - ( x 1 , y 1 ) , · · · , ( x N , y N ) generated b y x y x y 1 1 N N x P ( ) N � - Out-of-sample � � h ( x n ) , f ( x n ) E N n =1 e - is no w E x ,y e out ( h ) = E x out ( h ) P ( x , y ) = P ( x ) P ( y | x ) � � �� � � �� h ( x ) , f ( x ) h ( x ) , y E E
Lea rning F rom Data Y aser S. Abu-Mostafa Califo rnia Institute of T e hnology Le ture 5 : T raining versus T esting Sp onso red b y Calte h's Provost O� e, E&AS Division, and IST T uesda y , Ap ril 17, 2012 •
Outline F rom training to testing Illustrative examples • Key notion: b reak p oint • Puzzle • • Creato r: Y aser Abu-Mostafa - LFD Le ture 5 2/20 M � A L
The �nal exam T esting: P P [ | E in − E out | > ǫ ] ≤ 2 e − 2 ǫ 2 N T raining: P P [ | E in − E out | > ǫ ] ≤ 2 M e − 2 ǫ 2 N Creato r: Y aser Abu-Mostafa - LFD Le ture 5 3/20 M � A L
Where did the M ome from? The B ad events B m a re � | E out ( h m ) | > ǫ � in ( h m ) − E B 2 B 1 The union b ound: P [ B 1 or B 2 or · · · or B M ] no overlaps: M terms B 3 ≤ P [ B 1 ] + P [ B 2 ] + · · · + P [ B M ] Creato r: Y aser Abu-Mostafa - LFD Le ture 5 4/20 � �� � M � A L
Can w e imp rove on M ? Y es, bad events a re very overlapping! : hange in +1 and − 1 a reas out up : hange in lab els of data p oints in ∆ E 1 ∆ E +1 in ( h 1 ) − E out ( h 1 ) | ≈ | E in ( h 2 ) − E out ( h 2 ) | | E Creato r: Y aser Abu-Mostafa - LFD Le ture 5 5/20 down M � A L
What an w e repla e M with? Instead of the whole input spa e, w e onsider a �nite set of input p oints, and ount the numb er of di hotomies Creato r: Y aser Abu-Mostafa - LFD Le ture 5 6/20 M � A L
Di hotomies: mini-hyp otheses A hyp othesis A di hotomy h : X → {− 1 , +1 } Numb er of hyp otheses |H| an b e in�nite h : { x 1 , x 2 , · · · , x N } → {− 1 , +1 } Numb er of di hotomies |H ( x 1 , x 2 , · · · , x N ) | is at most 2 N Candidate fo r repla ing M Creato r: Y aser Abu-Mostafa - LFD Le ture 5 7/20 M � A L
The gro wth fun tion The gro wth fun tion ounts the most di hotomies on any N p oints The gro wth fun tion satis�es: m H ( N )= x 1 , ··· , x N ∈X |H ( x 1 , · · · , x N ) | max Let's apply the de�nition. m H ( N ) ≤ 2 N Creato r: Y aser Abu-Mostafa - LFD Le ture 5 8/20 M � A L
PSfrag repla ements PSfrag repla ements 0 0 Applying m H ( N ) de�nition - p er eptrons 0.5 0.5 1 1 1.5 1.5 PSfrag repla ements 2 2 2.5 2.5 0 3 3 0.5 3.5 3.5 1 4 4 1.5 1 1 2 1.2 1.2 2.5 1.4 1.4 3 1.6 1.6 3.5 1.8 1.8 4 2 2 0.5 2.2 2.2 1 2.4 2.4 1.5 2.6 2.6 2 2.8 2.8 2.5 3 3 3 N = 3 N = 3 N = 4 8 14 Creato r: Y aser Abu-Mostafa - LFD Le ture 5 9/20 m H (3) = m H (4) = M � A L
Outline F rom training to testing Illustrative examples • Key notion: b reak p oint • Puzzle • • Creato r: Y aser Abu-Mostafa - LFD Le ture 5 10/20 M � A L
PSfrag repla ements 0 0.2 0.4 0.6 0.8 1 -0.1 -0.08 -0.06 -0.04 -0.02 0 0.02 0.04 0.06 0.08 Example 1: p ositive ra ys 0.1 h ( x ) = − 1 h ( x ) = +1 a x 1 x 2 x 3 . . . x N is set of h : R → {− 1 , +1 } H h ( x ) = sign( x − a ) Creato r: Y aser Abu-Mostafa - LFD Le ture 5 11/20 m H ( N ) = N + 1 M � A L
PSfrag repla ements 0 0.2 0.4 0.6 0.8 1 -0.1 -0.08 -0.06 -0.04 -0.02 0 0.02 0.04 0.06 0.08 Example 2: p ositive intervals 0.1 h ( x ) = − 1 h ( x ) = +1 h ( x ) = − 1 x 1 x 2 x 3 . . . x N is set of h : R → {− 1 , +1 } Pla e interval ends in t w o of N + 1 sp ots H Creato r: Y aser Abu-Mostafa - LFD Le ture 5 12/20 � � 2 N 2 + 1 N +1 +1 = 1 m H ( N ) = 2 N + 1 2 M � A L
Example 3: onvex sets is set of h : R 2 → {− 1 , +1 } is onvex up + H − + h ( x ) = +1 + The N p oints a re `shattered' b y onvex sets m H ( N ) = 2 N − − − + Creato r: Y aser Abu-Mostafa - LFD Le ture 5 13/20 + − bottom M � A L
The 3 gro wth fun tions is p ositive ra ys: • H is p ositive intervals: m H ( N ) = N + 1 • H is onvex sets: 2 N 2 + 1 m H ( N ) = 1 2 N + 1 • H m H ( N ) = 2 N Creato r: Y aser Abu-Mostafa - LFD Le ture 5 14/20 M � A L
Ba k to the big pi ture Rememb er this inequalit y? P P [ | E in − E out | > ǫ ] ≤ 2 M e − 2 ǫ 2 N What happ ens if m H ( N ) repla es M ? p olynomial Go o d! Just p rove that m H ( N ) is p olynomial? m H ( N ) = ⇒ Creato r: Y aser Abu-Mostafa - LFD Le ture 5 15/20 M � A L
Outline F rom training to testing Illustrative examples • Key notion: b reak p oint • Puzzle • • Creato r: Y aser Abu-Mostafa - LFD Le ture 5 16/20 M � A L
PSfrag repla ements 0 0.5 1 1.5 2 Break p oint of H 2.5 3 3.5 De�nition: 4 If no data set of size k an b e shattered b y H , 0.5 then k is a b reak p oint fo r H 1 1.5 2 2.5 m H ( k ) < 2 k F o r 2D p er eptrons, k = 4 3 A bigger data set annot b e shattered either Creato r: Y aser Abu-Mostafa - LFD Le ture 5 17/20 M � A L
Break p oint - the 3 examples P ositive ra ys m H ( N ) = N + 1 b reak p oint k = 2 • P ositive intervals m H ( N ) = 1 • • b reak p oint k = 3 2 N 2 + 1 • 2 N + 1 Convex sets m H ( N ) = 2 N • • • b reak p oint k = ` ∞ ' • Creato r: Y aser Abu-Mostafa - LFD Le ture 5 18/20 M � A L
Main result No b reak p oint m H ( N ) = 2 N = ⇒ Any b reak p oint is p olynomial in N = ⇒ m H ( N ) Creato r: Y aser Abu-Mostafa - LFD Le ture 5 19/20 M � A L
Puzzle x 1 x 2 x 3 ◦ ◦ ◦ ◦ ◦ • ◦ • ◦ • ◦ ◦ • ◦ • Creato r: Y aser Abu-Mostafa - LFD Le ture 5 20/20 M � A L
Recommend
More recommend
