trt t ttst r - PowerPoint PPT Presentation

■♥tr♦❞✉❝t✐♦♥ t♦ ❙t❛t✐st✐❝❛❧ ▲❡❛r♥✐♥❣ ◆✐❝♦❧❛s ❱❛②❛t✐s ▲❡❝t✉r❡ ★ ✺ ✲ ❙t❛t✐st✐❝❛❧ ❛♥❛❧②s✐s ♦❢ ♠❛✐♥str❡❛♠ ▼▲ ❛❧❣♦r✐t❤♠s P❛rt ■ ✲ ❇♦♦st✐♥❣✱ ❙❱▼

▼❛✐♥ t❤❡♦r❡t✐❝❛❧ ♦❜❥❡❝t✐✈❡s ♦❢ t❤❡ ❝♦✉rs❡ • ❈♦♥s✐st❡♥❝② ♦❢ ❛ ✭r❛♥❞♦♠✮ s❡q✉❡♥❝❡ ♦❢ ❞❡❝✐s✐♦♥ r✉❧❡s ( � f n ) n ≥ ✶ ✿ f n ) → L ∗ ✐♥ ♣r♦❜❛❜✐❧✐t② ❛s n → ∞ , L ( � • ❯♣♣❡r ❜♦✉♥❞s ✿ ❈♦♥s✐❞❡r � f n ∈ F ✳ ❲✐t❤ ♣r♦❜❛❜✐❧✐t② ❛t ❧❡❛st ✶ − δ ✱ t❤❡r❡ ❡①✐sts s♦♠❡ ❝♦♥st❛♥t c s✉❝❤ t❤❛t ✿ � log( ✶ /δ ) L ( � f n ) − inf F L ≤ C ( F , n ) + c , n ✇❤❡r❡ C ( F , n )) = O ( ✶ / √ n ) ❛❢t❡r ♣r♦❝❡ss✐♥❣ s♦♠❡ ❝♦♠♣❧❡①✐t②✴st❛❜✐❧✐t② ♠❡❛s✉r❡

▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣ ▼❡t❤♦❞s ❖♣t✐♠✐③❛t✐♦♥ ✐s ❝❡♥tr❛❧ ❙♦♠❡ ♣♦♣✉❧❛r ❡①❛♠♣❧❡s ✿ • ✭❙♣❛rs❡✮ ▲✐♥❡❛r ♠♦❞❡❧s − → ❣r❛❞✐❡♥t ♠❡t❤♦❞ ✭❛♥❞ ❡①t❡♥s✐♦♥s✮ • ❑❡r♥❡❧ r✐❞❣❡ r❡❣r❡ss✐♦♥ − → q✉❛❞r❛t✐❝ ♦♣t✐♠✐③❛t✐♦♥ ✭✇✐t❤ ❑❑❚ ❝♦♥❞✐t✐♦♥s✮ • ❉❡❡♣ ❧❡❛r♥✐♥❣ − → ♥♦♥❝♦♥✈❡① ♦♣t✐♠✐③❛t✐♦♥ ✭st♦❝❤❛st✐❝ ❣r❛❞✐❡♥t ❞❡s❝❡♥t✮ ✰ ✐♠♣❧✐❝✐t r❡❣✉❧❛r✐③❛t✐♦♥ ✭tr✐❝❦s✮

❑❡② ♣r✐♥❝✐♣❧❡ ✿ ❘❡❣✉❧❛r✐③❡❞ ♦♣t✐♠✐③❛t✐♦♥ • ❖❜❥❡❝t✐✈❡ ✿ t♦ ✜♥❞ ❛ ❢✉♥❝t✐♦♥ t❤❛t ✜ts t❤❡ ❞❛t❛ ❛♥❞ ❞✐s♣❧❛②s ♣r❡❞✐❝t✐✈❡ ♣♦✇❡r • ❯♥t✐❧ ♥♦✇ ✿ ▲❡❛r♥✐♥❣ ❛♠♦✉♥ts t♦ t❤❡ ♠✐♥✐♠✐③❛t✐♦♥ ♦❢ tr❛✐♥✐♥❣ ❡rr♦r ❢♦r s♦♠❡ ❧♦ss ❢✉♥❝t✐♦♥ ♦✈❡r t❤❡ ❤②♣♦t❤❡s✐s ❝❧❛ss ♦❢ ❢✉♥❝t✐♦♥s h ∈ H ♣❧✉s s♦♠❡ ♣❡♥❛❧t② ˆ C n ( h ) = L n ( h ) + λ ♣❡♥ ( h , n ) � �� � � �� � ❚r❛✐♥✐♥❣ ❡rr♦r ❘❡❣✉❧❛r✐③❛t✐♦♥ • ❊①❛♠♣❧❡ ✿ r✐❞❣❡ r❡❣r❡ss✐♦♥ ✇❤❡r❡ h ( x ) = θ T x ✿ � n i = ✶ ( Y i − θ T X i ) ✷ ❛♥❞ ♣❡♥ ( h , n ) = ✶ ˆ L n ( h ) = ✶ n � θ � ✷ ✷ n • ❚❤❡ ♣❡♥❛❧t② ❣r♦✇s ✇✐t❤ t❤❡ ❝♦♠♣❧❡①✐t② ♦❢ h ✭♦r t❤❡ s✐③❡ ♦❢ H ✮ ❛♥❞ ✈❛♥✐s❤❡s ✇❤❡♥ n → ∞

❖t❤❡r ❢♦r♠s ♦❢ r❡❣✉❧❛r✐③❛t✐♦♥ • ●❡♥❡r❛❧ ✐❞❡❛ ✿ ❘❡❣✉❧❛r✐③❡❞ ❢✉♥❝t✐♦♥ ❡st✐♠❛t✐♦♥ ✇✐t❤♦✉t ❣❧♦❜❛❧ ♦♣t✐♠✐③❛t✐♦♥ • ❚✇♦ ❞✐r❡❝t✐♦♥s ✿ • ▲♦❝❛❧ ♠❡t❤♦❞s ✿ ♥❡❛r❡st✲♥❡✐❣❤❜♦rs ❛♥❞ ❞❡❝✐s✐♦♥ tr❡❡s • ❊♥s❡♠❜❧❡ ♠❡t❤♦❞s ✿ ❜❛❣❣✐♥❣✱ ❜♦♦st✐♥❣✱ r❛♥❞♦♠ ❢♦r❡sts

❘❡❣✉❧❛r✐③❛t✐♦♥ ✇✐t❤♦✉t ♦♣t✐♠✐③❛t✐♦♥ ❚❤❡ ❝❛s❡ ♦❢ ❤✐st♦❣r❛♠s ❉✐str✐❜✉t✐♦♥ ♦❢ t❤❡ ❛❣❡ ♦❢ t❤❡ ♣❛ss❡♥❣❡rs ♦❢ t❤❡ ❚✐t❛♥✐❝ ✇✐t❤ ❜✐♥s ✈❛r②✐♥❣ ❢r♦♠ ✶ ②❡❛r t♦ ✶✺ ②❡❛rs

■♥❣r❡❞✐❡♥ts ❢♦r t❤❛t t②♣❡ ♦❢ r❡❣✉❧❛r✐③❛t✐♦♥ • ❍✐st♦❣r❛♠s ✉s❡ t✇♦ ❣❡♥❡r❛❧ ✐❞❡❛s ♦❢ ❧♦❝❛❧✐t② ✭❜✐♥s✮ ❛♥❞ ❛✈❡r❛❣✐♥❣ ✭♣✐❡❝❡✇✐s❡ ❝♦♥st❛♥t ❢✉♥❝t✐♦♥✮ • ❞❡✜♥❡ ❧♦❝❛❧ ✿ ✇❤✐❝❤ tr❛✐♥✐♥❣ ❞❛t❛ ❝❛♥ ❜❡ ❝♦♥s✐❞❡r❡❞ t♦ ❜❡ ❝❧♦s❡ t♦ t❤❡ ♣♦✐♥t ✇❤❡r❡ ❛ ♣r❡❞✐❝t✐♦♥ ❤❛s t♦ ❜❡ ♠❛❞❡ ❄ • ❛✈❡r❛❣✐♥❣ ✭♦r ✈♦t✐♥❣ ✐❢ ❞✐s❝r❡t❡ ♦✉t❝♦♠❡✮ ✿ t❛❦❡ t❤❡ ❛✈❡r❛❣❡ ♦❢ t❤❡ ✈❛❧✉❡s ♦✈❡r ❡❛❝❤ ❜✐♥ • ❘❡❣✉❧❛r✐③❛t✐♦♥ t❤r♦✉❣❤ ❤②♣❡r♣❛r❛♠❡t❡r s❡❧❡❝t✐♦♥ ✿ ✜♥❞ t❤❡ ♦♣t✐♠❛❧ ❜✐♥ s✐③❡ ❛♠♦✉♥ts t♦ ✜♥❞✐♥❣ t❤❡ r✐❣❤t ❤②♣♦t❤❡s✐s ❝❧❛ss

❋r♦♠ ❤✐st♦❣r❛♠s t♦ ▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣ • ■♥ t❤❡ ♣r❡✈✐♦✉s ❡①❛♠♣❧❡✱ t❤❡ ♦❜❥❡❝t✐✈❡ ✇❛s t♦ ❡st✐♠❛t❡ ❛ ❞❡♥s✐t② ❢✉♥❝t✐♦♥ ❢r♦♠ ❛ s❛♠♣❧❡ ❞r❛✇♥ ❢r♦♠ t❤✐s ❞✐str✐❜✉t✐♦♥ ✭♣r♦❜❧❡♠ ❦♥♦✇♥ ✐♥ t❤❡ ❧✐t❡r❛t✉r❡ ❛s ♥♦♥♣❛r❛♠❡tr✐❝ ❞❡♥s✐t② ❡st✐♠❛t✐♦♥ ♦r ❦❡r♥❡❧ ❞❡♥s✐t② ❡st✐♠❛t✐♦♥ ✮ • ❉❡♥s✐t② ❡st✐♠❛t✐♦♥ ❝❛♥ ❜❡ s❡❡♥ ❛s ❛♥ ✉♥s✉♣❡r✈✐s❡❞ ❧❡❛r♥✐♥❣ ♣r♦❜❧❡♠ • ■♥ t❤❡ s✉♣❡r✈✐s❡❞ s❡tt✐♥❣✱ ✇❡ ❡st❛❜❧✐s❤ t❤❡ ✈❛❧✉❡s ♦❢ t❤❡ ❢✉♥❝t✐♦♥ ♦♥ ❡✈❡r② ❜✐♥ ❡✐t❤❡r ❜② ❛✈❡r❛❣✐♥❣ ✭r❡❣r❡ss✐♦♥ s❡t✉♣✮ ♦r ❜② ✈♦t✐♥❣ ✭❝❧❛ss✐✜❝❛t✐♦♥ s❡t✉♣✮✳ ❚❤❡ ❣❡♥❡r❛❧ t❡r♠✐♥♦❧♦❣② ❢♦r ❛✈❡r❛❣✐♥❣✴✈♦t✐♥❣ ✐s ❛❣❣r❡❣❛t✐♥❣✴❝♦♠❜✐♥✐♥❣✳

❖✈❡r✈✐❡✇ ♦❢ ❈❤❛♣t❡r ✸ ✶✳ ❈♦♥s✐st❡♥❝② ♦❢ ❧♦❝❛❧ ♠❡t❤♦❞s ✿ ❛✳ ❦✲◆❡❛r❡st ◆❡✐❣❤❜♦rs ❜✳ ❞❡❝✐s✐♦♥ tr❡❡s ❝✳ ✭❧♦❝❛❧ ❛✈❡r❛❣✐♥❣✮ ✷✳ ❈♦♥s✐st❡♥❝② ♦❢ ❣❧♦❜❛❧ ♠❡t❤♦❞s ❛✳ ❇♦♦st✐♥❣ ❜✳ ❙✉♣♣♦rt ❱❡❝t♦r ▼❛❝❤✐♥❡s ❝✳ ◆❡✉r❛❧ ♥❡t✇♦r❦s ✸✳ ❈♦♥s✐st❡♥❝② ♦❢ ❡♥s❡♠❜❧❡ ♠❡t❤♦❞s ✰ ❇❛❣❣✐♥❣✱ ❘❛♥❞♦♠ ❋♦r❡sts

✶✳ ❖❧❞❡r ▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣ ❛♣♣r♦❛❝❤❡s ✿ ▲♦❝❛❧ ♠❡t❤♦❞s ❛✳ ◆❡❛r❡st ♥❡✐❣❤❜♦rs ❜✳ ❉❡❝✐s✐♦♥ tr❡❡s

❚✇♦ ♣♦♣✉❧❛r t②♣❡s ♦❢ ❧♦❝❛❧ ♠❡t❤♦❞s • ◆❡❛r❡st ♥❡✐❣❤❜♦rs ✿ ❧♦❝❛❧ ❛r❡ t❤❡ ❝❧♦s❡st ♣♦✐♥ts • P❛rt✐t✐♦♥✲❜❛s❡❞ r✉❧❡s ✭❛❧s♦ ❝❛❧❧❡❞ ❞❡❝✐s✐♦♥ tr❡❡s ✮ ✿ ❧♦❝❛❧ ❛r❡ t❤❡ ♣♦✐♥ts ✇✐t❤✐♥ ❛ ❝❡❧❧ ❢r♦♠ ❛ ♣❛rt✐t✐♦♥ ♦❢ t❤❡ ✐♥♣✉t s♣❛❝❡ ♦♥❧② ❲♦r❦s ❢♦r ❝❧❛ss✐✜❝❛t✐♦♥✱ r❡❣r❡ss✐♦♥ ❛♥❞ ♦t❤❡r ♣r♦❜❧❡♠s✳✳✳ ❜✉t ❤❡r❡ ✇❡ ✇✐❧❧ ❢♦❝✉s ♦♥ ❝❧❛ss✐✜❝❛t✐♦♥

Pr♦❜❧❡♠ ❝♦♥s✐❞❡r❡❞ ✭▼✉❧t✐❝❧❛ss✮ ❈❧❛ss✐✜❝❛t✐♦♥ • ●✐✈❡♥ ✿ • ❈♦♥s✐❞❡r ❛ s❛♠♣❧❡ ♦❢ ❝❧❛ss✐✜❝❛t✐♦♥ ❞❛t❛ ( X ✶ , Y ✶ ) ... ( X n , Y n ) ✇❤❡r❡ X i ∈ R d ✈❡❝t♦r ♦❢ ✐♥❞❡♣❡♥❞❡♥t ✈❛r✐❛❜❧❡s✱ Y i ∈ { ✶ , . . . , C } t❤❡ ❧❛❜❡❧ • ❲❛♥t ✿ • t♦ ♣r❡❞✐❝t t❤❡ ❧❛❜❡❧ y ❛t ❛♥② ♣♦s✐t✐♦♥ x

✶✳ ▲♦❝❛❧ ♠❡t❤♦❞s ❛✳ k ✲◆❡❛r❡st ♥❡✐❣❤❜♦rs ✭ k ✲◆◆✮

k ✲◆❡❛r❡st ◆❡✐❣❤❜♦r ✭✶✴✹✮ Pr✐♥❝✐♣❧❡ ♦❢ t❤❡ k ✲◆◆ ❛❧❣♦r✐t❤♠ ✶ ❈♦♠♣✉t❡ ❞✐st❛♥❝❡s • ❈♦♠♣✉t❡ ♣❛✐r✇✐s❡ ❞✐st❛♥❝❡s d ( x , X i ) ❢♦r ❛❧❧ i = ✶ , . . . , n ✷ ❙♦rt tr❛✐♥✐♥❣ ❞❛t❛ • ❙♦rt t❤❡ ❞❛t❛ ♣♦✐♥ts ❢r♦♠ t❤❡ ❝❧♦s❡st X ( ✶ ) t♦ t❤❡ ❢❛rt❤❡st X ( n ) ✭✐✳❡✳ d ( x , X ( ✶ ) ) ≤ . . . ≤ d ( x , X ( n ) ) ✸ Pr❡❞✐❝t✐♦♥ ˆ h ( x , k ) ❂ ▼❛❥♦r✐t② ✈♦t❡ ♦❢ t❤❡ k ✲◆◆ • ❈♦♥s✐❞❡r t❤❡ ❧❛❜❡❧s Y ( ✶ ) , . . . , Y ( k ) ♦❢ t❤❡ k ❝❧♦s❡st ♣♦✐♥ts t♦ x ❛♥❞ t❛❦❡ t❤❡ ♠❛❥♦r✐t② ✈♦t❡ h ( x , k ) = ❛r❣ ♠❛① c { � k ˆ l = ✶ I { Y ( l ) = c }}

k ✲◆❡❛r❡st ◆❡✐❣❤❜♦r ✭✷✴✹✮ Pr✐♥❝✐♣❧❡ ♦❢ t❤❡ k ✲◆◆ ❛❧❣♦r✐t❤♠