st rt tt - PowerPoint PPT Presentation

❋❛st ■♥❝r❡♠❡♥t❛❧ ❊①♣❡❝t❛t✐♦♥ ▼❛①✐♠✐③❛t✐♦♥✿ ❤♦✇ ♠❛♥② ✐t❡r❛t✐♦♥s ❢♦r ❛♥ ǫ ✲st❛t✐♦♥❛r② ♣♦✐♥t ❄ ●❡rs❡♥❞❡ ❋♦rt ❈◆❘❙ ✫ ■♥st✐t✉t ❞❡ ▼❛t❤é♠❛t✐q✉❡s ❞❡ ❚♦✉❧♦✉s❡✱ ❋r❛♥❝❡ ❈■❘▼ ✧❖♣t✐♠✐③❛t✐♦♥ ❢♦r ▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣✧✱ ▼❛r❝❤ ✷✵✷✵✳

❇❛s❡❞ ♦♥ ❛ ❥♦✐♥t ✇♦r❦ ✇✐t❤ • P✐❡rr❡ ●❛❝❤ ✲ ✭■▼❚✱ ❯♥✐✈✳ P❛✉❧ ❙❛❜❛t✐❡r✱ ❋r❛♥❝❡✮ • ❊r✐❝ ▼♦✉❧✐♥❡s ✲ ✭❈▼❆P✱ ❊❝♦❧❡ P♦❧②t❡❝❤♥✐q✉❡✱ ❋r❛♥❝❡✮ ❆❝❦♥♦✇❧❡❞❣♠❡♥ts✿ ✲ ❋♦♥❞❛t✐♦♥ ❙✐♠♦♥❡ ❡t ❈✐♥♦ ❉❡❧ ❉✉❝❛ ✲ ❖♣ t✐♠✐s❛t✐♦♥ ❡t ❙✐ ♠✉❧❛t✐♦♥ ▼♦ ♥t❡ ❈❛ r ❧♦ ✿ ❊ ♥tr❡❧❛❝❡♠❡♥ts ✲ ▲❛❜❡① ❈■▼■ ✲ ❙ t♦❝❤❛st✐❝ ❉ ❡s❝❡♥t ❆❧❣♦r✐t❤♠s ✇✐t❤ ▼ ❛r❦♦✈✐❛♥ ■ ♥♣✉ts

✳ ❖♣t✐♠✐③❛t✐♦♥ ♣❜♠ ❢♦r ❙t❛t✐st✐❝❛❧ ▲❡❛r♥✐♥❣ ✳ ❆♥ ♦♣t✐♠✐③❛t✐♦♥ ♣r♦❜❧❡♠ ♦❝❝✉rr✐♥❣✱ ❢♦r ❡①❛♠♣❧❡✱ ✐♥ ❧❛r❣❡ s❝❛❧❡ ❙t❛t✐st✐❝❛❧ ✐♥❢❡r❡♥❝❡

❚❤❡ ♦♣t✐♠✐③❛t✐♦♥ ♣r♦❜❧❡♠ n � = 1 F ( θ ) def L i ( θ ) + R ( θ ) argmin θ ∈ Θ F ( θ ) , n i =1 ✇✐t❤ • Θ ⊆ R d • n ✐s ❧❛r❣❡ • L i : Θ → R ✐s ♥♦t ❡①♣❧✐❝✐t ❛♥❞ ♦❢ t❤❡ ❢♦r♠ � L i ( θ ) def = − log Z exp ( � s i ( z ) , φ ( θ ) � ) d µ ( z ) • ◆♦ ❝♦♥✈❡①✐t② ❛ss✉♠♣t✐♦♥s • ✭❢♦r t❤❡ ❝✈❣ ❛♥❛❧②s✐s✮ ❘❡❣✉❧❛r✐t② ♣r♦♣❡rt✐❡s ♦❢ L i , R

▼♦t✐✈❛t✐♦♥✿ ❙t❛t✐st✐❝❛❧ ✐♥❢❡r❡♥❝❡ ✭✶✴✸✮ � n � = 1 def def argmin θ ∈ Θ F ( θ ) , F ( θ ) L i ( θ ) + R ( θ ) , L i ( θ ) = − log exp ( � s i ( z ) , φ ( θ ) � ) d µ ( z ) n Z i =1 • n ♦❜s❡r✈❛t✐♦♥s Y 1 , . . . , Y n ✿ ❛ ♣❛r❛♠❡t❡r✐❝ st❛t✐st✐❝❛❧ ♠♦❞❡❧ ✐♥❞❡①❡❞ ❜② θ ∈ Θ • R ( θ ) ✿ ❛ ♣❡♥❛❧t② t❡r♠✱ ❛ r❡❣✉❧❛r✐③❛t✐♦♥ t❡r♠✱ ❛ ♣r✐♦r ✭✐♥ ❛ ❇❛②❡s✐❛♥ s❡tt✐♥❣ s♦❧✈❡❞ ❜② ▼❆P✮ • ❆ ❧♦ss ❢✉♥❝t✐♦♥ ❛ss♦❝✐❛t❡❞ t♦ ❡❛❝❤ Y i ✿ L i ( θ ) • exp( −L i ) ✐s ❛ ✧s✉♠✧ ♦✈❡r ❧❛t❡♥t ✈❛r✐❛❜❧❡s z ∈ Z ✳ ❋♦r ❡①❛♠♣❧❡✱ � L i ( θ ) = − log Z . . . d µ ���� ���� � �� � ♥❡❣❛t✐✈❡ ❧♦❣✲ ❧✐❦❡❧✐❤♦♦❞ ♦❢ Y i

❊①❛♠♣❧❡✿ ♠✐①t✉r❡ ♠♦❞❡❧s ✭✷✴✸✮ • ❞❛t❛✿ y 1 , · · · , y n ♠♦❞❡❧❡❞ ❛s ✐✐❞ ❢r♦♠✿ � L ℓ =1 ω ℓ N ( µ ℓ , 1) θ = ( ω 1: L , µ 1: L ) • ❊q✉✐✈❛❧❡♥t❧②✿ z i ∈ { 1 , · · · , L } s✳t✳ L ( Y i | Z i = ℓ ) ∼ N ( µ ℓ , 1) L ( Z i ) ∼ ( ω ℓ ) ℓ ✳ • ❏♦✐♥t ❞✐str✐❜✉t✐♦♥ ♦❢ ( Y i , Z i ) ω ℓ /ω L = exp( α ℓ ) � L � L 1 z i = ℓ ω ℓ exp � − ( y i − µ ℓ ) 2 / 2 � 1 z i = ℓ exp � ln ω ℓ − ( y i − µ ℓ ) 2 / 2 � = ℓ =1 ℓ =1 � � � L � ln ω ℓ − ( y i − µ ℓ ) 2 / 2 � = exp 1 z i = ℓ ℓ =1 � L − 1 � � � L − 1 � α ℓ − ( y i − µ ℓ ) 2 / 2 + ( y i − µ L ) 2 / 2 � exp( α ℓ )) − ( y i − µ L ) 2 / 2 = exp 1 z i = ℓ − ln(1 + ℓ =1 ℓ =1 def − 1) , s i ( z ) = (1 z =1 , . . . , 1 z = L − 1 , y i 1 z =1 , . . . , y i 1 z = L − 1 , y i , � � L − 1 � , . . . , α L − 1 − µ 2 L − 1 − µ 2 α 1 − µ 2 1 − µ 2 def L L exp( α ℓ )) + µ 2 φ ( θ ) = , µ 1 , . . . , µ L − 1 , µ L , ln(1 + L / 2 2 2 ℓ =1

❊①❛♠♣❧❡✿ ▲♦❣✐st✐❝ r❡❣r❡ss✐♦♥ ✭✸✴✸✮ • ❞❛t❛✿ y 1 , · · · , y n ∈ { 0 , 1 } n ♠♦❞❡❧❡❞ ❛s ✐♥❞❡♣ ❢r♦♠ � � 1 • L ( Y i | Z i ) ∼ ❇❡r♥ L ( Z i ) ∼ N p ( θ, I) ✳ 1+exp( − η i ( z i )) • ❚❤❡ ❥♦✐♥t ❞✐str✐❜✉t✐♦♥ ♦❢ ( Y i , Z i ) � � Y i � � 1 − Y i exp � −� z i − θ � 2 / 2 � exp � −� z i − θ � 2 / 2 � exp( η i ( z i )) 1 exp( Y i η i ( z i )) = . 1 + exp( η i ( z i )) 1 + exp( η i ( z i )) 1 + exp( η i ( z i )) • ❚❤❡ ❧✐❦❡❧✐❤♦♦❞ ♦❢ Y i � � exp � −� z i − θ � 2 / 2 � exp( Y i η i ( z i )) d z = exp( � s i ( z ) , φ ( θ ) � ) d z 1 + exp( η i ( z i )) R p Z � y i η i ( z ) − ln(1 + exp( η i ( z ))) − � z � 2 / 2 , 1 , z ′ � s i ( z ) = � 1 , −� θ � 2 / 2 , θ ′ � φ ( θ ) =

❲❤✐❝❤ ♥✉♠❡r✐❝❛❧ t♦♦❧ ❄ n � = 1 F ( θ ) def argmin θ ∈ Θ F ( θ ) , L i ( θ ) + R ( θ ) , n i =1 � L i ( θ ) def = − log Z exp ( � s i ( z ) , φ ( θ ) � ) µ (d z ) . ❆♥ ❛❧❣♦r✐t❤♠✐❝ s♦❧✉t✐♦♥ ❞❡s✐❣♥❡❞ ❢♦r ✲ ❧❛r❣❡ n ✿ r❛r❡ ❝♦♠♣✉t❛t✐♦♥s ♦❢ ❛ s✉♠ ♦✈❡r n t❡r♠s ❛❧❧♦✇❡❞✱ ✲ ♥♦♥ ❝♦♥✈❡① s❡tt✐♥❣ ❙♦❧✉t✐♦♥✿ ❊①♣❡❝t❛t✐♦♥ ▼❛①✐♠✐③❛t✐♦♥✲❜❛s❡❞ ♠❡t❤♦❞s✿ ❉❡♠♣st❡r ❡t ❛❧✳ ✭✶✾✼✼✮✱ ❲✉ ✭✶✾✽✸✮

✳ ■■✲ ❊①♣❡❝t❛t✐♦♥ ▼❛①✐♠✐③❛t✐♦♥ ✭❊▼✮ ❛❧❣♦r✐t❤♠s ✳ ❊▼ ❛❧❣♦r✐t❤♠✿ ✐ts ❞❡r✐✈❛t✐♦♥ ❢♦r t❤✐s ♦♣t✐♠ ♣❜♠✱ ✐ts ✐♥tr❛❝t❛❜✐❧✐t②✱ ❛♥ ❛❧t❡r♥❛t✐✈❡✳

❊▼✿ ❆ ▼❛❥♦r✐③❡✲▼✐♥✐♠✐③❛t✐♦♥ ❛❧❣♦r✐t❤♠ ✭✶✴✷✮ � � n F ( θ ) = 1 Argmin θ F ( θ ) L i ( θ ) + R ( θ ) L i ( θ ) = − log exp( � s i ( z ) , φ ( θ ) � d µ ( z ) . n Z i =1 • ❚❤❡ s✉rr♦❣❛t❡ ❢✉♥❝t✐♦♥ ❛t t❤❡ ❝✉rr❡♥t ♣♦✐♥t θ k ∈ Θ ✱ ✭❏❡♥s❡♥✬s ✐♥❡q✉❛❧✐t②✮ n n � � 1 L i ( θ ) ≤ 1 L i ( θ k ) + � ¯ s ( θ k ) , φ ( θ k ) � − � ¯ s ( θ k ) , φ ( θ ) � n n i =1 i =1 ✇❤❡r❡ � n � = 1 Z s i ( z ) exp( � s i ( z ) , φ ( θ k ) � ) s ( θ k ) def s i ( θ k ) def ¯ ¯ s i ( θ k ) , ¯ = µ (d z ) . exp( −L i ( θ k )) n i =1 • ❬❊✲st❡♣❪ ❈♦♠♣✉t❡ ¯ s ( θ k ) • ❬▼✲st❡♣❪ ▼✐♥✐♠✐③❡ t❤❡ ♠❛❥♦r✐③✐♥❣ ❢✉♥❝t✐♦♥ ✭✉♥❞❡r ❤②♣✿ ✉♥✐q✉❡ ❛r❣♠✐♥✮ s ( θ k ) def θ k +1 = T ◦ ¯ = Argmin θ ∈ Θ {− � ¯ s ( θ k ) , φ ( θ ) � + R ( θ ) } .

❊▼ ✐♥ t❤❡ ❙t❛t✐st✐❝✲s♣❛❝❡ ✭✷✴✷✮ ❊ ▼ ❊ ▼ θ k ¯ s ( θ k ) θ k +1 = T ◦ ¯ s ( θ k ) ¯ s ◦ T ◦ ¯ s ( θ k ) θ k +2 = T ◦ ¯ s ◦ T ◦ ¯ s ( θ k ) F ( θ ) s ◦ T ( s k ) T ◦ ¯ s ◦ T ( s k ) s k T ( s k ) s k +1 = ¯ V ( s ) ❲❡ ✇✐❧❧ s❡❡ ❊▼✲❜❛s❡❞ ❛❧❣♦r✐t❤♠s ❛s ❡✈♦❧✈✐♥❣ ✐♥ t❤❡ ✧ s ✲s♣❛❝❡✧❀ t❤❡ ♦❜❥❡❝t✐✈❡ ❢✉♥❝t✐♦♥✿ V = F ◦ T • ■❢ ❝♦♥✈❡r❣❡♥❝❡✱ t♦ t❤❡ r♦♦ts ♦❢ h ( s ) def = ¯ s ◦ T ( s ) − s. ❯♥❞❡r ❛ss✉♠♣t✐♦♥s✱ t❤❡ r♦♦ts ♦❢ h ❛r❡ t❤❡ r♦♦ts ♦❢ ˙ V ✳ ❉♦ ❙t♦❝❤❛st✐❝ ❊▼ ❛✈♦✐❞ tr❛♣s ❄ ♥♦t t❤❡ t♦♣✐❝ t♦❞❛② ❊▼ ✐s ❞❡s✐❣♥❡❞ t♦ ✜♥❞ t❤❡ r♦♦ts ♦❢ h