r s trs t - PowerPoint PPT Presentation

❊♥❝♦❞✐♥❣ ❲♦r❞ ❈♦♥❢✉s✐♦♥ ◆❡t✇♦r❦s ✇✐t❤ ❘❡❝✉rr❡♥t ◆❡✉r❛❧ ◆❡t✇♦r❦s ❢♦r ❉✐❛❧♦❣ ❙t❛t❡ ❚r❛❝❦✐♥❣ ●❧♦r✐❛♥♥❛ ❏❛❣❢❡❧❞ ✱ ◆❣♦❝ ❚❤❛♥❣ ❱✉ ■♥st✐t✉t❡ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣ ✭■▼❙✮✱ ❯♥✐✈❡rs✐t② ♦❢ ❙t✉tt❣❛rt ✼t❤ ♦❢ ❙❡♣t❡♠❜❡r ✷✵✶✼ ❙❈◆▲P ❅ ❊▼◆▲P ✷✵✶✼ ✲ ❙❤♦rt P❛♣❡r

▼♦t✐✈❛t✐♦♥

■♥tr♦❞✉❝t✐♦♥ ✫ ▼♦t✐✈❛t✐♦♥ ▼♦❞❡❧ ❊①♣❡r✐♠❡♥ts ❘❡s✉❧ts ❈♦♥❝❧✉s✐♦♥s ❘❡❢❡r❡♥❝❡s ▼♦❞✉❧❛r ❙♣♦❦❡♥ ❉✐❛❧♦❣ ❙②st❡♠ ❊♥❝♦❞✐♥❣ ❈♥❡ts ✇✐t❤ ❘◆◆s ❢♦r ❉❙❚ ❏❛❣❢❡❧❞✱ ❱✉ ✸✴✶✼

■♥tr♦❞✉❝t✐♦♥ ✫ ▼♦t✐✈❛t✐♦♥ ▼♦❞❡❧ ❊①♣❡r✐♠❡♥ts ❘❡s✉❧ts ❈♦♥❝❧✉s✐♦♥s ❘❡❢❡r❡♥❝❡s ❲♦r❞ ❈♦♥❢✉s✐♦♥ ◆❡t✇♦r❦ ✭❈♥❡t✮ ❤❛✈❡ ❬✵✳✹❪ ✈❡❛❧ ❬✵✳✸❪ ✜♥❡ ❬✵✳✶❪ ■ ❬✵✳✾❪ ✐t ❬✵✳✷❪ ♦❢t❡♥ ❬✵✳✸❪ ε ❬✵✳✶❪ ε ❬✵✳✽❪ ♠♦✈❡ ❬✵✳✻❪ ✈❡r② ❬✵✳✼❪ ❢❛st ❬✵✳✻❪ • ❘✐❝❤❡r ❆❙❘ ❤②♣♦t❤❡s✐s s♣❛❝❡ t❤❛♥ ♥✲❜❡st ❧✐st • ▼♦r❡ ❝♦♠♣❛❝t ❞❛t❛ str✉❝t✉r❡ t❤❛♥ s♣❡❡❝❤ ❧❛tt✐❝❡s • ❊✈❡r② ❧❛tt✐❝❡ ❝❛♥ ❜❡ ❝♦♥✈❡rt❡❞ t♦ ❛ ❝♥❡t ✇✐t❤♦✉t s✐❣♥✐✜❝❛♥t ❧♦ss ♦❢ ❤②♣♦t❤❡s❡s ✭▼❛♥❣✉ ❡t ❛❧✳✱ ✷✵✵✵❀ ❄ ✮ ❊♥❝♦❞✐♥❣ ❈♥❡ts ✇✐t❤ ❘◆◆s ❢♦r ❉❙❚ ❏❛❣❢❡❧❞✱ ❱✉ ✹✴✶✼

■♥tr♦❞✉❝t✐♦♥ ✫ ▼♦t✐✈❛t✐♦♥ ▼♦❞❡❧ ❊①♣❡r✐♠❡♥ts ❘❡s✉❧ts ❈♦♥❝❧✉s✐♦♥s ❘❡❢❡r❡♥❝❡s ❈♦♥tr✐❜✉t✐♦♥s • Pr♦♣♦s❡ t♦ ♠✐t✐❣❛t❡ ❞❛♠❛❣❡ ♦❢ ❆❙❘ ❡rr♦rs ❜② ✉s✐♥❣ ❝♥❡ts • ❋✐rst st❡♣ t♦✇❛r❞s t✐❣❤t❡r ✐♥t❡❣r❛t✐♦♥ ♦❢ ❆❙❘ ✐♥t♦ ❡♥❞✲t♦✲❡♥❞ ❙❉❙s • ◆♦✈❡❧ ❛❧❣♦r✐t❤♠ t♦ ❡♥❝♦❞❡ ❝♥❡ts ✈✐❛ r❡❝✉rr❡♥t ♥❡✉r❛❧ ♥❡t✇♦r❦ ✭❘◆◆s✮ ✇✐t❤ ❣❛t❡❞ r❡❝✉rr❡♥t ✉♥✐ts ✭●❘❯s✮ ✭ ❄ ✮ • ❙❤♦✇ t❤❛t ❡♥❝♦❞✐♥❣ ❝♥❡ts ✐♠♣r♦✈❡s ❉❙❚ ♣❡r❢♦r♠❛♥❝❡ ♦✈❡r ❆❙❘ ✶✲❜❡st ❜❛s❡❧✐♥❡ ❊♥❝♦❞✐♥❣ ❈♥❡ts ✇✐t❤ ❘◆◆s ❢♦r ❉❙❚ ❏❛❣❢❡❧❞✱ ❱✉ ✺✴✶✼

▼♦❞❡❧

■♥tr♦❞✉❝t✐♦♥ ✫ ▼♦t✐✈❛t✐♦♥ ▼♦❞❡❧ ❊①♣❡r✐♠❡♥ts ❘❡s✉❧ts ❈♦♥❝❧✉s✐♦♥s ❘❡❢❡r❡♥❝❡s ▼♦❞❡❧ p 0 (slot) p 1 (slot) p 2 (slot) ❝❧❛ss✐✜❡r s j , u j ✿ ●❘❯✲❜❛s❡❞ ❝♥❡t ❡♥✲ ❝♦❞❡r ♦✉t♣✉ts ❛t t❤❡ ❡♥❞ ♦❢ softmax softmax softmax ❡❛❝❤ s②st❡♠ ❛♥❞ ✉s❡r ✉tt❡r✲ s 0 ⊕ u 0 s 1 ⊕ u 1 s 2 ⊕ u 2 ❛♥❝❡ confusion network GRU := W s s j + W u u j + b ✱ ⊕ h 0 h 1 h 2 h 3 h 4 h 5 h 6 h 7 h 8 h 9 h 10 h 11 ✇❡✐❣❤t❡❞ s✉♠ ♦❢ s②st❡♠ ❛♥❞ x 3 2 x 6 2 x 11 2 x 1 1 x 3 1 x 5 1 x 6 1 x 10 1 x 11 1 ✉s❡r ✐♥❢♦r♠❛t✐♦♥ x 0 0 x 1 0 x 2 0 x 3 0 x 4 0 x 5 0 x 6 0 x 7 0 x 8 0 x 9 0 x 10 0 x 11 0 ❡♥❝♦❞❡r dense layer with ReLU activation d t ✿ ♦♥❡✲❤♦t ✈❡❝t♦rs ♦❢ s②st❡♠ e 3 2 e 6 2 e 11 2 ❞✐❛❧♦❣ ❛❝ts e 1 1 e 3 1 e 5 1 e 6 1 e 10 1 e 11 1 e 0 0 e 1 0 e 2 0 e 3 0 e 4 0 e 5 0 e 6 0 e 7 0 e 8 0 e 9 0 e 10 0 e 11 0 w t i ✿ ♦♥❡✲❤♦t ✈❡❝t♦rs ♦❢ ✇♦r❞ ❤②♣♦t❤❡s❡s ✐♥ t❤❡ ❝♥❡t embedding layer t✐♠❡st❡♣s ♦❢ ✉s❡r ✉tt❡r❛♥❝❡s w 2 2 w 4 2 w 6 2 w 0 1 w 2 1 w 3 1 w 4 1 w 5 1 w 6 1 d 0 w 0 0 w 1 0 w 2 0 d 1 w 3 0 w 4 0 d 2 d 3 d 4 w 5 0 w 6 0 ❇❛s✐s✿ ❩✐❧❦❛ ❛♥❞ ❏✉r❝í❝❡❦ ✭✷✵✶✺✮ ❊♥❝♦❞✐♥❣ ❈♥❡ts ✇✐t❤ ❘◆◆s ❢♦r ❉❙❚ ❏❛❣❢❡❧❞✱ ❱✉ ✼✴✶✼

■♥tr♦❞✉❝t✐♦♥ ✫ ▼♦t✐✈❛t✐♦♥ ▼♦❞❡❧ ❊①♣❡r✐♠❡♥ts ❘❡s✉❧ts ❈♦♥❝❧✉s✐♦♥s ❘❡❢❡r❡♥❝❡s ●❘❯✲❜❛s❡❞ ❈♥❡t ❊♥❝♦❞❡r ❊♥❝♦❞✐♥❣ k ❛❧t❡r♥❛t✐✈❡ ❤②♣♦t❤❡s❡s ❛t t✐♠❡st❡♣ t ♦❢ ❛ ❝♥❡t✿ z t ✵ ˜ x t ✵ h t ✵ h t ✲✶ ✵ h t i = z t i · h t − ✶ + ( ✶ − z t i ) · ˜ r t ✵ h t i · · · · · · ✶ − z t ✵ z t i = σ ( W z x t i + U z h t − ✶ + b z ) ˜ · · · · · · · · · h t ✲✶ h t h t i = t❛♥❤ ( W h x t i + U h ( r t i · h t − ✶ ) + b h ) ✶ − z t k − ✶ · · · · · · r t k − ✶ r t i = σ ( W r x t i + U r h t − ✶ + b r ) ˜ x t k − ✶ h t k − ✶ h t = f ♣♦♦❧ ( h t ✵ . . . h t k − ✶ ) h t ✲✶ k − ✶ z t k − ✶ ❇❛s❡❞ ♦♥ r❡❝❡♥t ❛♣♣r♦❛❝❤❡s t♦ ❡♥❝♦❞❡ ❧❛tt✐❝❡s ✈✐❛ ❘◆◆s ✭▲❛❞❤❛❦ ❡t ❛❧✳✱ ✷✵✶✻❀ ❙✉ ❡t ❛❧✳✱ ✷✵✶✼✮ ❊♥❝♦❞✐♥❣ ❈♥❡ts ✇✐t❤ ❘◆◆s ❢♦r ❉❙❚ ❏❛❣❢❡❧❞✱ ❱✉ ✽✴✶✼

■♥tr♦❞✉❝t✐♦♥ ✫ ▼♦t✐✈❛t✐♦♥ ▼♦❞❡❧ ❊①♣❡r✐♠❡♥ts ❘❡s✉❧ts ❈♦♥❝❧✉s✐♦♥s ❘❡❢❡r❡♥❝❡s ❈❤♦✐❝❡s ❢♦r t❤❡ P♦♦❧✐♥❣ ❋✉♥❝t✐♦♥ � k i = ✶ h t i ❛✈❡r❛❣❡ ♣♦♦❧✐♥❣ : f ❛✈❡r❛❣❡ = k k � ✇❡✐❣❤t❡❞ ♣♦♦❧✐♥❣ : f ✇❡✐❣❤t❡❞ = s❝♦r❡ i · h t i ✱ ✇❤❡r❡ s❝♦r❡ i ✐s t❤❡ i = ✶ ❝♦♥✜❞❡♥❝❡ s❝♦r❡ ♦❢ ❝♥❡t ❤②♣♦t❤❡s✐s x t i ❊♥❝♦❞✐♥❣ ❈♥❡ts ✇✐t❤ ❘◆◆s ❢♦r ❉❙❚ ❏❛❣❢❡❧❞✱ ❱✉ ✾✴✶✼

❊①♣❡r✐♠❡♥ts

■♥tr♦❞✉❝t✐♦♥ ✫ ▼♦t✐✈❛t✐♦♥ ▼♦❞❡❧ ❊①♣❡r✐♠❡♥ts ❘❡s✉❧ts ❈♦♥❝❧✉s✐♦♥s ❘❡❢❡r❡♥❝❡s ❉❛t❛ • ❉❛t❛s❡t ♦❢ t❤❡ s❡❝♦♥❞ ❉✐❛❧♦❣ ❙t❛t❡ ❚r❛❝❦✐♥❣ ❈❤❛❧❧❡♥❣❡ ✭❉❙❚❈✷✮ ❬❍❡♥❞❡rs♦♥ ❡t ❛❧✳✱ ✷✵✶✹❪ ✿ ✉s❡r ✐♥t❡r❛❝t✐♦♥s ✇✐t❤ r❡st❛✉r❛♥t ❞♦♠❛✐♥ ❙❉❙ • ✶✻✶✷ tr❛✐♥✐♥❣✱ ✺✵✻ ❞❡✈❡❧♦♣♠❡♥t✱ ✶✶✶✼ t❡st ❞✐❛❧♦❣s • ❉✐❛❧♦❣ st❛t❡✿ t❤r❡❡ ❣♦❛❧s✿ ❛r❡❛ ✭✼ ✈❛❧✉❡s✮✱ ❢♦♦❞ ✭✾✸ ✈❛❧✉❡s✮✱ ♣r✐❝❡ r❛♥❣❡ ✭✺ ✈❛❧✉❡s✮❀ ✽ r❡q✉❡sts ✭❡✳❣✳ ♣❤♦♥❡ ♥✉♠❜❡r✱ ❛❞❞r❡ss ✮ • ❚r❛✐♥ ♦♥ ♠❛♥✉❛❧ tr❛♥s❝r✐♣ts ✰ ❝♥❡ts✱ t❡st ♦♥ ❝♥❡ts • ❘❡♣r❡s❡♥t t♦❦❡♥s ♦❢ s②st❡♠ ❞✐❛❧♦❣ ❛❝ts✱ ♠❛♥✉❛❧ tr❛♥s❝r✐♣ts✱ ❛♥❞ ✶✲❜❡st ❤②♣♦t❤❡s✐s ❛s t✐♠❡st❡♣s ✇✐t❤ s✐♥❣❧❡ ❤②♣♦t❤❡s✐s • ❈♥❡t ♣r❡♣r♦❝❡ss✐♥❣✿ ✶✷✺ ❤②♣♦t❤❡s❡s ✐♥ ❛✈❡r❛❣❡ ❝♥❡t✱ ❜✉t ❛✈❡r❛❣❡ ❧❡♥❣t❤ ♦❢ ❜❡st ❤②♣♦t❤❡s✐s ✐s ♦♥❧② ✹ t♦❦❡♥s → r❡♠♦✈❡ ✐♥t❡r❥❡❝t✐♦♥s ✭✉❤✱ ♦❤✱ ✳ ✳ ✳ ✮ → ♣r✉♥❡ ❤②♣♦t❤❡s❡s ✇✐t❤ ❧♦✇ s❝♦r❡s ❊♥❝♦❞✐♥❣ ❈♥❡ts ✇✐t❤ ❘◆◆s ❢♦r ❉❙❚ ❏❛❣❢❡❧❞✱ ❱✉ ✶✶✴✶✼