Fast(er) Exact Decoding and Global Training for Transition-Based - PowerPoint PPT Presentation

Fast(er) Exact Decoding and Global Training for Transition-Based Dependency Parsing via a Minimal Feature Set Tianze Shi* Liang Huang† Lillian Lee* * Cornell † Oregon State University University

Short Version • Transition-based dependency parsing has an exponentially-large search space • 𝑃 𝑜 # exact solutions exist • In practice, however, we needed rich features ⟹ 𝑃 𝑜 % • (This work) with bi-LSTMs, now we can do 𝑃(𝑜 # ) ! • And we get state-of-the-art results 2

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Dependency Parsing root obj xcomp nsubj mark det OUTPUT She wanted to eat an apple INPUT 3

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Transition-based Dependency Parsing … Terminal … states Initial state … … … 4

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Transition-based Dependency Parsing Goal: max score( ) … … = max ∑ score( ) Terminal … states Initial state … … … 5

Background 𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Results Exact Decoding • Dynamic programming (Huang and Sagae, 2010; Kuhlmann, Gómez-Rodríguez and Satta, 2011) … since transition (sub-)sequences can be reused 6

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Exact Decoding Goal: max score( ) … = max ∑ score( ) … Terminal … states Initial state … … … Exponential to polynomial 7

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Transition Systems DP Complexity # Transitions 𝑃 𝑜 . Arc-standard 3 In our 𝑷 𝒐 𝟒 Arc-eager 4 paper Arc-hybrid 𝑷 𝒐 𝟒 3 Presentational convenience 8

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Arc-hybrid Transition System Search State … … 𝑡 5 𝑡 4 𝑡 3 𝑐 3 𝑐 4 Stack Buffer Initial State ROOT She wanted … Terminal State (Yamada and Matsumoto, 2003) ROOT (Gómez-Rodríguez et al., 2008) (Kuhlmann et al., 2011) 9

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Arc-hybrid Transition System Transitions … … … … … … 𝑡 4 𝑡 3 𝑡 3 𝑐 3 𝑐 3 reduce ↷ reduce ↶ shift … 𝑡 4 … … … … … 𝑐 3 𝑐 3 𝑡 3 𝑡 3 Same as arc-standard 10

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Arc-hybrid Transition System Transitions … … … … … … 𝑡 4 𝑡 3 𝑡 3 𝑐 3 𝑐 3 reduce ↷ reduce ↶ shift … 𝑡 4 … … … … … 𝑐 3 𝑐 3 𝑡 3 𝑡 3 Same as arc-standard 11

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Arc-hybrid Transition System Transitions … … … … … … 𝑡 4 𝑡 3 𝑡 3 𝑐 3 𝑐 3 reduce ↷ reduce ↶ shift … 𝑡 4 … … … … … 𝑐 3 𝑐 3 𝑡 3 𝑡 3 Same as arc-standard 12

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Arc-hybrid Transition System Stack Buffer ROOT She wanted to eat an apple initial She wanted to eat an apple shift ROOT shift wanted to eat an apple ROOT She reduce ↶ wanted to eat an apple ROOT She shift to eat an apple ROOT wanted shift eat an apple ROOT wanted to 13

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Arc-hybrid Transition System Stack Buffer ROOT She wanted to eat an apple initial She wanted to eat an apple shift ROOT shift wanted to eat an apple ROOT She reduce ↶ wanted to eat an apple ROOT She shift to eat an apple ROOT wanted shift eat an apple ROOT wanted to 14

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Arc-hybrid Transition System Stack Buffer ROOT She wanted to eat an apple initial She wanted to eat an apple shift ROOT shift wanted to eat an apple ROOT She reduce ↶ wanted to eat an apple ROOT She shift to eat an apple ROOT wanted shift eat an apple ROOT wanted to 15

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Arc-hybrid Transition System Stack Buffer ROOT She wanted to eat an apple initial She wanted to eat an apple shift ROOT shift wanted to eat an apple ROOT She reduce ↶ wanted to eat an apple ROOT She shift to eat an apple ROOT wanted shift eat an apple ROOT wanted to 16

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Arc-hybrid Transition System Stack Buffer ROOT She wanted to eat an apple initial She wanted to eat an apple shift ROOT shift wanted to eat an apple ROOT She reduce ↶ wanted to eat an apple ROOT She shift to eat an apple ROOT wanted shift eat an apple ROOT wanted to 17

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Arc-hybrid Transition System Stack Buffer ROOT She wanted to eat an apple initial She wanted to eat an apple shift ROOT shift wanted to eat an apple ROOT She reduce ↶ wanted to eat an apple ROOT She shift to eat an apple ROOT wanted shift eat an apple ROOT wanted to 18

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Arc-hybrid Transition System Stack Buffer reduce ↶ eat an apple ROOT wanted to shift an apple ROOT wanted eat shift apple ROOT wanted eat an reduce ↶ apple ROOT wanted eat an shift ROOT wanted eat apple 19

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Arc-hybrid Transition System Stack Buffer reduce ↶ eat an apple ROOT wanted to shift an apple ROOT wanted eat shift apple ROOT wanted eat an reduce ↶ apple ROOT wanted eat an shift ROOT wanted eat apple 20

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Arc-hybrid Transition System Stack Buffer reduce ↶ eat an apple ROOT wanted to shift an apple ROOT wanted eat shift apple ROOT wanted eat an reduce ↶ apple ROOT wanted eat an shift ROOT wanted eat apple 21

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Arc-hybrid Transition System Stack Buffer reduce ↶ eat an apple ROOT wanted to shift an apple ROOT wanted eat shift apple ROOT wanted eat an reduce ↶ apple ROOT wanted eat an shift ROOT wanted eat apple 22

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Arc-hybrid Transition System Stack Buffer reduce ↶ eat an apple ROOT wanted to shift an apple ROOT wanted eat shift apple ROOT wanted eat an reduce ↶ apple ROOT wanted eat an shift ROOT wanted eat apple 23

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Arc-hybrid Transition System Stack Buffer reduce ↷ ROOT wanted eat apple reduce ↷ ROOT wanted eat reduce ↷ ROOT (terminal) wanted 24

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Arc-hybrid Transition System Stack Buffer reduce ↷ ROOT wanted eat apple reduce ↷ ROOT wanted eat reduce ↷ ROOT (terminal) wanted 25

𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Background Results Arc-hybrid Transition System Stack Buffer reduce ↷ ROOT wanted eat apple reduce ↷ ROOT wanted eat reduce ↷ ROOT (terminal) wanted 26

Background 𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Results Deduction System for Arc-hybrid 0 1 2 3 4 5 6 ( 𝑜 ) ROOT She wanted to eat an apple Stack Buffer • Deduction Item • Goal … 𝑜 + 1 … 𝑘 0 𝑗 [𝑗,𝑘] [0, 𝑜 + 1] 27

Background 𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Results Deduction System for Arc-hybrid … … 𝑗 𝑘 [𝑗,𝑘] shift shift [𝑘, 𝑘 + 1] … … 𝑘 𝑘 + 1 𝑗 28

Background 𝑃(𝑜 # ) in theory 𝑃(𝑜 % ) in practice Back to 𝑃(𝑜 # ) Results Deduction System for Arc-hybrid 𝑗, 𝑙 [𝑙, 𝑘] reduce ↶ [? , 𝑘] … … 𝑘 𝑙 reduce ↶ … … 𝑘 ? 29 𝑙