Source side Dependency Tree Reordering Models with Subtree Movements - PowerPoint PPT Presentation

Source ‐ side Dependency Tree Reordering Models with Subtree Movements and Constraints Nguyen Bach, Qin Gao and Stephan Vogel Carnegie Mellon University 1

Overview • We introduce source ‐ side dependency tree reordering models • Inspired by lexicalized reordering model (Koehn et. al 2005) , hierarchical dependency translation (Shen et. al, 2008) and cohesive decoding (Cherry, 2008) • We model reordering events of phrases associated with source ‐ side dependency trees • Inside/Outside subtree movements efficiently capture the statistical distribution of the subtree ‐ to ‐ subtree transitions in training data • Utilize subtree movements directly at the decoding time alongside with cohesive constraints to guide the search process • Improvements are shown in English ‐ Spanish and English ‐ Iraqi tasks 2

Outline • Background & Motivations • Source ‐ side dependency tree reordering models – Modeling – Training – Decoding • Experiments & Analysis • Conclusions 3

Background of Reordering Models Put syntactic analysis of Explicitly model phrase the target language Use source language reordering distances into both modeling and syntax decoding 4

Put syntactic analysis of the target Explicitly model phrase language into both modeling and Use source language syntax reordering distances decoding Preprocessing with syntactic Distance ‐ based (Och, 2002; reordering rules (Xia and Direct model target language Koehn et.al., 2003) McCord, 2004; Collins et.al., constituents movement in 2005; Rottmann and Vogel, either constituency trees 2007; Wang et.al., 2007; Xu Lexicalized phrase (Tillmann, (Yamada and Knight, 2001; et.al. 2009) 2004; Koehn, et.al., 2005; Al ‐ Galley et.al., 2006; Zollmann Onaizan and Papineni, 2006) et.al., 2008) or dependency Use syntactical analysis to trees (Quirk, et.al., 2005) provide multiple source Hierarchical phrase (Galley sentence reordering options and Manning, 2008) through word lattices (Zhang et.al., 2007; Li et.al., Hierarchical phrase ‐ based 2007; Elming, 2008). (Chiang, 2005; Shen et. al., 2008) MaxEnt classifier (Zens and Ney, 2006; Xiong, et.al., 2006; Chang, et. al., 2009) 5

Put syntactic analysis of the target Explicitly model phrase language into both modeling and Use source language syntax reordering distances decoding Preprocessing with syntactic Distance ‐ based (Och, 2002; Direct modeling of target reordering rules (Xia and Koehn et.al., 2003) language constituents McCord, 2004; Collins et.al., Source ‐ side Dependency Tree Reordering Models movement in either 2005; Rottmann and Vogel, constituency trees (Yamada and 2007; Wang et.al., 2007; Xu Lexicalized phrase (Tillmann, Knight, 2001; Galley et.al., 2006; et.al. 2009) with Subtree Movements and Constraints 2004; Koehn, et.al., 2005; Al ‐ Zollmann et.al., 2008) or Onaizan and Papineni, 2006) dependency trees (Quirk, et.al., Use syntactical analysis to 2005) provide multiple source Hierarchical phrase (Galley sentence reordering options and Manning, 2008) through word lattices (Zhang et.al., 2007; Li et.al., Hierarchical phrase ‐ based 2007; Elming, 2008). (Chiang, 2005; Shen et. al., 2008) MaxEnt classifier (Zens and Ney, 2006; Xiong, et.al., 2006; Chang, et. al., 2009) 6

What are the differences? • Instead of using flat word structures to extract reordering events, utilize source ‐ side dependency structures – Provide more linguistic cues for reordering events • Instead of using pre ‐ defined reordering patterns, learn reordering feature distributions from training data – Capture reordering events from real data • Instead of preprocessing the data, discriminatively train the reordering model via MERT – Tighter integration with the decoder 7

Cohesive Decoding • A cohesive decoding (Cherry, 08; Bach et. al., 09) is forcing the cohesive constraint: – When the decoder begins translation any part of a source subtree, it must cover all words under that subtree before it can translate anything outside. • Source ‐ side dependency tree reordering models – Efficiently capture the statistical distribution of the subtree ‐ to ‐ subtree transitions in training data. – Directly utilize it at the decoding time to guide the search process. 8

Outline • Background of Reordering Models • Source ‐ side dependency tree reordering models – Modeling – Training – Decoding • Experiments & Analysis • Conclusions 9

Lexicalized Reordering Models (Tillmann, 2004; Koehn, et.al., 2005; Al ‐ Onaizan & Papineni, 2006) n ∏ = ( | , ) ( | , , , ) p O e f p o e f a a − 1 i i a i i i = 1 i where is the input sentence; f = ( ,..., ) is the target language phrases; e e e 1 n = ( ) is phrase alignments ; a a ,...,a 1 n is a source phrase which has a translate d phrase defined by an alignment ; f e a a i i i is orientatio n phrase sequence; each has a value over 3 possibles ( ); O o M, S,D i 10

16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 11

16 15 14 13 12 11 10 9 8 7 6 5 4 quisiera ● 3 ● 2 tanto ● 1 lo ● 0 Por 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 12

16 15 14 13 12 11 10 9 8 7 6 5 ● ● 4 pedirle quisiera ● 3 ● 2 Discontinuous tanto ● 1 lo ● 0 Por 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 13

16 15 14 13 12 11 10 9 8 7 Swap 6 ● ● 5 nuevamente ● ● 4 pedirle quisiera ● 3 ● 2 Discontinuous tanto ● 1 lo ● 0 Por 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 14

16 15 14 13 12 11 Discontinuous ● 10 que ● 9 de ● 8 encargue 7 se Swap 6 que ● ● 5 nuevamente ● ● 4 Pedirle quisiera ● 3 ● 2 Discontinuous tanto ● 1 lo ● 0 Por 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 15

● ● 16 neerlandés ● 15 canal ● 14 Un 13 también Monotone ● 12 ver ● ● 11 podamos Discontinuous ● 10 que ● 9 de ● 8 encargue 7 se Swap 6 que ● ● 5 nuevamente ● ● 4 Pedirle quisiera ● 3 ● 2 Discontinuous tanto ● 1 lo ● 0 Por 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 16

Pros & Cons of Lexicalized Reordering Models • Pros – intuitively model flat word movements – well ‐ defined for phrase ‐ based framework • Cons – No linguistics structures – Need alignment matrix to determine movements 17

Completed/Open subtrees g A completed subtree b d a c e f All words under a node have been translated then we call a completed subtree 18

Completed/Open subtrees g An open subtree b d a c e f A subtree that has begun translation but not yet complete, an open subtree 19

Inside/Outside subtree movements A structure is moving “c” is moving inside a subtree inside a subtree if it rooted at “b” g helps the subtree to be completed or less open b d a c e f Inside 20

Inside/Outside subtree movements Outside g A structure is moving outside a subtree if it leaves the subtree to be open b d a c e f “d e” is moving outside a subtree rooted at “b” 21

Source ‐ side Dependency Tree (SDT) Reordering Models n ∏ = ( | , ) ( | , , , , ) p D e f p d e f a s s − 1 i i a i i i i = 1 i where f is the input sentence; = e ( e ,..., e ) is the target language phrases; 1 n = ( ) is phrase alignments ; a a ,...,a 1 n f is a source phrase which has a translate d phrase e defined by an alignment a ; a i i i and are dependency structures of source phrases and ; s s f f 1 a a i i- i i - 1 represents the sequence of syntactic phrase movements D over source dependency tree; = each ; d {I, O} i 22

would ask I therefore once get more ensure that we you to channel as a Dutch well neerlandés canal un también ver podamos que de encargue se que nuevamente Pedirle quisiera tanto 23 lo Por

would ask I therefore once get more ensure that we you to channel as a Dutch well neerlandés canal un también ver podamos que de encargue se que nuevamente Pedirle ● quisiera ● tanto 24 ● lo ● Por

would ask I therefore once get more ensure that we you to channel as Inside a Dutch well neerlandés canal un también ver podamos que de encargue se que nuevamente ● ● Pedirle ● quisiera ● tanto 25 ● Discontinuous lo ● Por

would ask I therefore once get more ensure Outside that we you to channel as Inside a Dutch well neerlandés canal un también ver podamos que de encargue Swap se que ● ● nuevamente ● ● Pedirle ● quisiera ● tanto 26 ● Discontinuous lo ● Por

would Inside ask I therefore once get more ensure Outside that we you to channel as Inside a Dutch well neerlandés canal un también ver podamos Discontinuous ● que ● de ● encargue Swap se que ● ● nuevamente ● ● Pedirle ● quisiera ● tanto 27 ● Discontinuous lo ● Por