Chapter 11 Tree-based models Statistical Machine Translation - PowerPoint PPT Presentation

Insertion Rule S VP VP VP PP NP PRP MD VB VBG RP TO PRP DT NNS I shall be passing on to you some comments Ich werde Ihnen die entsprechenden Anmerkungen aushändigen Extracted rule: pp → x | to prp Chapter 11: Tree-Based Models 30

Non-Lexical Rule S VP VP VP PP NP PRP MD VB VBG RP TO PRP DT NNS I shall be passing on to you some comments Ich werde Ihnen die entsprechenden Anmerkungen aushändigen Extracted rule: np → x 1 x 2 | dt 1 nns 2 Chapter 11: Tree-Based Models 31

Lexical Rule with Syntactic Context S VP VP VP PP NP PRP MD VB VBG RP TO PRP DT NNS I shall be passing on to you some comments Ich werde Ihnen die entsprechenden Anmerkungen aushändigen Extracted rule: vp → x 1 x 2 aush¨ andigen | passing on pp 1 np 2 Chapter 11: Tree-Based Models 32

Lexical Rule with Syntactic Context S VP VP VP PP NP PRP MD VB VBG RP TO PRP DT NNS I shall be passing on to you some comments Ich werde Ihnen die entsprechenden Anmerkungen aushändigen Extracted rule: vp → werde x | shall be vp (ignoring internal structure) Chapter 11: Tree-Based Models 33

Non-Lexical Rule S VP VP VP PP NP PRP MD VB VBG RP TO PRP DT NNS I shall be passing on to you some comments Ich werde Ihnen die entsprechenden Anmerkungen aushändigen Extracted rule: s → x 1 x 2 | prp 1 vp 2 done — note: one rule per alignable constituent Chapter 11: Tree-Based Models 34

Unaligned Source Words S VP VP VP PP NP PRP MD VB VBG RP TO PRP DT NNS I shall be passing on to you some comments Ich werde Ihnen die entsprechenden Anmerkungen aushändigen Attach to neighboring words or higher nodes → additional rules Chapter 11: Tree-Based Models 35

Too Few Phrasal Rules? • Lexical rules will be 1-to-1 mappings (unless word alignment requires otherwise) • But: phrasal rules very beneficial in phrase-based models • Solutions – combine rules that contain a maximum number of symbols (as in hierarchical models, recall: ”Option 1”) – compose minimal rules to cover a maximum number of non-leaf nodes Chapter 11: Tree-Based Models 36

Composed Rules x 1 x 2 = np • Current rules dt 1 nns 1 die = dt entsprechenden Anmerkungen = nns some comments • Composed rule die entsprechenden Anmerkungen = np dt nns some comments (1 non-leaf node: np ) Chapter 11: Tree-Based Models 37

Composed Rules • Minimal rule: x 1 x 2 aush¨ andigen = vp prp prp pp 1 np 2 3 non-leaf nodes: passing on vp , pp , np • Composed rule: Ihnen x 1 aush¨ andigen = vp prp prp pp np 1 3 non-leaf nodes: to prp passing on vp , pp and np to you Chapter 11: Tree-Based Models 38

Relaxing Tree Constraints • Impossible rule x = md vb werde shall be • Create new non-terminal label: md+vb ⇒ New rule x = md+vb md vb werde shall be Chapter 11: Tree-Based Models 39

Zollmann Venugopal Relaxation • If span consists of two constituents , join them: x+y • If span conststs of three constituents, join them: x+y+z • If span covers constituents with the same parent x and include – every but the first child y , label as x \ y – every but the last child y , label as x/y • For all other cases, label as fail ⇒ More rules can be extracted, but number of non-terminals blows up Chapter 11: Tree-Based Models 40

Special Problem: Flat Structures • Flat structures severely limit rule extraction np dt nnp nnp nnp nnp the Israeli Prime Minister Sharon • Can only extract rules for individual words or entire phrase Chapter 11: Tree-Based Models 41

Relaxation by Tree Binarization np dt np the nnp np Israeli nnp np Prime nnp nnp Minister Sharon More rules can be extracted Left-binarization or right-binarization? Chapter 11: Tree-Based Models 42

Scoring Translation Rules • Extract all rules from corpus • Score based on counts – joint rule probability: p ( lhs , rhs f , rhs e ) – rule application probability: p ( rhs f , rhs e | lhs ) – direct translation probability: p ( rhs e | rhs f , lhs ) – noisy channel translation probability: p ( rhs f | rhs e , lhs ) – lexical translation probability: � e i ∈ rhs e p ( e i | rhs f , a ) Chapter 11: Tree-Based Models 43

Syntactic Decoding Inspired by monolingual syntactic chart parsing: During decoding of the source sentence, a chart with translations for the O ( n 2 ) spans has to be filled Sie will eine Tasse Kaffee trinken PPER VAFIN ART NN NN VVINF NP VP S Chapter 11: Tree-Based Models 44

➏ Syntax Decoding VB drink Sie will eine Tasse Kaffee trinken PPER VAFIN ART NN NN VVINF NP VP S German input sentence with tree Chapter 11: Tree-Based Models 45

➏ Syntax Decoding ➊ VB PRO she drink Sie will eine Tasse Kaffee trinken PPER VAFIN ART NN NN VVINF NP VP S Purely lexical rule: filling a span with a translation (a constituent in the chart) Chapter 11: Tree-Based Models 46

➏ Syntax Decoding ➊ ➋ NN VB PRO she coffee drink Sie will eine Tasse Kaffee trinken PPER VAFIN ART NN NN VVINF NP VP S Purely lexical rule: filling a span with a translation (a constituent in the chart) Chapter 11: Tree-Based Models 47

➏ Syntax Decoding ➊ ➋ ➌ NN VB PRO she coffee drink Sie will eine Tasse Kaffee trinken PPER VAFIN ART NN NN VVINF NP VP S Purely lexical rule: filling a span with a translation (a constituent in the chart) Chapter 11: Tree-Based Models 48

➏ Syntax Decoding ➍ NP NP PP NN DET NN IN | | | a cup of ➊ ➋ ➌ NN VB PRO she coffee drink Sie will eine Tasse Kaffee trinken PPER VAFIN ART NN NN VVINF NP VP S Complex rule: matching underlying constituent spans, and covering words Chapter 11: Tree-Based Models 49

➏ Syntax Decoding ➎ VP VP VBZ | TO VB NP wants | to ➍ NP NP PP NN DET NN IN | | | a cup of ➊ ➋ ➌ NN VB PRO she coffee drink Sie will eine Tasse Kaffee trinken PPER VAFIN ART NN NN VVINF NP VP S Complex rule with reordering Chapter 11: Tree-Based Models 50

➏ S Syntax Decoding PRO VP ➎ VP VP VBZ | TO VB NP wants | to ➍ NP NP PP NN DET NN IN | | | a cup of ➊ ➋ ➌ PRO NN VB she coffee drink Sie will eine Tasse Kaffee trinken PPER VAFIN ART NN NN VVINF NP VP S Chapter 11: Tree-Based Models 51

Bottom-Up Decoding • For each span, a stack of (partial) translations is maintained • Bottom-up: a higher stack is filled, once underlying stacks are complete Chapter 11: Tree-Based Models 52

Naive Algorithm Input: Foreign sentence f = f 1 , ...f l f , with syntax tree Output: English translation e 1: for all spans [start,end] (bottom up) do for all sequences s of hypotheses and words in span [start,end] do 2: for all rules r do 3: if rule r applies to chart sequence s then 4: create new hypothesis c 5: add hypothesis c to chart 6: end if 7: end for 8: end for 9: 10: end for 11: return English translation e from best hypothesis in span [0, l f ] Chapter 11: Tree-Based Models 53

Chart Organization Sie will eine Tasse Kaffee trinken PPER VAFIN ART NN NN VVINF NP VP S • Chart consists of cells that cover contiguous spans over the input sentence • Each cell contains a set of hypotheses 1 • Hypothesis = translation of span with target-side constituent 1 In the book, they are called chart entries. Chapter 11: Tree-Based Models 54

Dynamic Programming Applying rule creates new hypothesis NP: a cup of coffee apply rule: NP → NP Kaffee ; NP → NP+P coffee NP+P: a cup of NP: coffee eine Tasse Kaffee trinken ART NN NN VVINF Chapter 11: Tree-Based Models 55

Dynamic Programming Another hypothesis NP: a cup of coffee NP: a cup of coffee apply rule: NP → eine Tasse NP ; NP → a cup of NP NP+P: a cup of NP: coffee eine Tasse Kaffee trinken ART NN NN VVINF Both hypotheses are indistiguishable in future search → can be recombined Chapter 11: Tree-Based Models 56

Recombinable States Recombinable? NP: a cup of coffee NP: a cup of coffee NP: a mug of coffee Chapter 11: Tree-Based Models 57

Recombinable States Recombinable? NP: a cup of coffee NP: a cup of coffee NP: a mug of coffee Yes, iff max. 2-gram language model is used Chapter 11: Tree-Based Models 58

Recombinability Hypotheses have to match in • span of input words covered • output constituent label • first n –1 output words not properly scored, since they lack context • last n –1 output words still affect scoring of subsequently added words, just like in phrase-based decoding ( n is the order of the n-gram language model) Chapter 11: Tree-Based Models 59

Language Model Contexts When merging hypotheses, internal language model contexts are absorbed S (minister of Germany met with Condoleezza Rice) the foreign ... ... in Frankfurt NP VP (minister) (Condoleezza Rice) the foreign ... ... of Germany met with ... ... in Frankfurt relevant history un-scored words p LM (met | of Germany) p LM (with | Germany met) Chapter 11: Tree-Based Models 60

Stack Pruning • Number of hypotheses in each chart cell explodes ⇒ need to discard bad hypotheses e.g., keep 100 best only • Different stacks for different output constituent labels? • Cost estimates – translation model cost known – language model cost for internal words known → estimates for initial words – outside cost estimate? (how useful will be a NP covering input words 3–5 later on?) Chapter 11: Tree-Based Models 61

Naive Algorithm: Blow-ups • Many subspan sequences for all sequences s of hypotheses and words in span [start,end] • Many rules for all rules r • Checking if a rule applies not trivial rule r applies to chart sequence s ⇒ Unworkable Chapter 11: Tree-Based Models 62

Solution • Prefix tree data structure for rules • Dotted rules • Cube pruning Chapter 11: Tree-Based Models 63

Storing Rules • First concern: do they apply to span? → have to match available hypotheses and input words • Example rule np → x 1 des x 2 | np 1 of the nn 2 • Check for applicability – is there an initial sub-span that with a hypothesis with constituent label np ? – is it followed by a sub-span over the word des ? – is it followed by a final sub-span with a hypothesis with label nn ? • Sequence of relevant information np • des • nn • np 1 of the nn 2 Chapter 11: Tree-Based Models 64

Rule Applicability Check Trying to cover a span of six words with given rule NP • des • NN → NP: NP of the NN das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 65

Rule Applicability Check First: check for hypotheses with output constituent label np NP • des • NN → NP: NP of the NN das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 66

Rule Applicability Check Found np hypothesis in cell, matched first symbol of rule NP • des • NN → NP: NP of the NN NP das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 67

Rule Applicability Check Matched word des , matched second symbol of rule NP • des • NN → NP: NP of the NN NP das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 68

Rule Applicability Check Found a nn hypothesis in cell, matched last symbol of rule NP • des • NN → NP: NP of the NN NP NN das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 69

Rule Applicability Check Matched entire rule → apply to create a np hypothesis NP • des • NN → NP: NP of the NN NP NP NN das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 70

Rule Applicability Check Look up output words to create new hypothesis (note: there may be many matching underlying np and nn hypotheses) NP • des • NN → NP: NP of the NN NP: the house of the architect Frank Gehry NP: the house NN: architect Frank Gehry das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 71

Checking Rules vs. Finding Rules • What we showed: – given a rule – check if and how it can be applied • But there are too many rules (millions) to check them all • Instead: – given the underlying chart cells and input words – find which rules apply Chapter 11: Tree-Based Models 72

Prefix Tree for Rules NP: NP 1 IN 2 NP 3 NP DET NN NP: NP 1 of DET 2 NP 3 NP … ... NP: NP 1 NP: NP 1 of IN 2 NP 3 ... ... PP … des NP: NP 1 of the NN 2 NN um VP … NP: NP 2 NP 1 NP: NP 1 of NP 2 ... ... VP … ... NP: DET 1 NN 2 DET NN ... ... das Haus NP: the house ... ... ... Highlighted Rules np → np 1 det 2 nn 3 | np 1 in 2 nn 3 np → np 1 | np 1 np → np 1 des nn 2 | np 1 of the nn 2 np → np 1 des nn 2 | np 2 np 1 np → det 1 nn 2 | det 1 nn 2 np → das Haus | the house Chapter 11: Tree-Based Models 73

Dotted Rules: Key Insight • If we can apply a rule like p → A B C | x to a span • Then we could have applied a rule like q → A B | y to a sub-span with the same starting word ⇒ We can re-use rule lookup by storing A B • (dotted rule) Chapter 11: Tree-Based Models 74

Finding Applicable Rules in Prefix Tree das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 75

Covering the First Cell das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 76

Looking up Rules in the Prefix Tree das ❶ ● das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 77

Taking Note of the Dotted Rule das ❶ ● das ❶ das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 78

Checking if Dotted Rule has Translations das ❶ DET : the ● DET : that das ❶ das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 79

Applying the Translation Rules das ❶ DET : the ● DET : that DET : that DET : the das ❶ das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 80

Looking up Constituent Label in Prefix Tree das ❶ ● DET ❷ DET : that DET : the das ❶ das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 81

Add to Span’s List of Dotted Rules das ❶ ● DET ❷ DET : that DET : the DET ❷ das ❶ das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 82

Moving on to the Next Cell das ❶ ● DET ❷ DET : that DET : the DET ❷ das ❶ das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 83

Looking up Rules in the Prefix Tree das ❶ ● DET ❷ Haus ❸ DET : that DET : the DET ❷ das ❶ das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 84

Taking Note of the Dotted Rule das ❶ ● DET ❷ Haus ❸ DET : that DET : the DET ❷ house ❸ das ❶ das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 85

Checking if Dotted Rule has Translations das ❶ ● DET ❷ Haus ❸ NN : house NP : house DET : that DET : the DET ❷ house ❸ das ❶ das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 86

Applying the Translation Rules das ❶ ● DET ❷ Haus ❸ NN : house NP : house DET : that NP : house DET : the NN : house DET ❷ house ❸ das ❶ das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 87

Looking up Constituent Label in Prefix Tree das ❶ ● DET ❷ Haus ❸ NN ❹ NP ❺ DET : that NP : house DET : the NN : house DET ❷ das ❶ house ❸ das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 88

Add to Span’s List of Dotted Rules das ❶ ● DET ❷ Haus ❸ NN ❹ NP ❺ DET : that NP : house DET : the NN : house DET ❷ NN ❹ NP ❺ das ❶ house ❸ das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 89

More of the Same das ❶ ● DET ❷ Haus ❸ NN ❹ NP ❺ DET : that NP : house IN : of NP : architect DET : the NN : house DET : the NN : architect NNP : Frank NNP : Gehry DET ❷ NN ❹ NP ❺ DET ❷ NN ❹ NNP • NNP • das ❶ house ❸ des • Architekten • Frank • Gehry • das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 90

Moving on to the Next Cell das ❶ ● DET ❷ Haus ❸ NN ❹ NP ❺ DET : that NP : house IN : of NP : architect DET : the NN : house DET : the NN : architect NNP : Frank NNP : Gehry DET ❷ NN ❹ NP ❺ DET ❷ NN ❹ NNP • NNP • das ❶ house ❸ des • Architekten • Frank • Gehry • das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 91

Covering a Longer Span Cannot consume multiple words at once All rules are extensions of existing dotted rules Here: only extensions of span over das possible DET : that NP : house IN : of NP : architect DET : the NN : house DET : the NN : architect NNP : Frank NNP : Gehry DET ❷ NN ❹ NP ❺ DET ❷ NN ❹ NNP • NNP • das ❶ house ❸ des • Architekten • Frank • Gehry • das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 92

Extensions of Span over das das ❶ NN , NP , Haus? ● DET ❷ NN , NP , Haus? Haus ❸ NN ❹ NP ❺ DET : that NP : house IN : of NP : architect DET : the NN : house DET : the NN : architect NNP : Frank NNP : Gehry DET ❷ NN ❹ NP ❺ DET ❷ NN ❹ NNP • NNP • das ❶ house ❸ des • Architekten • Frank • Gehry • das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 93

Looking up Rules in the Prefix Tree das ❶ Haus ❻ ● NN ❼ DET ❷ Haus ❽ NN ❾ DET : that NP : house IN : of NP : architect DET : the NN : house DET : the NN : architect NNP : Frank NNP : Gehry DET ❷ NN ❹ NP ❺ DET ❷ NN ❹ NNP • NNP • das ❶ house ❸ des • Architekten • Frank • Gehry • das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 94

Taking Note of the Dotted Rule das ❶ Haus ❻ ● NN ❼ DET ❷ Haus ❽ NN ❾ DET NN ❾ DET Haus ❽ das NN ❼ das Haus ❻ DET : that NP : house IN : of NP : architect DET : the NN : house DET : the NN : architect NNP : Frank NNP : Gehry DET ❷ NN ❹ NP ❺ DET ❷ NN ❹ NNP • NNP • das ❶ house ❸ des • Architekten • Frank • Gehry • das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 95

Checking if Dotted Rules have Translations das ❶ Haus ❻ ● NP : the house NN ❼ NP : the NN DET ❷ Haus ❽ NP : DET house NN ❾ NP : DET NN DET NN ❾ DET Haus ❽ das NN ❼ das Haus ❻ DET : that NP : house IN : of NP : architect DET : the NN : house DET : the NN : architect NNP : Frank NNP : Gehry DET ❷ NN ❹ NP ❺ DET ❷ NN ❹ NNP • NNP • das ❶ house ❸ des • Architekten • Frank • Gehry • das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 96

Applying the Translation Rules das ❶ Haus ❻ ● NP : the house NN ❼ NP : the NN DET ❷ Haus ❽ NP : DET house NN ❾ NP : DET NN NP : that house NP : the house DET NN ❾ DET Haus ❽ das NN ❼ das Haus ❻ DET : that NP : house IN : of NP : architect DET : the NN : house DET : the NN : architect NNP : Frank NNP : Gehry DET ❷ NN ❹ NP ❺ DET ❷ NN ❹ NNP • NNP • das ❶ house ❸ des • Architekten • Frank • Gehry • das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 97

Looking up Constituent Label in Prefix Tree das ❶ Haus ❻ ● NP : the house NN ❼ NP : the NN DET ❷ Haus ❽ NP : DET house NN ❾ NP : DET NN NP ❺ NP : that house NP : the house DET NN ❾ DET Haus ❽ das NN ❼ das Haus ❻ DET : that NP : house IN : of NP : architect DET : the NN : house DET : the NN : architect NNP : Frank NNP : Gehry DET ❷ NN ❹ NP ❺ DET ❷ NN ❹ NNP • NNP • das ❶ house ❸ des • Architekten • Frank • Gehry • das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 98

Add to Span’s List of Dotted Rules das ❶ Haus ❻ ● NP : the house NN ❼ NP : the NN DET ❷ Haus ❽ NP : DET house NN ❾ NP : DET NN NP ❺ NP : that house NP : the house DET NN ❾ NP ❺ DET Haus ❽ das NN ❼ das Haus ❻ DET : that NP : house IN : of NP : architect DET : the NN : house DET : the NN : architect NNP : Frank NNP : Gehry DET ❷ NN ❹ NP ❺ DET ❷ NN ❹ NNP • NNP • das ❶ house ❸ des • Architekten • Frank • Gehry • das Haus des Architekten Frank Gehry Chapter 11: Tree-Based Models 99