Forest Rescoring Faster Decoding with Integrated Language Models - - PowerPoint PPT Presentation

Forest Rescoring

Faster Decoding with

Integrated Language Models

Liang Huang David Chiang

ACL 2007, Praha, Česká republika

Huang and Chiang Forest Rescoring

Statistical Machine Translation

2

(Knight and Koehn, 2003)

translation model (TM) competency language model (LM) fluency

Spanish Broken English English Spanish/English Bilingual Text English Text Statistical Analysis Statistical Analysis Que hambre tengo yo What hunger have I Hungry I am so Have I that hunger I am so hungry How hunger have I ... I am so hungry

Huang and Chiang Forest Rescoring

Statistical Machine Translation

2

(Knight and Koehn, 2003)

translation model (TM) competency language model (LM) fluency

Spanish Broken English English Spanish/English Bilingual Text English Text Statistical Analysis Statistical Analysis Que hambre tengo yo What hunger have I Hungry I am so Have I that hunger I am so hungry How hunger have I ... I am so hungry

n-best rescoring

Huang and Chiang Forest Rescoring

Statistical Machine Translation

3

translation model (TM) competency language model (LM) fluency

Spanish Broken English English Spanish/English Bilingual Text English Text Statistical Analysis Statistical Analysis

Huang and Chiang Forest Rescoring

Statistical Machine Translation

3

translation model (TM) competency language model (LM) fluency

Spanish Broken English English Spanish/English Bilingual Text English Text Statistical Analysis Statistical Analysis

computationally challenging! ☹

Que hambre tengo yo I am so hungry

decoder (LM-integrated)

integrated decoder

Huang and Chiang Forest Rescoring

Statistical Machine Translation

3

translation model (TM) competency language model (LM) fluency

Spanish Broken English English Spanish/English Bilingual Text English Text Statistical Analysis Statistical Analysis

phrase-based TM syntax-based

n-gram LM

computationally challenging! ☹

Que hambre tengo yo I am so hungry

decoder (LM-integrated)

integrated decoder

Huang and Chiang Forest Rescoring

Forest Rescoring

4

translation model (TM) competency language model (LM) fluency

Spanish Broken English English Spanish/English Bilingual Text English Text Statistical Analysis Statistical Analysis

phrase-based TM syntax-based

n-gram LM

Que hambre tengo yo I am so hungry

decoder (LM-integrated)

integrated decoder packed forest

computationally challenging! ☹

Huang and Chiang Forest Rescoring

Forest Rescoring

4

translation model (TM) competency language model (LM) fluency

Spanish Broken English English Spanish/English Bilingual Text English Text Statistical Analysis Statistical Analysis

phrase-based TM syntax-based

n-gram LM

Que hambre tengo yo I am so hungry

decoder (LM-integrated)

integrated decoder packed forest

computationally challenging! ☹

• n-the-fly rescoring

Huang and Chiang Forest Rescoring

Forest Rescoring

4

translation model (TM) competency language model (LM) fluency

Spanish Broken English English Spanish/English Bilingual Text English Text Statistical Analysis Statistical Analysis

phrase-based TM syntax-based

n-gram LM

significant speed-up: 10~30 times faster! ☺

Que hambre tengo yo I am so hungry

decoder (LM-integrated)

integrated decoder packed forest forest rescorer

• n-the-fly rescoring

The Forest Framework

unifying phrase- and syntax-based decoding

Huang and Chiang Forest Rescoring

Phrase-based Decoding

6

yu Shalong juxing le huitan

与 沙龙 举行 了 会谈

held a talk with Sharon

_ _●

• held a talk

held a talk with Sharon

_ _ _ _ _

... ... ...

• ...

_ _●

• held a talk

source-side: coverage vector target-side: grow hypotheses strictly left-to-right

Huang and Chiang Forest Rescoring

Syntax-based Translation

7

• synchronous context-free grammars (SCFGs)
• context-free grammar in two dimensions
• generating pairs of strings/trees simultaneously
• co-indexed nonterminal further rewritten as a unit

VP PP yu Shalong VP juxing le huitan VP VP held a meeting PP with Sharon

VP → PP(1) VP(2), VP(2) PP(1) VP → juxing le huitan, held a meeting PP → yu Shalong, with Sharon

Huang and Chiang Forest Rescoring

Translation as Parsing

8

• translation with SCFGs => monolingual parsing
• parse the source input with the source projection
• build the corresponding target sub-strings in parallel

PP1, 3 VP3, 6 VP1, 6

yu Shalong juxing le huitan

VP → PP(1) VP(2), VP(2) PP(1) VP → juxing le huitan, held a meeting PP → yu Shalong, with Sharon

Huang and Chiang Forest Rescoring

Translation as Parsing

8

• translation with SCFGs => monolingual parsing
• parse the source input with the source projection
• build the corresponding target sub-strings in parallel

PP1, 3 VP3, 6 VP1, 6

yu Shalong juxing le huitan

VP → PP(1) VP(2), VP(2) PP(1) VP → juxing le huitan, held a meeting PP → yu Shalong, with Sharon

Huang and Chiang Forest Rescoring

Translation as Parsing

8

• translation with SCFGs => monolingual parsing
• parse the source input with the source projection
• build the corresponding target sub-strings in parallel

PP1, 3 VP3, 6 VP1, 6

yu Shalong juxing le huitan

with Sharon held a talk held a talk with Sharon

VP → PP(1) VP(2), VP(2) PP(1) VP → juxing le huitan, held a meeting PP → yu Shalong, with Sharon

Huang and Chiang Forest Rescoring

Packed Forest

• a compact representation of all translations
• has a structure of hypergraph (graph is a special case)

9

_ _●

• _ _ _ _ _
• PP1, 3

VP3, 6 VP1, 6

phrase-based: graph syntax-based: hypergraph

Huang and Chiang Forest Rescoring

Packed Forest

• a compact representation of all translations
• has a structure of hypergraph (graph is a special case)

9

_ _●

• _ _ _ _ _
• nodes

(hyper-)edges

PP1, 3 VP3, 6 VP1, 6

phrase-based: graph syntax-based: hypergraph

Huang and Chiang Forest Rescoring

Adding a Bigram Model

10

PP1, 3 VP3, 6 VP1, 6

_ _●

• ... talk

_ _ _ _ _

• ●●●● ... Sharon

_ _●

• ... talks

_ _●

• ... meeting
• ●●●● ... Shalong

with ... Sharon along ... Sharon with ... Shalong held ... talk held ... meeting hold ... talks

+LM items

Huang and Chiang Forest Rescoring

Adding a Bigram Model

10

PP1, 3 VP3, 6 VP1, 6

_ _●

• ... talk

_ _ _ _ _

• ●●●● ... Sharon

_ _●

• ... talks

_ _●

• ... meeting
• ●●●● ... Shalong

with ... Sharon along ... Sharon with ... Shalong held ... talk held ... meeting hold ... talks

+LM items

with Sharon

Huang and Chiang Forest Rescoring

Adding a Bigram Model

10

PP1, 3 VP3, 6 VP1, 6

_ _●

• ... talk

_ _ _ _ _

• ●●●● ... Sharon

_ _●

• ... talks

_ _●

• ... meeting
• ●●●● ... Shalong

with ... Sharon along ... Sharon with ... Shalong held ... talk held ... meeting hold ... talks

+LM items

with Sharon

bigram

Huang and Chiang Forest Rescoring

Adding a Bigram Model

10

PP1, 3 VP3, 6 VP1, 6

_ _●

• ... talk

_ _ _ _ _

• ●●●● ... Sharon

_ _●

• ... talks

_ _●

• ... meeting
• ●●●● ... Shalong

with ... Sharon along ... Sharon with ... Shalong held ... talk held ... meeting hold ... talks

+LM items

with Sharon

bigram

held ... talk with ... Sharon

Huang and Chiang Forest Rescoring

Adding a Bigram Model

10

PP1, 3 VP3, 6 VP1, 6

_ _●

• ... talk

_ _ _ _ _

• ●●●● ... Sharon

_ _●

• ... talks

_ _●

• ... meeting
• ●●●● ... Shalong

with ... Sharon along ... Sharon with ... Shalong held ... talk held ... meeting hold ... talks

+LM items

with Sharon

bigram

held ... talk with ... Sharon

bigram

Huang and Chiang Forest Rescoring

Adding a Bigram Model

10

PP1, 3 VP3, 6 VP1, 6

_ _●

• ... talk

_ _ _ _ _

• ●●●● ... Sharon

_ _●

• ... talks

_ _●

• ... meeting
• ●●●● ... Shalong

with ... Sharon along ... Sharon with ... Shalong held ... talk held ... meeting hold ... talks held ... Sharon

+LM items

with Sharon

bigram

held ... talk with ... Sharon

bigram

Huang and Chiang Forest Rescoring

Adding a Bigram Model

10

PP1, 3 VP3, 6 VP1, 6

_ _●

• ... talk

_ _ _ _ _

• ●●●● ... Sharon

_ _●

• ... talks

_ _●

• ... meeting
• ●●●● ... Shalong

with ... Sharon along ... Sharon with ... Shalong held ... talk held ... meeting hold ... talks held ... Sharon held ... Shalong hold ... Sharon hold ... Shalong

+LM items

with Sharon

bigram

held ... talk with ... Sharon

bigram

Huang and Chiang Forest Rescoring

Conventional Beam Search

• beam search: only keep top-k +LM items at each node
• but there are many ways to derive each node
• can we avoid enumerating all combinations?
• best-first enumeration?

11

VP1, 6

hyperedge

PP1, 3 VP3, 6 PP1, 4 VP4, 6 NP1, 4 VP4, 6

1.0 1.1 2.5 2.3 4.6 7.2

Huang and Chiang Forest Rescoring

Cube Pruning

12

(VP held meeting

3,6

) (VP held talk

3,6

) (VP hold conference

3,6

)

( P P

w i t h

• S

h a r

• n

1 , 3

)

( P P

a l

• n

g

• S

h a r

• n

1 , 3

) ( P P

w i t h

• S

h a l

• n

g 1 , 3

)

PP1, 3 VP3, 6 VP1, 6

monotonic grid?

1.0 3.0 8.0 1.0

2.0 4.0 9.0

1.1

2.1 4.1 9.1

3.5

4.5

6.5

11.5

Huang and Chiang Forest Rescoring

Cube Pruning

13

(VP held meeting

3,6

) (VP held talk

3,6

) (VP hold conference

3,6

)

( P P

w i t h

• S

h a r

• n

1 , 3

)

( P P

a l

• n

g

• S

h a r

• n

1 , 3

) ( P P

w i t h

• S

h a l

• n

g 1 , 3

)

PP1, 3 VP3, 6 VP1, 6

non-monotonic grid due to LM combo costs

1.0 3.0 8.0 1.0 2.0 + 0.5 4.0 + 5.0 9.0 + 0.5 1.1 2.1 + 0.3 4.1 + 5.4 9.1 + 0.3 3.5 4.5 + 0.6 6.5 +10.5 11.5 + 0.6

Huang and Chiang Forest Rescoring

Cube Pruning

13

(VP held meeting

3,6

) (VP held talk

3,6

) (VP hold conference

3,6

)

( P P

w i t h

• S

h a r

• n

1 , 3

)

( P P

a l

• n

g

• S

h a r

• n

1 , 3

) ( P P

w i t h

• S

h a l

• n

g 1 , 3

)

PP1, 3 VP3, 6 VP1, 6

non-monotonic grid due to LM combo costs

1.0 3.0 8.0 1.0 2.0 + 0.5 4.0 + 5.0 9.0 + 0.5 1.1 2.1 + 0.3 4.1 + 5.4 9.1 + 0.3 3.5 4.5 + 0.6 6.5 +10.5 11.5 + 0.6

bigram (meeting, with)

Huang and Chiang Forest Rescoring

Cube Pruning

14

1.0 3.0 8.0 1.0

2.5 9.0 9.5

1.1

2.4 9.5 9.4

3.5

5.1 17.0 12.1

(VP held meeting

3,6

) (VP held talk

3,6

) (VP hold conference

3,6

)

( P P

w i t h

• S

h a r

• n

1 , 3

)

( P P

a l

• n

g

• S

h a r

• n

1 , 3

) ( P P

w i t h

• S

h a l

• n

g 1 , 3

)

PP1, 3 VP3, 6 VP1, 6

non-monotonic grid due to LM combo costs

Huang and Chiang Forest Rescoring

Cube Pruning

15

1.0 3.0 8.0 1.0

2.5 9.0 9.5

1.1

2.4 9.5 9.4

3.5

5.1 17.0 12.1

(VP held meeting

3,6

) (VP held talk

3,6

) (VP hold conference

3,6

)

( P P

w i t h

• S

h a r

• n

1 , 3

)

( P P

a l

• n

g

• S

h a r

• n

1 , 3

) ( P P

w i t h

• S

h a l

• n

g 1 , 3

)

k-best parsing

(Huang and Chiang, 2005)

• a priority queue of candidates
• extract the best candidate

Huang and Chiang Forest Rescoring

Cube Pruning

16

(VP held meeting

3,6

) (VP held talk

3,6

) (VP hold conference

3,6

)

( P P

w i t h

• S

h a r

• n

1 , 3

)

( P P

a l

• n

g

• S

h a r

• n

1 , 3

) ( P P

w i t h

• S

h a l

• n

g 1 , 3

)

• a priority queue of candidates
• extract the best candidate
• push the two successors

1.0 3.0 8.0 1.0

2.5 9.0 9.5

1.1

2.4 9.5 9.4

3.5

5.1 17.0 12.1

k-best parsing

(Huang and Chiang, 2005)

Huang and Chiang Forest Rescoring

Cube Pruning

17

(VP held meeting

3,6

) (VP held talk

3,6

) (VP hold conference

3,6

)

( P P

w i t h

• S

h a r

• n

1 , 3

)

( P P

a l

• n

g

• S

h a r

• n

1 , 3

) ( P P

w i t h

• S

h a l

• n

g 1 , 3

)

1.0 3.0 8.0 1.0

2.5 9.0 9.5

1.1

2.4 9.5 9.4

3.5

5.1 17.0 12.1

• a priority queue of candidates
• extract the best candidate
• push the two successors

k-best parsing

(Huang and Chiang, 2005)

Huang and Chiang Forest Rescoring

Cube Pruning

18

(VP held meeting

3,6

) (VP held talk

3,6

) (VP hold conference

3,6

)

( P P

w i t h

• S

h a r

• n

1 , 3

)

( P P

a l

• n

g

• S

h a r

• n

1 , 3

) ( P P

w i t h

• S

h a l

• n

g 1 , 3

)

1.0 3.0 8.0 1.0

2.5 9.0 9.5

1.1

2.4 9.5 9.4

3.5

5.1 17.0 12.1

items are popped out-of-order solution: keep a buffer of pop-ups

2.5 2.4 5.1

Huang and Chiang Forest Rescoring

Cube Pruning

18

(VP held meeting

3,6

) (VP held talk

3,6

) (VP hold conference

3,6

)

( P P

w i t h

• S

h a r

• n

1 , 3

)

( P P

a l

• n

g

• S

h a r

• n

1 , 3

) ( P P

w i t h

• S

h a l

• n

g 1 , 3

)

1.0 3.0 8.0 1.0

2.5 9.0 9.5

1.1

2.4 9.5 9.4

3.5

5.1 17.0 12.1

finally re-sort the buffer and return inorder:

2.4 2.5 5.1

items are popped out-of-order solution: keep a buffer of pop-ups

2.5 2.4 5.1

Huang and Chiang Forest Rescoring

Across Hyperedges

19

VP

process all hyperedges simultaneously! significant savings of computation

PP1, 3 VP3, 6 PP1, 4 VP4, 6 NP1, 4 VP4, 6

k-best parsing

(Huang and Chiang, 2005)

hyperedge

Huang and Chiang Forest Rescoring

Across Hyperedges

19

VP

process all hyperedges simultaneously! significant savings of computation

PP1, 3 VP3, 6 PP1, 4 VP4, 6 NP1, 4 VP4, 6

k-best parsing

(Huang and Chiang, 2005)

hyperedge

• n-the-fly rescoring at each node,

instead of only at the root node

Huang and Chiang Forest Rescoring

Cube Growing

• an even faster variant of cube pruning
• motivation
• why do we have a fixed beam of size k at each node?
• why don’t we on-the-fly figure out the minimum k?
• cube growing uses
• lazy k-best parsing (Huang and Chiang, 2005, Algorithm 3)
• on-demand computation
• but harder to implement

20

Syntax-based Experiments

Huang and Chiang Forest Rescoring

Tree-to-String System

• syntax-directed, English to Chinese (Huang, Knight, Joshi, 2006)
• first parse input, and then recursively transfer

22

synchronous tree- substitution grammars (STSG)

(Galley et al., 2004; Eisner, 2003)

tested on 140 sentences slightly better BLEU scores than Pharaoh

VP VBD was VP-C VP VBN shot PP TO to NP-C NN death PP IN by NP-C DT the NN police

search space still a hypergraph

Huang and Chiang Forest Rescoring

Tree-to-String System

• syntax-directed, English to Chinese (Huang, Knight, Joshi, 2006)
• first parse input, and then recursively transfer

22

synchronous tree- substitution grammars (STSG)

(Galley et al., 2004; Eisner, 2003)

tested on 140 sentences slightly better BLEU scores than Pharaoh

VP VBD was VP-C VP VBN shot PP TO to NP-C NN death PP IN by NP-C DT the NN police !"""#$%&"""'$

bei

VP VBD was VP-C VP VBN shot PP TO to NP-C NN death PP IN by NP-C DT the NN police

search space still a hypergraph

Huang and Chiang Forest Rescoring

Speed vs. Search Quality

23

speed ++

quality ++

( - log Prob )

Huang and Chiang Forest Rescoring

Speed vs. Search Quality

23

speed ++

quality ++

10 times faster

( - log Prob )

Huang and Chiang Forest Rescoring

Speed vs. Search Quality

23

speed ++

quality ++

10 times faster

( - log Prob )

same parameters

1 1 1

Huang and Chiang Forest Rescoring

Speed vs. Search Quality

23

speed ++

quality ++

10 times faster

( - log Prob )

same parameters

1 1 1

2 2 2

Huang and Chiang Forest Rescoring

Speed vs. Search Quality

23

speed ++

quality ++

10 times faster

( - log Prob )

same parameters

1 1 1

2 2 2

3 3 3

Huang and Chiang Forest Rescoring

Speed vs. Translation Accuracy

24

speed ++

quality++

Cube-Pruning for Phrase-based Decoding

Huang and Chiang Forest Rescoring

Syntax vs. Phrase-based

26

VP PP1, 3 VP3, 6 PP1, 4 VP4, 6 NP1, 4 VP4, 6

_ _●

• _ _ _
• _

... talk ... meeting ... talks ... Sharon ... Shalong ... minister ... held ... hold ... did

with Sharon

held a talk

a talk

Huang and Chiang Forest Rescoring

Syntax vs. Phrase-based

26

VP PP1, 3 VP3, 6 PP1, 4 VP4, 6 NP1, 4 VP4, 6

_ _●

• _ _ _
• _

... talk ... meeting ... talks ... Sharon ... Shalong ... minister ... held ... hold ... did

with Sharon

held a talk

a talk

Huang and Chiang Forest Rescoring

a n d S h a r

• n

w i t h A r i e l S h a r

• n

w i t h S h a r

• n

Alternative Phrase-Pairs

27

_ _●

• _ _ _
• _

... talk ... meeting ... talks

held a meeting held a talk hold a reunion a meeting a talk conference

... Sharon ... Shalong ... minister ... held ... hold ... did

grouping into hyperedge bundles

Huang and Chiang Forest Rescoring

a n d S h a r

• n

w i t h A r i e l S h a r

• n

w i t h S h a r

• n

Alternative Phrase-Pairs

27

_ _●

• _ _ _
• _

... talk ... meeting ... talks

held a meeting held a talk hold a reunion a meeting a talk conference

... Sharon ... Shalong ... minister ... held ... hold ... did

grouping into hyperedge bundles Pharaoh would explore all cells

Huang and Chiang Forest Rescoring

a n d S h a r

• n

w i t h A r i e l S h a r

• n

w i t h S h a r

• n

Cube Pruning

28

_ _●

• _ _ _
• _

... talk ... meeting ... talks

held a meeting held a talk hold a reunion a meeting a talk conference

... Sharon ... Shalong ... minister ... held ... hold ... did

but we explore the grids in a best-first fashion

Huang and Chiang Forest Rescoring

a n d S h a r

• n

w i t h A r i e l S h a r

• n

w i t h S h a r

• n

Cube Pruning

28

_ _●

• _ _ _
• _

... talk ... meeting ... talks

held a meeting held a talk hold a reunion a meeting a talk conference

... Sharon ... Shalong ... minister ... held ... hold ... did

but we explore the grids in a best-first fashion in practice we use per-bin pruning as in Pharaoh

Huang and Chiang Forest Rescoring

In Practice: per-bin Pruning

29

Pharaoh

Huang and Chiang Forest Rescoring

In Practice: per-bin Pruning

29

Pharaoh Cube Pruning

Huang and Chiang Forest Rescoring

In Practice: per-bin Pruning

29

Cube Pruning

Huang and Chiang Forest Rescoring

In Practice: per-bin Pruning

29

Cube Pruning

close up

Huang and Chiang Forest Rescoring

In Practice: per-bin Pruning

29

Cube Pruning hyperedge bundles

close up

Huang and Chiang Forest Rescoring

In Practice: per-bin Pruning

29

Cube Pruning hyperedge bundles

close up

Huang and Chiang Forest Rescoring

In Practice: per-bin Pruning

29

Cube Pruning

2.5 8.3 2.4

hyperedge bundles

close up

Huang and Chiang Forest Rescoring

In Practice: per-bin Pruning

29

Cube Pruning

2.5 8.3 2.4

hyperedge bundles

close up

Huang and Chiang Forest Rescoring

Speed vs. Search Quality

30

speed ++

quality ++

tested on our faithful clone of Pharaoh ( - log Prob )

Huang and Chiang Forest Rescoring

Speed vs. Search Quality

30

speed ++

quality ++

32 times faster

tested on our faithful clone of Pharaoh ( - log Prob )

Huang and Chiang Forest Rescoring

Speed vs. Search Quality

30

speed ++

quality ++

32 times faster

tested on our faithful clone of Pharaoh ( - log Prob )

same parameters

Huang and Chiang Forest Rescoring

Speed vs. Translation Accuracy

31

speed ++

quality++

Huang and Chiang Forest Rescoring

Speed vs. Translation Accuracy

31

speed ++

quality++

~100 times faster

Huang and Chiang Forest Rescoring

Conclusions

• forest-rescoring: cube pruning and cube growing
• on-the-fly rescoring using k-best parsing
• applicable to both phrase- and syntax-based systems
• significant speed-up against conventional beam search
• general technique for reducing search spaces
• effectiveness depends on scale of non-monotonicity
• future work
• forest-reranking: parsing with non-local features

32

Thanks!

www.cis.upenn.edu/~lhuang3/cubit/

try out Cubit

a cube pruning decoder for phrase-based translation