Attention Shifting for Parsing Speech Keith Hall and Mark Johnson - - PowerPoint PPT Presentation

Attention Shifting for Parsing Speech

Keith Hall and Mark Johnson

Brown University

ACL 2004

July 22, 2004

Attention Shifting

• Iterative best-first word-lattice parsing algorithm
• Posits a complete syntactic analyses for each path of a word-lattice
• Goals of Attention Shifting

– Improve accuracy of best-first parsing on word-lattices

(Oracle Word Error Rate)

– Improve efficiency of word-lattice parsing

(Number of parser operations)

– Improve syntactic language modeling based on multi-stage parsing

(Word Error Rate)

• Inspired by edge demeriting for efficient parsing

Blaheta & Charniak demeriting (ACL99)

7/22/2004 ACL04: Attention Shifting for Parsing Speech 1

Outline

• Syntactic language modeling
• Word-lattice parsing
• Multi-stage best-first parsing

7/22/2004 ACL04: Attention Shifting for Parsing Speech 2

Noisy Channel

Langauge Source

Noisy Channel

Language Output

P(A, W) = P(A|W)P(W)

Language Model Noise Model

• Speech recognition: Noise model = Acoustic model

arg max

W P(W|A) = arg max W P(A, W)

7/22/2004 ACL04: Attention Shifting for Parsing Speech 3

Syntactic Language Modeling

w1, ..., wi, ..., wn1 ... Language Model w1, ..., wi, ..., wn2 w1, ..., wi, ..., wn3 w1, ..., wi, ..., wn4 w1, ..., wi, ..., wnm

• 1, ..., oi, ..., on

8 2 3 5 1 6 4 7 10 9 the/0 man/0 is/0 duh/1.385 man/0 is/0 surely/0 early/0 mans/1.385 man's/1.385 surly/0 surly/0.692 early/0 early/0

n-best list extractor

• Adding syntactic information to context (conditioning information)

P (W ) = Qk

1 P (wi|π(wk, . . . , w1))

• n-best reranking

– Select n-best strings using some model (trigram) – Process each string independently – Select string with highest P (A, W )

• Charniak (ACL01), Chelba & Jelinek (CS&L00,ACL02), Roark (CL01)

7/22/2004 ACL04: Attention Shifting for Parsing Speech 4

Parsing word-lattice

1 <s>/0 2 the/0 3 duh/1.223 4 man/0 5 mans/0.510 6 man’s/0.916 9 man/0 is/0 7 early/0 surly/0.694 early/0 10 is/0 8/1.307 </s>/0 surely/0 NN VB VBZ VP VB DT JJ

• Compress lattice with Weighted FSM determinization and minimization

(Mohri, Pereira, & Riley CS&L02)

• Use compressed word-lattice graph as the parse chart
• Structure sharing due to compressed lattice

VP → NN VB covers string man is VP → VBZ covers string mans

7/22/2004 ACL04: Attention Shifting for Parsing Speech 5

Word-lattice example

• I WOULD NOT SUGGEST ANYONE MAKE A DECISION ON WHO TO VOTE FOR

BASED ON A STUDY LIKE THIS (160 arcs, 72 nodes)

• ne/0.5

• n/0

• n/10.41

• n/10.99

• n/11.01

• f/63.72

• n/11.00

• n/120.3

• n/105.9

• n/106

• n/11.00

• n/0

• compressed NIST ’93 HUB-1 lattices

– average of 800 arcs/lattice (max 15000 arcs) – average of 100 nodes/lattice (max 500 nodes)

7/22/2004 ACL04: Attention Shifting for Parsing Speech 6

Best-first Word-lattice Parsing

1 <s>/0 2 the/0 3 duh/1.223 4 man/0 5 mans/0.510 6 man’s/0.916 9 man/0 is/0 7 early/0 surly/0.694 early/0 10 is/0 8/1.307 </s>/0 surely/0 Agenda (Priority Queue)

NP(0,4) 0.567 NN(3,9) 0.550

Compute FOM Grammar

VB VB VP VB DT NN

• Bottom-up best-first PCFG parser
• Stack-based search technique based on figure-of-merit
• Attempts to work on “likely” parts of the chart
• Ideal figure-of-merit: P (edge) = inside(edge) ∗ outside(edge)

details in (Hall & Johnson ASRU03)

7/22/2004 ACL04: Attention Shifting for Parsing Speech 7

Word-lattice Parsing

Compresed lattice Outside HMM Best-first PCFG Parser Syntactic Category Lattice Inside-Outside Prunning Local Trees Lexicalized Syntactic Language Model (Charniak Parser) Word String from Optimal Parse Word-lattice First Stage Second Stage

• First stage: best-first bottom-up PCFG parser
• Second stage: Charniak Parser Language Model

(Charniak ACL01)

• Parsing from lattice allows structure sharing
• Combines search for candidate lattice paths

with search for candidate parses

7/22/2004 ACL04: Attention Shifting for Parsing Speech 8

Multi-stage Deficiency

• First-stage PCFG parser selects parses for a subset of word-lattice paths
• Lexicalized syntactic analysis not performed on all of the word-lattice
• Covering entire word-lattice requires excessive over-parsing

– 100X over-parsing produces forests too large for lexical-parser – additional pruning required, resulting in loss of lattice-paths

• Attention shifting algorithm addresses the coverage problem

7/22/2004 ACL04: Attention Shifting for Parsing Speech 9

Attention Shifting

Identify Unused Words Clear Agenda/ Add Edges for Unused Words Is Agenda Empty? no Continue Multi-stage Parsing yes PCFG Word-lattice Parser

• Iterative reparsing
• 1. Perform best-first PCFG parsing (over-parse as

with normal best-first parsing)

• 2. Identify words not covered by a complete parse

(unused word has 0 outside probability)

• 3. Reset parse Agenda to contain unused words
• 4. If Agenda = ∅ repeat
• Prune chart using inside/outside pruning
• At most |A| iterations (|A| = number of arcs)
• Forces coverage of word-lattice

7/22/2004 ACL04: Attention Shifting for Parsing Speech 10

Experimental Setup

• PCFG Parser trained on Penn WSJ Treebank f2-21,24

(speech-normalization via Roark’s normalization)

– Generated at most 30k local-trees for second-stage parser

• Lexicalized parser: Charniak’s Language Model Parser

(Charniak ACL01)

– trained on parsed BLLIP99 corpus (30 million words of WSJ) – BLLIP99 parsed using Charniak string parser trained on Penn WSJ

7/22/2004 ACL04: Attention Shifting for Parsing Speech 11

Evaluation

• Evaluation set: NIST ’93 HUB-1

– 213 utterances – Professional readers reading WSJ text

• Word-lattices evaluated on:

– n-best word-lattices using Chelba A∗ decoder (50-best paths) – compressed acoustic word-lattices

• Metrics

– Word-lattice accuracy (first-stage parser): Oracle Word Error Rate – Word-string accuracy (multi-stage parser): Word Error Rate – Efficiency: number of parser agenda operations

7/22/2004 ACL04: Attention Shifting for Parsing Speech 12

Results: n-best word-lattices

• Charniak parser run on each of the n-best strings (reranking)

(4X over-parsing)

• n-best word-lattice: pruned acoustic word-lattices containing only n-best

word-strings

• Oracle WER of n-best lattices: 7.75

Model # edge pops Oracle WER WER n–best (Charniak) 2.5 million 7.75 11.8 100x LatParse 3.4 million 8.18 12.0 10x AttShift 564,895 7.78 11.9

7/22/2004 ACL04: Attention Shifting for Parsing Speech 13

Results: Acoustic word-lattices

• Compressed acoustic lattices

Model # edge pops Oracle WER WER acoustic lats N/A 3.26 N/A 100x LatParse 3.4 million 5.45 13.1 10x AttShift 1.6 million 4.17 13.1

7/22/2004 ACL04: Attention Shifting for Parsing Speech 14

Conclusion

• Attention shifting

– Improves parsing efficiency – Increases first-stage accuracy (correcting for best-first search errors) – Does not improve multi-stage accuracy

• Pruning for second-stage parser constrains number of edges
• Useful for best-first word-lattices parsing

7/22/2004 ACL04: Attention Shifting for Parsing Speech 15