Parsing Probabilistic Context Free Grammars CMSC 473/673 UMBC - PowerPoint PPT Presentation

Parsing Probabilistic Context Free Grammars CMSC 473/673 UMBC November 8 th , 2017

Recap from last time…

Constituents Help Form Grammars constituent: spans of words that act (syntactically) as a group “X phrase” (noun phrase) Baltimore is a great place to be . This house is a great place to be . This red house is a great place to be . This red house on the hill is a great place to be . This red house near the hill is a great place to be . This red house atop the hill is a great place to be . The hill is a great place to be . S  NP VP PP  P NP NP  Det Noun AdjP  Adj Noun NP  Noun VP  V NP NP  Det AdjP Noun  Baltimore NP  NP PP

Context Free Grammar S  NP VP PP  P NP NP  Det Noun AdjP  Adj Noun NP  Noun VP  V NP NP  Det AdjP Noun  Baltimore NP  NP PP Set of rewrite rules, comprised of terminals and non-terminals Terminals: the words in the language (the lexicon), e.g., Baltimore Non-terminals: symbols that can trigger rewrite rules, e.g., S, NP , Noun (Sometimes) Pre-terminals: symbols that can only trigger lexical rewrites, e.g., Noun

Generate from a Context Free Grammar S  NP VP PP  P NP NP  Det Noun AdjP  Adj Noun NP  Noun VP  V NP NP  Det AdjP Noun  Baltimore NP  NP PP … S NP VP Baltimore is a great city NP Noun Verb Baltimore is a great city

Assign Structure (Parse) with a Context Free Grammar S  NP VP PP  P NP NP  Det Noun AdjP  Adj Noun NP  Noun VP  V NP NP  Det AdjP Noun  Baltimore NP  NP PP … S Baltimore is a great city NP VP [ S [ NP [ Noun Baltimore] ] [ VP [ Verb is] [ NP a great city]]] bracket notation NP Noun Verb (S (NP (Noun Baltimore)) (VP (V is) (NP a great city))) Baltimore is a great city S-expression

Parsing as a Core NLP Problem Gold (correct) reference trees sentence 1 sentence 2 score Evaluation Parser sentence 3 sentence 4 Grammar Other NLP task independent (entity coref., operations MT, Q&A, …)

Grammars Aren’t Just for Syntax N overgeneralization N  N N over- generalization V  N V generalize -tion A  V A general -ize overgeneralization

Clearly Show Ambiguity… But Not Necessarily All Ambiguity PP Attachment Semantic (a common source of Ambiguities errors, even still today) I ate the meal with friends I ate the meal with gusto I ate the meal with a fork VP NP PP Issue 1: Which grammar? NP VP Issue 2: Discourse demands S flexibility

How Do We Robustly Handle Ambiguities?

How Do We Robustly Handle Ambiguities? Add probabilities (to what?)

Probabilistic Context Free Grammar S  NP VP PP  P NP NP  Det Noun AdjP  Adj Noun NP  Noun VP  V NP NP  Det AdjP Noun  Baltimore NP  NP PP … Set of weighted (probabilistic) rewrite rules, comprised of terminals and non-terminals Terminals: the words in the language (the lexicon), e.g., Baltimore Non-terminals: symbols that can trigger rewrite rules, e.g., S, NP , Noun (Sometimes) Pre-terminals: symbols that can only trigger lexical rewrites, e.g., Noun

Probabilistic Context Free Grammar S  NP VP PP  P NP NP  Det Noun AdjP  Adj Noun NP  Noun VP  V NP NP  Det AdjP Noun  Baltimore NP  NP PP … Set of weighted (probabilistic) rewrite Q: What are the distributions? rules, comprised of terminals and What must sum to 1? non-terminals Terminals: the words in the language (the lexicon), e.g., Baltimore Non-terminals: symbols that can trigger rewrite rules, e.g., S, NP , Noun (Sometimes) Pre-terminals: symbols that can only trigger lexical rewrites, e.g., Noun

Probabilistic Context Free Grammar 1.0 S  NP VP 1.0 PP  P NP .4 NP  Det Noun .34 AdjP  Adj Noun .3 NP  Noun .26 VP  V NP .2 NP  Det AdjP .0003 Noun  Baltimore .1 NP  NP PP … Set of weighted (probabilistic) rewrite Q: What are the distributions? rules, comprised of terminals and What must sum to 1? non-terminals Terminals: the words in the language (the lexicon), e.g., Baltimore A: P(X  Y Z | X) Non-terminals: symbols that can trigger rewrite rules, e.g., S, NP , Noun (Sometimes) Pre-terminals: symbols that can only trigger lexical rewrites, e.g., Noun

Probabilistic Context Free Grammar S p( )= NP VP product of probabilities of individual rules used in the derivation NP Noun Verb Baltimore is a great city

Probabilistic Context Free Grammar S p( VP ) * NP S p( )= NP VP NP Noun Verb Baltimore is a great city product of probabilities of individual rules used in the derivation

Probabilistic Context Free Grammar S p( VP ) * NP S NP Noun p( ) * p( ) * p( )= NP VP Noun Baltimore NP Noun Verb Baltimore is a great city product of probabilities of individual rules used in the derivation

Probabilistic Context Free Grammar S p( VP ) * NP S NP Noun p( ) * p( ) * p( )= NP VP Noun Baltimore VP NP Noun Verb Verb p( ) * p( ) * is Baltimore is a great city NP Verb product of probabilities of NP p( ) individual rules used in the derivation a great city

Log Probabilistic Context Free Grammar S lp( VP ) + NP S NP Noun lp( ) + lp( ) + lp( )= NP VP Noun Baltimore VP NP Noun Verb Verb lp( ) + lp( ) + is Baltimore is a great city NP Verb sum of log probabilities of NP lp( ) individual rules used in the derivation a great city

Estimating PCFGs Attempt 1: • Get access to a treebank (corpus of syntactically annotated sentences), e.g., the English Penn Treebank • Count productions • Smooth these counts • This gets ~75 F1

Probabilistic Context Free Grammar (PCFG) Tasks Find the most likely parse (for an observed sequence) Calculate the (log) likelihood of an observed sequence w 1 , …, w N Learn the grammar parameters

Probabilistic Context Free Grammar (PCFG) Tasks Find the most likely parse (for an observed sequence) Calculate the (log) likelihood of an observed sequence w 1 , …, w N Learn the grammar parameters

Probabilistic Context Free Grammar (PCFG) Tasks any Find the most likely parse (for an observed sequence) Calculate the (log) likelihood of an observed sequence w 1 , …, w N Learn the grammar parameters

Parsing with a CFG Top-down backtracking (brute force) CKY Algorithm: dynamic bottom-up Earley’s Algorithm: dynamic top-down

Parsing with a CFG Top-down backtracking (brute force) CKY Algorithm: dynamic bottom-up Earley’s Algorithm: dynamic top-down

CKY Precondition Grammar must be in Chomsky Normal Form (CNF) non-terminal  non-terminal non-terminal non-terminal  terminal

CKY Precondition Grammar must be in Chomsky Normal Form (CNF) non-terminal  non-terminal non-terminal X  Y Z non-terminal  terminal X  a

CKY Precondition Grammar must be in Chomsky Normal Form (CNF) non-terminal  non-terminal non-terminal X  Y Z binary rules can only involve non-terminals non-terminal  terminal X  a unary rules can only involve terminals no ternary (+) rules

S  NP VP NP  Papa NP  Det N N  caviar NP  NP PP N  spoon VP  V NP V  spoon VP  VP PP V  ate PP  P NP P  with Det  the Entire grammar Assume uniform weights Det  a Example from Jason Eisner

0 1 2 3 4 5 6 7 “Papa ate the caviar with a spoon” S  NP VP NP  Papa NP  Det N N  caviar NP  NP PP N  spoon VP  V NP V  spoon VP  VP PP V  ate PP  P NP P  with Det  the Entire grammar Assume uniform weights Det  a Example from Jason Eisner

0 1 2 3 4 5 6 7 “Papa ate the caviar with a spoon” S  NP VP NP  Papa NP  Det N N  caviar Goal: NP  NP PP N  spoon VP  V NP V  spoon (S, 0, 7) VP  VP PP V  ate PP  P NP P  with Det  the Entire grammar Assume uniform weights Det  a Example from Jason Eisner

0 1 2 3 4 5 6 7 “Papa ate the caviar with a spoon” Check 1 : What are the non- terminals? S  NP VP NP  Papa NP  Det N N  caviar NP  NP PP N  spoon VP  V NP V  spoon VP  VP PP V  ate PP  P NP P  with Det  the Entire grammar Det  a Assume uniform weights Example from Jason Eisner

0 1 2 3 4 5 6 7 “Papa ate the caviar with a spoon” Check 1 : What are the non- terminals? S  NP VP NP  Papa S N NP  Det N N  caviar NP V VP P NP  NP PP N  spoon PP Det VP  V NP V  spoon Check 2 : What are the terminals? VP  VP PP V  ate PP  P NP P  with Det  the Entire grammar Det  a Assume uniform weights Example from Jason Eisner

0 1 2 3 4 5 6 7 “Papa ate the caviar with a spoon” Check 1 : What are the non- terminals? S  NP VP NP  Papa S N NP  Det N N  caviar NP V VP P NP  NP PP N  spoon PP Det VP  V NP V  spoon Check 2 : What are the terminals? VP  VP PP V  ate Papa with PP  P NP P  with caviar the Det  the spoon a ate Entire grammar Det  a Assume uniform weights Check 3 : What are the pre- terminals? Example from Jason Eisner