{Probabilistic | Stochastic} Context-Free Grammars (PCFGs) 116 The - PowerPoint PPT Presentation

{Probabilistic | Stochastic} Context-Free Grammars (PCFGs) 116

� The velocity of the seismic waves rises to . . . S NP sg VP sg DT NN PP rises to . . . The velocity IN NP pl of the seismic waves 117

PCFGs A PCFG G consists of: � A set of terminals, { w k } , k = 1 , . . . , V � A set of nonterminals, { N i } , i = 1 , . . . , n � A designated start symbol, N 1 � A set of rules, { N i → ζ j } , (where ζ j is a sequence of terminals and nonterminals) � A corresponding set of probabilities on rules such that: P(N i → ζ j ) = 1 � ∀ i j 118

PCFG notation Sentence: sequence of words w 1 · · · w m w ab : the subsequence w a · · · w b ab : nonterminal N i dominates w a · · · w b N i N j w a · · · w b ∗ � ⇒ ζ : Repeated derivation from N i gives ζ . N i 119

PCFG probability of a string � P(w 1 n ) = P(w 1 n , t) t a parse of w 1 n t � = P(t) { t :yield ( t ) = w 1n } 120

A simple PCFG (in CNF) S → NP VP NP → NP PP 1.0 0.4 PP → P NP NP → astronomers 1.0 0.1 VP → V NP NP → ears 0.7 0.18 VP → VP PP NP → saw 0.3 0.04 P → with 1.0 NP → stars 0.18 V → saw NP → telescopes 1.0 0.1 121

t 1 : S 1 . 0 NP 0 . 1 VP 0 . 7 astronomers V 1 . 0 NP 0 . 4 saw NP 0 . 18 PP 1 . 0 stars P 1 . 0 NP 0 . 18 with ears 122

t 2 : S 1 . 0 NP 0 . 1 VP 0 . 3 astronomers VP 0 . 7 PP 1 . 0 V 1 . 0 NP 0 . 18 P 1 . 0 NP 0 . 18 saw stars with ears 123

The two parse trees’ probabilities and the sen- tence probability P(t 1 ) = 1 . 0 × 0 . 1 × 0 . 7 × 1 . 0 × 0 . 4 × 0 . 18 × 1 . 0 × 1 . 0 × 0 . 18 = 0 . 0009072 P(t 2 ) = 1 . 0 × 0 . 1 × 0 . 3 × 0 . 7 × 1 . 0 × 0 . 18 × 1 . 0 × 1 . 0 × 0 . 18 = 0 . 0006804 P(w 15 ) = P(t 1 ) + P(t 2 ) = 0 . 0015876 124

Assumptions of PCFGs 1. Place invariance (like time invariance in HMM): P(N j ∀ k k(k + c) → ζ) is the same 2. Context-free: P(N j kl → ζ | words outside w k . . . w l ) = P(N j kl → ζ) 3. Ancestor-free: P(N j kl → ζ | ancestor nodes of N j kl ) = P(N j kl → ζ) 125

Let the upper left index in i N j be an arbitrary identifying index for a particular token of a nonterminal. Then,  1 S    P( 1 S 13 → 2 NP 12 3 VP 33 , 2 NP 12 → the 1 man 2 , 3 VP 33 → sno   2 NP 3 VP = P       the man snores = . . . = P(S → NP VP)P(NP → the man)P(VP → snores) 126

Some features of PCFGs Reasons to use a PCFG, and some idea of their limitations: � Partial solution for grammar ambiguity: a PCFG gives some idea of the plausibility of a sentence. � But not a very good idea, as not lexicalized. � Better for grammar induction (Gold 1967) � Robustness. (Admit everything with low probability.) 127

Some features of PCFGs � Gives a probabilistic language model for English. � In practice, a PCFG is a worse language model for English than a trigram model. � Can hope to combine the strengths of a PCFG and a trigram model. � PCFG encodes certain biases, e.g., that smaller trees are normally more probable. 128

Improper (inconsistent) distributions P = 1 � S → rhubarb 3 P = 2 S → S S 3 1 � rhubarb 3 3 × 1 2 3 × 1 3 = 2 rhubarb rhubarb 27 � 2 × � 3 × 2 = � 2 � 1 8 rhubarb rhubarb rhubarb 3 3 243 . . . � P( L ) = 1 3 + 2 243 + . . . = 1 8 27 + 2 � Improper/inconsistent distribution � Not a problem if you estimate from parsed treebank: Chi and Geman 1998). 129

Questions for PCFGs Just as for HMMs, there are three basic questions we wish to answer: � P(w 1 m | G) � arg max t P(t | w 1 m , G) � Learning algorithm. Find G such that P(w 1 m | G) is max- imized. 130

Chomsky Normal Form grammars We’ll do the case of Chomsky Normal Form grammars, which only have rules of the form: N i → N j N k N i → w j Any CFG can be represented by a weakly equivalent CFG in Chomsky Normal Form. It’s straightforward to generalize the algorithm (recall chart parsing). 131

PCFG parameters We’ll do the case of Chomsky Normal Form grammars, which only have rules of the form: N i → N j N k N i → w j The parameters of a CNF PCFG are: P(N j → N r N s | G) A n 3 matrix of parameters P(N j → w k | G) An nt matrix of parameters For j = 1 , . . . , n , P(N j → N r N s ) + P(N j → w k ) = 1 � � r,s k 132

Probabilistic Regular Grammar: N i → w j N k N i → w j Start state, N 1 HMM: � P(w 1 n ) = 1 ∀ n w 1 n whereas in a PCFG or a PRG: � P(w) = 1 w ∈ L 133

Consider: P( John decided to bake a ) High probability in HMM, low probability in a PRG or a PCFG. Implement via sink state. A PRG Π HMM Start Finish 134

Comparison of HMMs (PRGs) and PCFGs N ′ N ′ N ′ � → � → � → � → X : NP sink | | | | O : the big brown box NP N ′ the N ′ big N 0 brown box 135

Inside and outside probabilities This suggests: whereas for an HMM we have: Forwards = α i (t) = P(w 1 (t − 1 ) , X t = i) Backwards = β i (t) = P(w tT | X t = i) for a PCFG we make use of Inside and Outside probabilities, defined as follows: Outside = α j (p, q) = P(w 1 (p − 1 ) , N j pq , w (q + 1 )m | G) Inside = β j (p, q) = P(w pq | N j pq , G) A slight generalization of dynamic Bayes Nets covers PCFG inference by the inside-outside algorithm (and-or tree of conjunctive daughters disjunctively chosen) 136

Inside and outside probabilities in PCFGs. N 1 α N j β w p − 1 w p w q w q + 1 · · · · · · · · · w 1 w m 137

Probability of a string Inside probability P(w 1 m | G) = P(N 1 ⇒ w 1 m | G) = P(w 1 m , N 1 = β 1 ( 1 , m) 1 m , G) Base case: We want to find β j (k, k) (the probability of a rule N j → w k ): β j (k, k) = P(w k | N j kk , G) = P(N j → w k | G) 138

Induction: We want to find β j (p, q) , for p < q . As this is the inductive step using a Chomsky Normal Form grammar, the first rule must be of the form N j → N r N s , so we can proceed by induction, dividing the string in two in various places and summing the result: N j N r N s w p w d w d + 1 w q These inside probabilities can be calculated bottom up. 139

For all j , P(w pq | N j = β j (p, q) pq , G) q − 1 (d + 1 )q | N j P(w pd , N r pd , w (d + 1 )q , N s � � = pq , G) r,s d = p q − 1 (d + 1 )q | N j P(N r pd , N s � � = pq , G) r,s d = p P(w pd | N j pq , N r pd , N s (d + 1 )q , G) P(w (d + 1 )q | N j pq , N r pd , N s (d + 1 )q , w pd , G) q − 1 (d + 1 )q | N j P(N r pd , N s � � = pq , G) r,s d = p P(w pd | N r pd , G)P(w (d + 1 )q | N s (d + 1 )q , G) q − 1 P(N j → N r N s )β r (p, d)β s (d + 1 , q) � � = r,s d = p 140

Calculation of inside probabilities (CKY algorithm) 1 2 3 4 5 1 β NP = 0.1 β S = 0.0126 β S = 0.0015876 2 β NP = 0.04 β VP = 0.126 β VP = 0.015876 β V = 1.0 3 β NP = 0.18 β NP = 0.01296 4 β P = 1.0 β PP = 0.18 5 β NP = 0.18 astronomers saw stars with ears 141

Outside probabilities Probability of a string: For any k , 1 ≤ k ≤ m , P(w 1 (k − 1 ) , w k , w (k + 1 )m , N j � P(w 1 m | G) = kk | G) j P(w 1 (k − 1 ) , N j � = kk , w (k + 1 )m | G) j × P(w k | w 1 (k − 1 ) , N j kk , w (k + 1 )n , G) α j (k, k)P(N j → w k ) � = j Inductive (DP) calculation: One calculates the outside probabilities top down (after determining the inside probabilities). 142

Outside probabilities Base Case: α 1 ( 1 , m) = 1 α j ( 1 , m) = 0 , for j �= 1 Inductive Case: N 1 N f pe N g N j pq (q + 1 )e w 1 · · · w p − 1 w p · · · w q w q + 1 · · · w e w e + 1 · · · w m 143

Outside probabilities Base Case: α 1 ( 1 , m) = 1 α j ( 1 , m) = 0 , for j �= 1 Inductive Case: it’s either a left or right branch – we will some over both possibilities and calculate using outside and inside probabilities N 1 N f pe N j N g pq (q + 1 )e w 1 · · · w p − 1 w p · · · w q w q + 1 · · · w e w e + 1 · · · w m 144

Outside probabilities – inductive case A node N j pq might be the left or right branch of the parent node. We sum over both possibilities. N 1 N f eq N g N j pq e(p − 1 ) w 1 · · · w e − 1 w e · · · w p − 1 w p · · · w q w q + 1 · · · w m 145

Inductive Case: m P(w 1 (p − 1 ) , w (q + 1 )m , N f pe , N j pq , N g � � α j (p, q) = [ (q + 1 )e )] e = q + 1 f,g p − 1 P(w 1 (p − 1 ) , w (q + 1 )m , N f eq , N g e(p − 1 ) , N j � � + [ pq )] e = 1 f,g m P(w 1 (p − 1 ) , w (e + 1 )m , N f pe )P(N j pq , N g (q + 1 )e | N f � � = [ pe ) e = q + 1 f,gnej p − 1 × P(w (q + 1 )e | N g P(w 1 (e − 1 ) , w (q + 1 )m , N f � � (q + 1 )e )] + [ eq ) e = 1 f,g × P(N g e(p − 1 ) , N j pq | N f eq )P(w e(p − 1 ) | N g e(p − 1 )] m α f (p, e)P(N f → N j N g )β g (q + 1 , e)] � � = [ e = q + 1 f,g p − 1 α f (e, q)P(N f → N g N j )β g (e, p − 1 )] � � + [ e = 1 f,g 146

Overall probability of a node existing As with a HMM, we can form a product of the inside and outside probabilities. This time: α j (p, q)β j (p, q) = P(w 1 (p − 1 ) , N j pq , w (q + 1 )m | G)P(w pq | N j pq , G) = P(w 1 m , N j pq | G) Therefore, � p(w 1 m , N pq | G) = α j (p, q)β j (p, q) j Just in the cases of the root node and the preterminals, we know there will always be some such constituent. 147