Statistical Parsing Recap (dashed ellipse) are adequate for - - PDF document

SLIDE 1

Statistical Parsing

Parsing context-free languages Çağrı Çöltekin

University of Tübingen Seminar für Sprachwissenschaft

November 8, 2016

Recap Parsing basics CKY Earley Closing

Ingredients of a (natural language) parser

• A grammar
• An algorithm for parsing
• A method for ambiguity resolution

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 1 / 27 Recap Parsing basics CKY Earley Closing

Grammars

• A formal grammar is a fjnite specifjcation of a (possibly

infjnite) language

• In this course, we are interested in two broad classes of

grammars Constitunecy (or phrase structure) grammars, Dependency grammars

• Various theories of ‘grammar’ (e.g., HPSG, LFG, CCG)
• ften use ideas/notions from both constituencies and

dependencies

• We will study these grammars in their relation to parsing,

we do not study or focus on any specifjc theory

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 2 / 27 Recap Parsing basics CKY Earley Closing

Dependency vs. constituency

• Constituency grammars are based
• n units formed by a group of lexical

items (constituents or phrases)

• Dependency grammars model

binary head–dependent relations between words

• Most of the theory of parsing is

developed with constituency grammars

• Dependency grammars has recently

become popular in CL

S NP John VP V saw NP Marry John saw Marry

subject

• bject

root

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 3 / 27 Recap Parsing basics CKY Earley Closing

Phrase structure grammars

A phrase structure grammar is specifjed by, Σ is a set of terminal symbols N is a set of non-terminal symbols S ∈ N is a distinguished start symbol R is a set of rules of the form

αAβ → γ for A ∈ N α, β, γ ∈ Σ ∪ N

• The grammar accepts a sentence if it can be derived from S

with the rewrite rules R

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 4 / 27 Recap Parsing basics CKY Earley Closing

Chomsky hierarchy and natural languages

Regular Context Free Context Sensitive Recursively Enumerable

• Chomsky hierarchy of languages form a set-inclusion hierarchy
• It is often claimed that mildly context sensitive grammars

(dashed ellipse) are adequate for representing natural languages

• Note, however, that the possible natural languages probably

cross-cut this hierarchy (shaded region)

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 5 / 27 Recap Parsing basics CKY Earley Closing

Context free grammars

• Context free grammars are suffjcient for expressing most

phenomena in natural language syntax

• Most of the parsing theory (and practice) is build on

parsing CF languages

• The context-free rules have the form

A → α where A is a single non-terminal symbol and α is a (possibly empty) sequence of terminal or non-terminal symbols

• We will mainly focus with parsing with context-free

grammars for the rest of this lecture

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 6 / 27 Recap Parsing basics CKY Earley Closing

An example grammar

S → NP VP S → Aux NP VP NP → Det N NP → Prn NP → NP PP VP → V NP VP → V VP → VP PP PP → Prp NP N → duck N → park N → parks V → duck V → ducks V → saw Prn → she | her Prp → in | with Det → a | the

Derivation of sentence ‘she saw a duck’ S ⇒ NP VP NP ⇒ Prn Prn ⇒ she VP ⇒ V NP V ⇒ saw NP ⇒ Det N Det ⇒ a N ⇒ duck S NP Prn she VP V saw NP Det a N duck

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 7 / 27

SLIDE 2

Recap Parsing basics CKY Earley Closing

Parsing as search

• Parsing can be seen as search constrained by the grammar

and the input

• Top down: start from S, fjnd the derivations that lead to

the sentence

• Bottom up: start from the sentence, fjnd series of

derivations (in reverse) that leads to S

• One can search depth fjrst or breadth fjrst in both

directions

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 8 / 27 Recap Parsing basics CKY Earley Closing

Parsing as search: top down

S NP Det Prn she NP VP V saw NP Det a N duck she saw a duck Backtrack!

S → NP VP S → Aux NP VP NP → Det N NP → Prn NP → NP PP VP → V NP VP → V VP → VP PP PP → Prp NP N → duck N → park N → parks V → duck V → ducks V → saw Prn → she | her Prp → in | with Det → a | the

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 9 / 27 Recap Parsing basics CKY Earley Closing

Parsing as search: top down

S NP Det Prn she NP VP V saw NP Det a N duck she saw a duck Backtrack!

S → NP VP S → Aux NP VP NP → Det N NP → Prn NP → NP PP VP → V NP VP → V VP → VP PP PP → Prp NP N → duck N → park N → parks V → duck V → ducks V → saw Prn → she | her Prp → in | with Det → a | the

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 9 / 27 Recap Parsing basics CKY Earley Closing

Parsing as search: bottom up

she saw a duck Prn V Det N NP NP VP S

S → NP VP S → Aux NP VP NP → Det N NP → Prn NP → NP PP VP → V NP VP → V VP → VP PP PP → Prp NP N → duck N → park N → parks V → duck V → ducks V → saw Prn → she | her Prp → in | with Det → a | the

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 10 / 27 Recap Parsing basics CKY Earley Closing

Problems with search procedures

• Top-down search considers productions incompatible with

the input, and cannot handle left recursion

• Bottom-up search considers non-terminals that would

never lead to S

• Repeated work because of backtracking

→ The result is exponential time complexity in the length of the sentence Some of these problems can be solved using dynamic programming techniques.

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 11 / 27 Recap Parsing basics CKY Earley Closing

CKY algorithm

• The CKY (Cocke–Younger–Kasami), or CYK, parsing

algorithm is a dynamic programming algorithm (Kasami 1965; Younger 1967; Cocke and Schwartz 1970)

• It processes the input bottom up, and saves the intermediate

results on a chart

• Time complexity for recognition is O(n3) (with a space

complexity of O(n2))

• It requires the CFG to be in Chomsky normal form (CNF)

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 12 / 27 Recap Parsing basics CKY Earley Closing

Chomsky normal form (CNF)

• A CFG is in CNF, if the rewrite rules are in one of the

following forms

– A → B C – A → a

where A, B, C are non-terminals and a is a terminal

• Any CFG can be converted to CNF
• Resulting grammar is weakly equivalent to the original

grammar: it generates/accepts the same language, but the derivations are difgerent

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 13 / 27 Recap Parsing basics CKY Earley Closing

Converting to CNF: example

• For rules with > 2 RHS symbols

S →Aux NP VP ⇒ S →Aux X X →NP VP

• For rules with < 2 RHS symbols

NP →Det ⇒ NP → a | the

S → NP VP S → Aux NP VP NP → Det N NP → Prn NP → NP PP VP → V NP VP → V VP → VP PP PP → Prp NP N → duck N → park N → parks V → duck V → ducks V → saw Prn → she | her Prp → in | with Det → a | the

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 14 / 27

SLIDE 3

Recap Parsing basics CKY Earley Closing

CKY demonstration

she saw a duck

1 2 3 4

NP V Det N, V NP VP S

S → NP VP S → Aux NP VP NP → Det N NP → Prn NP → NP PP VP → V NP VP → V VP → VP PP PP → Prp NP N → duck N → park N → parks V → duck V → ducks V → saw Prn → she | her Prp → in | with Det → a | the

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 15 / 27 Recap Parsing basics CKY Earley Closing

CKY demonstration

she saw a duck

1 2 3 4

NP V Det V, N NP VP S

S → NP VP S → Aux NP VP NP → Det N NP → Prn NP → NP PP VP → V NP VP → V VP → VP PP PP → Prp NP N → duck N → park N → parks V → duck V → ducks V → saw Prn → she | her Prp → in | with Det → a | the

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 16 / 27 Recap Parsing basics CKY Earley Closing

CKY summary

• We went through a recognition example, for parsing we

need minor alternations, but worse case complexity becomes exponential + CKY avoid re-computing the analyses for substrings by storing them in a table − It still computes/stores lower level constituents that are not possible − CKY requires the grammar to be in CNF

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 17 / 27 Recap Parsing basics CKY Earley Closing

Earley algorithm

• Earley algorithm is a top down parsing algorithm (Earley

1970)

• It allows arbitrary CFGs
• Keeps record of constituents that are

predicted using the grammar (top-down) in-progress with partial evidence completed based on input seen so far at every position in the input string

• Time complexity is O(n3)

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 18 / 27 Recap Parsing basics CKY Earley Closing

Earley chart entries (states or items)

Early chart entries are CF rules with a ‘dot’ on the RHS representing the state of the rule

• A → •α[i, i] predicted without any evidence (yet)
• A → α • β[i, j] partially matched
• A → αβ • [i, j] completed, the non-terminal A is found in

the given span

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 19 / 27 Recap Parsing basics CKY Earley Closing

Earley algorithm: an informal sketch

• 1. Start at position 0, predict S
• 2. Predict all possible states (rules that apply)
• 3. Read a word
• 4. Update the table, advance the dot if possible
• 5. Go to step 2
• 6. If we have a completed S production at the end of the

input, the input it recognized

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 20 / 27 Recap Parsing basics CKY Earley Closing

Earley algorithm: three operations

Predictor adds all possible rules that are possible at the given state Completer adds states from the earlier chart entries that match the completed state to the chart entry being processed, and advances their dot Scanner adds a completed state to the next chart entry if the current category is a POS tag, and the word matches

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 21 / 27 Recap Parsing basics CKY Earley Closing

Earley parsing example (chart[0])

she saw a duck

1 2 3 4

state rule position

• peration

γ →•S [0,0] initialization 1 S →•NP VP [0,0] predictor 2 S →•Aux NP VP [0,0] predictor 3 NP →•Det N [0,0] predictor 4 NP →•NP PP [0,0] predictor 5 NP →•Prn [0,0] predictor

S → NP VP S → Aux NP VP NP → Det N NP → Prn NP → NP PP VP → V NP VP → V VP → VP PP PP → Prp NP N → duck N → park N → parks V → duck V → ducks V → saw Prn → she | her Prp → in | with Det → a | the Aux → does | has

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 22 / 27

SLIDE 4

Recap Parsing basics CKY Earley Closing

Earley parsing example (chart[1])

she saw a duck

1 2 3 4

state rule position

• peration

6 Prn →she • [0,1] scanner 7 NP →Prn • [0,1] completer 8 S →NP •VP [0,1] completer 9 NP →NP •PP [0,1] completer 10 VP →•V NP [1,1] predictor 11 VP →•VP PP [1,1] predictor 12 PP →•Prp NP [1,1] predictor

S → NP VP S → Aux NP VP NP → Det N NP → Prn NP → NP PP VP → V NP VP → V VP → VP PP PP → Prp NP N → duck N → park N → parks V → duck V → ducks V → saw Prn → she | her Prp → in | with Det → a | the Aux → does | has

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 23 / 27 Recap Parsing basics CKY Earley Closing

Earley parsing example (chart[2])

she saw a duck

1 2 3 4

state rule position

• peration

13 V →saw • [1,2] scanner 14 VP →V •NP [1,2] completer 15 VP →V • [1,2] completer 16 NP →•Det N [2,2] predictor 17 NP →•NP PP [2,2] predictor 18 NP →•Prn [2,2] predictor 19 S →NP VP • [0,2] predictor

S → NP VP S → Aux NP VP NP → Det N NP → Prn NP → NP PP VP → V NP VP → V VP → VP PP PP → Prp NP N → duck N → park N → parks V → duck V → ducks V → saw Prn → she | her Prp → in | with Det → a | the Aux → does | has

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 24 / 27 Recap Parsing basics CKY Earley Closing

Earley parsing example (chart[3])

she saw a duck

1 2 3 4

state rule position

• peration

20 Det →a • [2,3] scanner 21 NP →Det •N [2,3] completer

S → NP VP S → Aux NP VP NP → Det N NP → Prn NP → NP PP VP → V NP VP → V VP → VP PP PP → Prp NP N → duck N → park N → parks V → duck V → ducks V → saw Prn → she | her Prp → in | with Det → a | the Aux → does | has

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 25 / 27 Recap Parsing basics CKY Earley Closing

Earley parsing example (chart[4])

she saw a duck

1 2 3 4

state rule position

• peration

22 N →duck • [3,4] scanner 23 NP →Det N • [2,4] completer 24 VP →V NP • [1,4] completer 25 S →NP VP • [0,4] completer

S → NP VP S → Aux NP VP NP → Det N NP → Prn NP → NP PP VP → V NP VP → V VP → VP PP PP → Prp NP N → duck N → park N → parks V → duck V → ducks V → saw Prn → she | her Prp → in | with Det → a | the Aux → does | has

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 26 / 27 Recap Parsing basics CKY Earley Closing

Summary

• Naive search for parsing is intractable
• Dynamic programming algorithms allow polynomial time

recognition

• Parsing may still be exponential in the worse case
• Ambiguity: CKY or Early parse tables can represent

ambiguity, but cannot say anything about which parse is the best Next: statistical parsing

Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 27 / 27

Bibliography

Cocke, John and J. T. Schwartz (1970). Programming languages and their compilers: preliminary notes. Tech. rep. Courant Institute of Mathematical Sciences, NYU. Earley, Jay (1970). “An Effjcient Context-free Parsing Algorithm”. In: Commun. ACM 13.2, pp. 94–102. issn: 0001-0782. doi: 10.1145/362007.362035. url: http://doi.acm.org/10.1145/362007.362035. Kasami, Tadao (1965). An Effjcient Recognition and Syntax-Analysis Algorithm for Context-Free Languages. Tech. rep. DTIC Document. Younger, Daniel H (1967). “Recognition and parsing of context-free languages in time n 3”. In: Information and control 10.2, pp. 189–208. Ç. Çöltekin, SfS / University of Tübingen November 8, 2016 A.1