Informatics 2A: Language Complexity and the Chomsky Hierarchy - - PowerPoint PPT Presentation

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Informatics 2A: Language Complexity and the Chomsky Hierarchy – Concluding Slides

Slides by Bonnie Webber (modified by Stuart Anderson) October 1, 2009

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Three Models for the Description of Language

◮ Linguistic theory attempts to explain the ability of a speaker

to produce and understand new sentences, and to reject as ungrammatical other new sequences, on the basis of his limited linguistic experience. [Chomsky 1956, p. 113]

◮ The adequacy of a linguistic theory can be tested by looking

at a grammar for a language constructed according to the theory and seeing if it makes predictions that accord with what’s found in a large corpus of sentences of that language.

◮ What about what is not found in a large corpus of sentences? ◮ Chomsky’s paper explores the sort of linguistic theory that is

“required as a basis for an English grammar what will describe the set of English sentences in an interesting and satisfying manner”.

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Three Models for the Description of Language

For that description to be “interesting and satisfying”, Chomsky felt that a grammar had to be

◮ finite ◮ “revealing”, in allowing strings to be associated with meaning

(semantics) in a systematic way The three models he considered were:

• 1. Grammars based on Finite-state Markov processes [Shannon

& Weaver 1947, The Mathematical Theory of Communication] – regular grammars

• 2. Phrase structure grammars reflecting pedagogical ideas of

“sentence diagramming”

• 3. Transformational grammars

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Dependency and Complexity

Much of Chomsky’s argument in 3MDL is based on the notion of dependency: Suppose s =a1a2 . . . an is a sentence of language L. We say that S has an i-j dependency if when symbol ai is replaced with symbol bi, the string is no longer a sentence of L and when symbol aj is then replaced by some new symbol bj, the resulting string is a sentence of L. We’ve already seen such a dependency in English: Mary persuaded John to wash himself with lavender soap. John ⇒ Sue himself ⇒ herself Mary persuaded Sue to wash herself with lavender soap.

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Dependency Sets

◮ If we restrict ourselves to binary dependencies, then for any

sentence s we can construct a dependency set D = {(i1, j1), . . . (ik, jk)} where each pair is a dependency in S.

◮ For example: If Mary has persuaded John to wash himself

with lavender soap, then he is clean. (dep set size = 4)

◮ Sentences in the language generated by a regular grammar

can have dependencies.

◮ Consider the regular language described by a regular

expression: L0 = (b∗ + (ab∗c))∗ I.e. where every a is eventually followed by a c and only bs may intervene

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

An example: L0

◮ bbbabbcbbbabcbbbb ∈ L0 is a typical sentence in the

language.

◮ {(4, 7), (11, 13)} is the dependency set for the sentence. ◮ If we use the convention that we colour the pair of symbols in

the dependency set the same colour and we can reuse colours for parts of the string after the later symbol in the dependency pair has appeared. How many colours do we need to colour the symbols in sentences in L0?

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

An example: L0

◮ bbbabbcbbbabcbbbb ∈ L0 is a typical sentence in the

language.

◮ {(4, 7), (11, 13)} is the dependency set for the sentence. ◮ If we use the convention that we colour the pair of symbols in

the dependency set the same colour and we can reuse colours for parts of the string after the later symbol in the dependency pair has appeared. How many colours do we need to colour the symbols in sentences in L0?

◮ bbbabbcbbbabcbbbb uses just one colour.

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Limits to Dependencies

◮ The number of colours we need to colour the dependency set

• f a sentence gives us a measure of the amount that has to be

remembered about earlier symbols to get the dependencies

• right. If we need k colours then we need to remember k

symbols at most at any one time.

◮ For any regular language R there must exist a constant kR

such that the dependency set for any sentence in the language can be coloured with at most kR colours.

◮ What do you make of this claim?

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Three Languages

Consider the following three languages. In the first two you can choose k to be any fixed value (e.g. 4 or 4000000):

◮ L1 ⊆ {a, b}∗ for any string s ∈ L1 the difference between the

number of as and the number of bs in all prefixes of s is less than k.

◮ L2 ⊆ {a, b}∗ for any string s ∈ L2 the difference between the

number of as and the number of bs in s is less than k.

◮ L3 = {s#s | s ∈ {a, b}∗}

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Dependencies in L1

◮ What is the dependency set for aaabbaabbbb for language L1?

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Dependencies in L1

◮ What is the dependency set for aaabbaabbbb for language L1? ◮ aaabbaabbbb

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Dependencies in L1

◮ What is the dependency set for aaabbaabbbb for language L1? ◮ aaabbaabbbb ◮ How many colours are required for dependencies in strings

from L1?

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Dependencies in L1

◮ What is the dependency set for aaabbaabbbb for language L1? ◮ aaabbaabbbb ◮ How many colours are required for dependencies in strings

from L1?

◮ at most k

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Dependencies in L2

◮ What is the dependency set for anbn−k for language L2?

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Dependencies in L2

◮ What is the dependency set for anbn−k for language L2? ◮ {(n − i + 1, n + i) | 1 ≤ i ≤ n − k}

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Dependencies in L2

◮ What is the dependency set for anbn−k for language L2? ◮ {(n − i + 1, n + i) | 1 ≤ i ≤ n − k} ◮ How many colours are required for dependencies in strings

from L2?

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Dependencies in L2

◮ What is the dependency set for anbn−k for language L2? ◮ {(n − i + 1, n + i) | 1 ≤ i ≤ n − k} ◮ How many colours are required for dependencies in strings

from L2?

◮ An unbounded number.

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Dependencies in L3

◮ What is the dependency set for an#an for language L3?

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Dependencies in L3

◮ What is the dependency set for an#an for language L3? ◮ {(i, n + i + 1) | 1 ≤ i ≤ n}

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Dependencies in L3

◮ What is the dependency set for an#an for language L3? ◮ {(i, n + i + 1) | 1 ≤ i ≤ n} ◮ How many colours are required for dependencies in strings

from L3?

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Dependencies in L3

◮ What is the dependency set for an#an for language L3? ◮ {(i, n + i + 1) | 1 ≤ i ≤ n} ◮ How many colours are required for dependencies in strings

from L3?

◮ An unbounded number.

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Dependencies in L3

◮ What is the dependency set for an#an for language L3? ◮ {(i, n + i + 1) | 1 ≤ i ≤ n} ◮ How many colours are required for dependencies in strings

from L3?

◮ An unbounded number. ◮ How do the dependencies in L3 differ from those in L2?

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Dependencies in L3

◮ What is the dependency set for an#an for language L3? ◮ {(i, n + i + 1) | 1 ≤ i ≤ n} ◮ How many colours are required for dependencies in strings

from L3?

◮ An unbounded number. ◮ How do the dependencies in L3 differ from those in L2? ◮ The dependencies in L2 are nested those in L3 are not.

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

The simplest languages – ones that can be described by a regular grammar – need at most a finite number of colours to colour any dpendency set in the language. They are at the lowest rung of the Chomsky Hierarchy. regular grammars Are all languages with arbitrarily many dependencies equally complex?

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Phrase Structure Grammars

Phrase structure grammars provide a way of analysing sentences very much like some of us were taught to do at school: the man took the book NP verb NP VP Sentence This is called an “Immediate Constituent Analysis”. It shows a sentence made of a noun phrase (NP) followed by a verb phrase (VP). . . . a verb phrase made of a verb folllowed by an NP. How is phrase structure specified?

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

A phrase structure grammar consists of

◮ a finite vocabulary V ◮ a finite set Σ of initial strings over V ◮ a finite set of rules of the form X → Y where

• 1. X and Y are strings over V
• 2. Y is formed from X by replacing one symbol of X with a string
• ver V
• 3. Neither the replaced symbol nor the replacing string is empty

(ǫ).

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Context-free Phrase Structure Grammars

Rules of the simplest PS Grammars contain only a single symbol

• n their left-hand side – e.g.,

Σ: {S} S → NP VP VP → verb NP NP → the man NP → the book verb → took These are called Context-free PSGs or, for short, Context-free Grammars (CFGs).

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Derivations in CFGs

The sequence of strings over V produced by a sequence of PS rule applications, starting from an initial string, is called a derivation: S ⇒ NP VP ⇒ NP verb NP ⇒ NP verb the book ⇒ NP took the book ⇒ the man took the book

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Dependency and PS Grammars

◮ Some dependencies that are beyond the capability of a regular

grammar can be captured by a context-free grammar.

◮ Such dependencies are ones that can be generated locally. ◮ These locally generated dependencies are nested

Recall L2: L2 ⊆ {a, b}∗ for any string s ∈ L2 the difference between the number of as and the number of bs in s is less than k. The Phrase Structure Gramma for generating L2 with k = 1: V = {a, b, S, A, B, C} Σ = {S} S → A | B | C C → ab | ba | aCb | bCa | CC A → a | AC | CA B → b | BC | CB

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Derivation

Sample derivation: S ⇒ CC ⇒ aCbC ⇒ aabbC ⇒ aabbCC ⇒ aabbbaC⇒ aabbbaab

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Dependency and Complexity Revisited

Are all dependencies local? Are there dependencies that cannot be captured by a CFG? context-free grammars regular grammars The dependencies in L3 = {s#s | s ∈ {a, b}∗} cannot be captured by a CFG because the dependencies are not nested and so cannot be captured by context-free rules.

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Context-sensitive PSGs

Phrase structure grammars with rules whose LHS contain > 1 symbols and whose RHS is at least as long as the LHS are called context-sensitive phrase structure grammars or simply, context-sensitive grammars. These can capture more complex dependencies than CFGs. Simple context-sensitive grammar for generating L3: V = {a, b, #, S, A, B, C} Σ = {S} S → aAS | bBS | C Aa → aA Ba → aB Ab → bA Bb → bB AC → Ca BC → Cb aC → a# bC → b#

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Sample Derivation

Sample derivation: S ⇒ aAS ⇒ aAbBS ⇒ aAbBbBS ⇒ aAbBbBC ⇒ aAbBbCb ⇒ aAbbBCb ⇒ aAbbCbb ⇒ abAbCbb ⇒ abbACbb ⇒abbCabb⇒abb#abb Context on the LHS allows for more dependencies and hence more complexity. context-sensitive grammars context-free grammars regular grammars

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy

Dependency as a measure of Complexity of Language The Chomsky Hierarchy

Top of the Chomsky Hierarchy

Arbitrary re-write systems that can take account of any amount of context on the LHS and re-write any number of symbols, called Type 0 grammars. Type 0 grammars context-sensitive grammars context-free grammars regular grammars This is what is normally called the Chomsky hierarchy.

Slides by Bonnie Webber (modified by Stuart Anderson) Inf2A: Chomsky Hierarchy