CSCI 3136 Principles of Programming Languages Lexical Analysis and - PowerPoint PPT Presentation

CSCI 3136 Principles of Programming Languages Lexical Analysis and Automata Theory - 1 Summer 2013 Faculty of Computer Science Dalhousie University 1 / 78

Phases of Compilation Character Stream Scanner (lexical analysis) Token Stream Parser (syntactic analysis) Parse Tree Symbol Table Semantic Analysis and Inter- mediate Code Generation Abstract Syntax Tree or Other Intermediate Form Machine-independent Code Improvement (Optional) Modified Intermediate Form Target Code Generation Target Language (e.g., assembly) Machine-specific Code Im- provement (Optional) Modified Target Language 2 / 78

Parser (PDA) recognizes Context-free Grammar are gen- erated by Context-free Language

Scanner Parser (DFA) (PDA) recognizes recognizes Regular Expression Context-free Grammar are gen- are gen- erated erated by by Regular Language Context-free Language

Scanner Parser (DFA) (PDA) recognizes recognizes Regular Expression Context-free Grammar are gen- are gen- erated erated by by Regular Language Context-free Language

Scanner Parser (DFA) (PDA) recognizes recognizes Regular Expression Context-free Grammar are gen- are gen- erated erated by by Regular Language Context-free Language Formal Language (a set of strings)

Scanner Parser (DFA) (PDA) recognizes recognizes Regular Expression Context-free Grammar are gen- are gen- erated erated by by Regular Language Context-free Language

Scanner Parser (DFA) (PDA) recognizes recognizes Regular Expression Context-free Grammar are gen- are gen- erated erated by by Regular Language Context-free Language

Scanner Parser (DFA) (PDA) recognizes recognizes Rules: 1. Concatenation Regular Expression Context-free Grammar 2. Alternation 3. Kleene closure are gen- are gen- erated erated by by Regular Language Context-free Language

Scanner Parser (DFA) (PDA) recognizes recognizes Regular Expression Context-free Grammar are gen- are gen- erated erated by by Regular Language Context-free Language

Scanner Parser (DFA) (PDA) recognizes recognizes Regular Expression Context-free Grammar are gen- are gen- erated erated by by Regular Language Context-free Language

Scanner Parser (DFA) (PDA) recognizes recognizes Rules: 1. Concatenation Regular Expression Context-free Grammar 2. Alternation 3. Kleene closure are gen- are gen- erated erated by by Regular Language Context-free Language

Scanner Parser (DFA) (PDA) recognizes recognizes Rules: 1. Concatenation Regular Expression 2. Alternation Context-free Grammar 3. Kleene closure 4. Recursion are gen- are gen- erated erated by by Regular Language Context-free Language 14 / 78

Formal Languages A formal language L is a set of strings over an alphabet Σ. 15 / 78

Formal Languages A formal language L is a set of strings over an alphabet Σ. • Alphabet Σ: set of characters that can be used to form strings (letters, digits, punctuation, . . . ) 16 / 78

Formal Languages A formal language L is a set of strings over an alphabet Σ. • Alphabet Σ: set of characters that can be used to form strings (letters, digits, punctuation, . . . ) • String : finite sequence of characters 17 / 78

Formal Languages A formal language L is a set of strings over an alphabet Σ. • Alphabet Σ: set of characters that can be used to form strings (letters, digits, punctuation, . . . ) • String : finite sequence of characters • ǫ denotes the empty string (string with no letters: “ ′′ ) 18 / 78

Formal Languages A formal language L is a set of strings over an alphabet Σ. • Alphabet Σ: set of characters that can be used to form strings (letters, digits, punctuation, . . . ) • String : finite sequence of characters • ǫ denotes the empty string (string with no letters: “ ′′ ) • Length | s | of a string s = number of characters in s | ǫ | = 0, | a | = 1, | abc | = 3, . . . 19 / 78

Formal Languages A formal language L is a set of strings over an alphabet Σ. • Alphabet Σ: set of characters that can be used to form strings (letters, digits, punctuation, . . . ) • String : finite sequence of characters • ǫ denotes the empty string (string with no letters: “ ′′ ) • Length | s | of a string s = number of characters in s | ǫ | = 0, | a | = 1, | abc | = 3, . . . • Σ 0 = set of strings of length 0: Σ 0 = { ǫ } Σ 1 = set of strings of length 1: Σ 1 = { a, b, c, ... } Σ 2 = set of strings of length 2: Σ 2 = { aa, ab,..., ca,... } . . . 20 / 78

Formal Languages A formal language L is a set of strings over an alphabet Σ. • Alphabet Σ: set of characters that can be used to form strings (letters, digits, punctuation, . . . ) • String : finite sequence of characters • ǫ denotes the empty string (string with no letters: “ ′′ ) • Length | s | of a string s = number of characters in s | ǫ | = 0, | a | = 1, | abc | = 3, . . . • Σ 0 = set of strings of length 0: Σ 0 = { ǫ } Σ 1 = set of strings of length 1: Σ 1 = { a, b, c, ... } Σ 2 = set of strings of length 2: Σ 2 = { aa, ab,..., ca,... } . . . • Kleene star : Σ ∗ = Σ 0 ∪ Σ 1 ∪ Σ 2 ∪ . . . (set of all strings over alphabet Σ) 21 / 78

Formal Languages A formal language L is a set of strings over an alphabet Σ. • Alphabet Σ: set of characters that can be used to form strings (letters, digits, punctuation, . . . ) • String : finite sequence of characters • ǫ denotes the empty string (string with no letters: “ ′′ ) • Length | s | of a string s = number of characters in s | ǫ | = 0, | a | = 1, | abc | = 3, . . . • Σ 0 = set of strings of length 0: Σ 0 = { ǫ } Σ 1 = set of strings of length 1: Σ 1 = { a, b, c, ... } Σ 2 = set of strings of length 2: Σ 2 = { aa, ab,..., ca,... } . . . • Kleene star : Σ ∗ = Σ 0 ∪ Σ 1 ∪ Σ 2 ∪ . . . (set of all strings over alphabet Σ) • A formal language L over alphabet Σ is a subset L ⊆ Σ ∗ 22 / 78

Regular Expressions (RegEx) Alphabet Σ = { a, b, c } 23 / 78

Regular Expressions (RegEx) Alphabet Σ = { a, b, c } • Concatenation 24 / 78

Regular Expressions (RegEx) Alphabet Σ = { a, b, c } • Concatenation ab 25 / 78

Regular Expressions (RegEx) Alphabet Σ = { a, b, c } • Concatenation • Alternation ( | ) ab 26 / 78

Regular Expressions (RegEx) Alphabet Σ = { a, b, c } • Concatenation • Alternation ( | ) ab|c 27 / 78

Regular Expressions (RegEx) Alphabet Σ = { a, b, c } • Concatenation • Alternation ( | ) • Kleene star (*) ab|c 28 / 78

Regular Expressions (RegEx) Alphabet Σ = { a, b, c } • Concatenation • Alternation ( | ) • Kleene star (*) a ∗ b|c 29 / 78

Regular Expressions (RegEx) Alphabet Σ = { a, b, c } • Concatenation • Alternation ( | ) • Kleene star (*) • Parentheses a ∗ b|c 30 / 78

Regular Expressions (RegEx) Alphabet Σ = { a, b, c } • Concatenation • Alternation ( | ) • Kleene star (*) • Parentheses a ∗ (b|c) 31 / 78

Regular Expressions (RegEx) Alphabet Σ = { a, b, c } • Concatenation • Alternation ( | ) • Kleene star (*) • Parentheses a ∗ (b|c) 32 / 78

RegEx Operator Precedence 33 / 78

RegEx Operator Precedence • Parentheses 34 / 78

RegEx Operator Precedence • Parentheses a ∗ (b | c)a ∗ | a 35 / 78

RegEx Operator Precedence • Parentheses a ∗ (b | c)a ∗ | a 36 / 78

RegEx Operator Precedence • Parentheses • Kleene star (*) a ∗ (b | c)a ∗ | a 37 / 78

RegEx Operator Precedence • Parentheses • Kleene star (*) a ∗ (b | c)a ∗ | a 38 / 78

RegEx Operator Precedence • Parentheses • Kleene star (*) • Concatenation a ∗ (b | c)a ∗ | a 39 / 78

RegEx Operator Precedence • Parentheses • Kleene star (*) • Concatenation a ∗ (b | c)a ∗ | a 40 / 78

RegEx Operator Precedence • Parentheses • Kleene star (*) • Concatenation • Alternation ( | ) a ∗ (b | c)a ∗ | a 41 / 78

RegEx Operator Precedence • Parentheses • Kleene star (*) • Concatenation • Alternation ( | ) a ∗ (b | c)a ∗ | a 42 / 78

Examples of Regular Expressions-1 Σ = { 0, 1 } 43 / 78

Examples of Regular Expressions-1 Σ = { 0, 1 } • 01 ∗ 44 / 78

Examples of Regular Expressions-1 Σ = { 0, 1 } • 01 ∗ 0 followed by any number of 1s (i.e., { 0, 01, 011, 0111, . . . } ) 45 / 78

Examples of Regular Expressions-1 Σ = { 0, 1 } • 01 ∗ 0 followed by any number of 1s (i.e., { 0, 01, 011, 0111, . . . } ) • (01 ∗ )(01) 46 / 78

Examples of Regular Expressions-1 Σ = { 0, 1 } • 01 ∗ 0 followed by any number of 1s (i.e., { 0, 01, 011, 0111, . . . } ) • (01 ∗ )(01) 0 followed by any number of 1s then 01 (i.e., { 001, 0101, 01101, 011101, . . . } ) 47 / 78

Examples of Regular Expressions-1 Σ = { 0, 1 } • 01 ∗ 0 followed by any number of 1s (i.e., { 0, 01, 011, 0111, . . . } ) • (01 ∗ )(01) 0 followed by any number of 1s then 01 (i.e., { 001, 0101, 01101, 011101, . . . } ) • (0|1) ∗ 0 48 / 78