1 Specifying Tokens with SableCC Recognizing Tokens with DFAs - - PowerPoint PPT Presentation

1

CS553 Lecture Scanning and Parsing 2

Scanning and Parsing

Announcements – Project 1 is 5% of total grade – Project 2 is 10% of total grade – Project 3 is 15% of total grade – Project 4 is 10% of total grade Today – Outline of planned topics for course – Overall structure of a compiler – Lexical analysis (scanning) – Syntactic analysis (parsing)

CS553 Lecture Scanning and Parsing 3

Structure of a Typical Interpreter

“sentences” Synthesis

• ptimization

code generation target language IR IR code generation IR Analysis character stream lexical analysis “words” tokens semantic analysis syntactic analysis AST annotated AST interpreter

Compiler

CS553 Lecture Scanning and Parsing 4

Lexical Analysis (Scanning)

– Tokens, lexemes, and patterns – Lexical analyzers are usually automatically generated from patterns (regular expressions) (e.g., lex)

“.*” “hi”, “mom” string [0-9]+ | [0-9]*.[0-9]+ 3.14159,570 number [a-zA-Z_]+[a-zA-Z0-9_]* foo,index identifier < | <= | = | != | ... <,<=,=,!=,... relation if if if const const const pattern lexeme(s) token const pi := 3.14159 ⇒ const, identifier(pi), assign,number(3.14159)

CS553 Lecture Scanning and Parsing 5

Interaction Between Scanning and Parsing

Lexical analyzer Parser character stream lexer.next() lexer.peek() token parse tree

• r AST

2

CS553 Lecture Scanning and Parsing 6

Specifying Tokens with SableCC

– Regular expressions, formal languages, grammars, parsing…

(not_star_slash not_star*)?)* '*/';

eol | tab)+;

CS553 Lecture Scanning and Parsing 7

Recognizing Tokens with DFAs

– Longest match – Rule priority

letter or digit letter 1 2 f i 1 4 5 t_if t_id

CS553 Lecture Scanning and Parsing 8

– Limited to syntactic structure (⇒ high-level) – Structure usually represented with an abstract syntax tree (AST) – Parsers are usually automatically generated from context-free grammars (e.g., yacc, bison, cup, javacc, sablecc)

Syntactic Analysis (Parsing)

for i 1 10 asg a i tms x 5 arr

CS553 Lecture Scanning and Parsing 9

Interaction Between Scanning and Parsing

Lexical analyzer Parser character stream lexer.next() lexer.peek() token parse tree

• r AST

3

CS553 Lecture Scanning and Parsing 10

Bottom-Up Parsing: Shift-Reduce

– LALR(1): look-ahead, left-to-right, rightmost derivation in reverse, 1 symbol lookahead – LALR is a parsing table construction method, smaller tables than canonical LR

(1) S -> E (2) E -> E + T (3) E -> T (4) T -> id

Grammer

S -> E

• > E + T
• > E + id
• > E + T + id
• > E + id + id
• > T + id + id
• > id + id + id

a + b + c

CS553 Lecture Scanning and Parsing 11

Shift-Reduce Parsing Example

(1) S -> E (2) E -> E + T (3) E -> T (4) T -> id

Stack Input Action

accept $ S reduce (1) $ E reduce (2) $ E + T reduce (4) $ E + c shift c $ E + shift + c $ E reduce (2) + c $ E + T reduce (4) + c $ E + b shift b + c $ E + shift + b + c $ E reduce (3) + b + c $ T reduce (4) + b + c $ a shift a + b + c $

CS553 Lecture Scanning and Parsing 12

Shift-Reduce Parsing Example (precedence problem)

(1) S -> E (2) E -> E + T (3) E -> E * T (4) E -> T (5) T -> id

Stack Input Action

shift a + b * c $

CS553 Lecture Scanning and Parsing 13

Syntax-directed Translation: AST Construction example

S: E { $$ = $1; }; E: E ‘+’ T { $$ = new node(“+”, $1, $3); } | T { $$ = $1; } ; T: T_ID { $$ = new leaf(“id”, $1); };

S E E T + a a b b c c T_ID T_ID T_ID T T + E + +

4

CS553 Lecture Scanning and Parsing 14

Using SableCC to specify grammar and generate AST

cst_main_class.main_class,[cst_class_decl.class_decl])} ;

t_id [args]:exp* | ...

CS553 Lecture Scanning and Parsing 15

Parsing Terms

– production rule – terminal – nonterminal – FOLLOW(X): “the set of terminals that can immediately follow X”

CS553 Lecture Scanning and Parsing 16

Parsing Terms cont …

– LL(1): left-to-right reading of tokens, leftmost derivation, 1 symbol look-ahead – Predictive parser: an efficient non-backtracking top-down parser that can handle LL(1) – More generally recursive descent parsing may involve backtracking

– LR(1): left-to-right reading of tokens, rightmost derivation in reverse, 1 symbol lookahead – Shift-reduce parsers: for example, bison, yacc, and SableCC generated parsers – Methods for producing an LR parsing table – SLR, simple LR – Canonical LR, most powerful – LALR(1)

CS553 Lecture Scanning and Parsing 17

Concepts

– Scanning, parsing, semantic analysis, intermediate code generation,

• ptimization, code generation

– Tools: SableCC, lex, flex, etc.

– Tools: SableCC, yacc, bison, etc.

5

CS553 Lecture Scanning and Parsing 18

Next Time

Lecture – More undergraduate compilers review

CS553 Lecture Scanning and Parsing 19

Language Implementation Timeline

Flow-sens. defined [Banning] Itanium ships & Jikes RVM [IBM] CS553 @ CSU

‘80 ‘90 2000 2010

Sparse cond. const. [Wegman&Zadeck] Superblock scheduling [Hwu] Java [Gosling&Sun] Trace sched. [Fisher] Coloring reg. alloc. [Chaitin] 1st RISC (IBM 801), Wolfe’s thesis C++ [Stroustrup] Dragon book [ASU] PDG [Ferante] Perl [Wall] SW pipelining [Lam] SSA [Cytron] 486 w/ cache Smalltalk [Kay] & PFC [Kennedy]

‘50 ‘60 ‘70 ‘80

A-0 [Hopper] Fortran [Backus] Algol [Comm.] LISP [McCarthy] COBOL [Short Range Comm.] Parser generators Simula [Dahl & Nygaard] BASIC [Kemeny & Kurtz] Value numbering [Cocke&Schwartz] Copying GC [Cheney] Pascal [Wirth] & 1st uproc [4004] C [Ritchie] & ML [Milner et al.] Prolog [Colmeraurer] Modern DFA [Kildall] & Lamport’s parallelism Lex & YACC [Johnson] GCD test [Banerjee & Towle] Parafrase [Kuck] May v. must [Barth] PRE [Morel et al.]

For entertainment purposes only!

• Dep. vectors [Karp et al.]

Ocaml [INRIA]