Compiler Development (CMPSC 401) Lexical Analysis Janyl Jumadinova - PowerPoint PPT Presentation

Compiler Development (CMPSC 401) Lexical Analysis Janyl Jumadinova January 29, 2019 Janyl Jumadinova Compiler Development (CMPSC 401) January 29, 2019 1 / 40

Automaton and Regular Expressions Deterministic Finite Automata (DFAs), Non-deterministic Finite Automata (NFAs) and REs have same expressive power i.e. allow precisely same patterns/sets to be specified. For every DFA there is an equivalent RE DFA RE For every NFA For every RE there is there is an an equivalent NFA equivalent DFA NFA Janyl Jumadinova Compiler Development (CMPSC 401) January 29, 2019 2 / 40

Finite State Automaton A finite automaton is a machine that has a finite number of states and a finite number of transitions between these. One marked as initial state. One or more marked as final states. States sometimes labeled or numbered. Janyl Jumadinova Compiler Development (CMPSC 401) January 29, 2019 3 / 40

Finite State Automaton A finite automaton is a machine that has a finite number of states and a finite number of transitions between these. One marked as initial state. One or more marked as final states. States sometimes labeled or numbered. A set of transitions from state to state. Each labeled with symbol from � (the alphabet), or ε . The symbols correspond to characters in the input string. Janyl Jumadinova Compiler Development (CMPSC 401) January 29, 2019 3 / 40

Example Janyl Jumadinova Compiler Development (CMPSC 401) January 29, 2019 4 / 40

Finite State Automaton Operate by reading input symbols (usually characters). Transition can be taken if labeled with current symbol. ε -transition can be taken at any time. Janyl Jumadinova Compiler Development (CMPSC 401) January 29, 2019 18 / 40

Finite State Automaton Operate by reading input symbols (usually characters). Transition can be taken if labeled with current symbol. ε -transition can be taken at any time. Accept when final state reached and no more input. Janyl Jumadinova Compiler Development (CMPSC 401) January 29, 2019 18 / 40

Finite State Automaton Operate by reading input symbols (usually characters). Transition can be taken if labeled with current symbol. ε -transition can be taken at any time. Accept when final state reached and no more input. Slightly different in a scanner, where the FSA is used as a subroutine to find the longest input string that matches a token RE. Janyl Jumadinova Compiler Development (CMPSC 401) January 29, 2019 18 / 40

Finite State Automaton Operate by reading input symbols (usually characters). Transition can be taken if labeled with current symbol. ε -transition can be taken at any time. Accept when final state reached and no more input. Slightly different in a scanner, where the FSA is used as a subroutine to find the longest input string that matches a token RE. Reject if no transition possible, or no more input and not in final state. Janyl Jumadinova Compiler Development (CMPSC 401) January 29, 2019 18 / 40

A More Complex Automaton Janyl Jumadinova Compiler Development (CMPSC 401) January 29, 2019 19 / 40

DFA vs. NFA Deterministic Finite Automata (DFA) No choice of which transition to make. Janyl Jumadinova Compiler Development (CMPSC 401) January 29, 2019 32 / 40

DFA vs. NFA Deterministic Finite Automata (DFA) No choice of which transition to make. Non-deterministic Finite Automata (NFA) Choice of transition in at least one case. Janyl Jumadinova Compiler Development (CMPSC 401) January 29, 2019 32 / 40

DFA vs. NFA Deterministic Finite Automata (DFA) No choice of which transition to make. Non-deterministic Finite Automata (NFA) Choice of transition in at least one case. ε transitions (arcs): If the current state has any outgoing ε arcs, we can follow any of them without consuming any input. Modeling choice option 1: guess path, backtrack if rejects Option 2: “clone” at choice point, accept if any clone accepts. Janyl Jumadinova Compiler Development (CMPSC 401) January 29, 2019 32 / 40

Simulating an NFA For each character in the input: For each current state: - Follow all transitions labeled with the current letter. - Add these states to the set of new states. Janyl Jumadinova Compiler Development (CMPSC 401) January 29, 2019 33 / 40

Simulating an NFA For each character in the input: For each current state: - Follow all transitions labeled with the current letter. - Add these states to the set of new states. Add every state reachable by an ε -move to the set of next states. Janyl Jumadinova Compiler Development (CMPSC 401) January 29, 2019 33 / 40

Simulating an NFA For each character in the input: For each current state: - Follow all transitions labeled with the current letter. - Add these states to the set of new states. Add every state reachable by an ε -move to the set of next states. Accept if some way to reach a final state on given input. Reject if no possible way to final state. Janyl Jumadinova Compiler Development (CMPSC 401) January 29, 2019 33 / 40

FAs in Scanners Want DFA for speed (no backtracking or cloning). But conversion from regular expressions to NFA is easier. Luckily, there is a well-defined procedure for converting an NFA to an equivalent DFA. Janyl Jumadinova Compiler Development (CMPSC 401) January 29, 2019 34 / 40

Usefulness of RE to NFA Construction Lexical Analysis Specify language tokens (identifiers, numerical constants, symbols etc.) as REs. Tools like lex automatically generate automaton-based code to decompose source code into constituent tokens. Pattern Matching e.g. text editors, grep Pattern specified as RE. Automaton-based search locates occurrences. Janyl Jumadinova Compiler Development (CMPSC 401) January 29, 2019 35 / 40

Lexical Analysis Generators Generates analyzer automatically from “descriptions” (regular expressions/ NFAs) of tokens in the programming language. Examples: lex/flex for C jFlex for Java Janyl Jumadinova Compiler Development (CMPSC 401) January 29, 2019 36 / 40

Terminology A token is a group of characters having collective meaning. A lexeme is an actual character sequence forming a specific instance of a token, such as num . A pattern is a rule expressed as a regular expression and describing how a particular token can be formed. For example, [A-Za-z][A-Za-z 0-9]* is a rule. Janyl Jumadinova Compiler Development (CMPSC 401) January 29, 2019 37 / 40

(jF)Lex Input: description of token structure (regular expressions) information on how to “process” different tokens Janyl Jumadinova Compiler Development (CMPSC 401) January 29, 2019 38 / 40

Compiler Development (CMPSC 401) Lexical Analysis Janyl Jumadinova - PowerPoint PPT Presentation

Compiler Development (CMPSC 401) Lexical Analysis Janyl Jumadinova January 29, 2019 Janyl Jumadinova Compiler Development (CMPSC 401) January 29, 2019 1 / 40 Automaton and Regular Expressions Deterministic Finite Automata (DFAs),

Compiler Development (CMPSC 401) ARM Architecture Janyl Jumadinova April 4, 2019 Janyl

Compiler Development (CMPSC 401) Semantic Analysis Janyl Jumadinova March 12, 2019 Janyl

Compiler Development (CMPSC 401) Lexical Analysis Janyl Jumadinova January 24, 2019 Janyl

Compiler Development (CMPSC 401) Syntax Analysis Bottom-Up Parsing Janyl Jumadinova March 5,

Compiler Development (CMPSC 401) Code Optimization Janyl Jumadinova April 15, 2019 Janyl

Compiler Development (CMPSC 401) Three Address Code Janyl Jumadinova April 2, 2019 Janyl

Compiler Development (CMPSC 401) Code Generation, ARM, x86 Janyl Jumadinova April 11, 2019

Compiler Development (CMPSC 401) Lexical Analysis: JFlex Janyl Jumadinova February 5, 2019

Compiler Development (CMPSC 401) Intermediate Representations Janyl Jumadinova March 28, 2019

Compiler Development (CMPSC 401) Syntax Analysis Janyl Jumadinova February 14, 2019 Janyl

Compiler Development (CMPSC 401) Janyl Jumadinova January 17, 2018 Janyl Jumadinova Compiler

Compiler Development (CMPSC 401) Type Checking Janyl Jumadinova March 14, 2019 Janyl Jumadinova

11/8/2012 The Structure of a Compiler (2) The Structure of a Compiler (1) Any compiler must

Compiler Construction Chapter 11 1 Compiler Construction Compiler Construction A New Compiler

CMPSC 311- Introduction to Systems Programming Module: Studying Professor Patrick McDaniel Fall

252-210: Compiler Design 1.1 Simple compiler model 1.2

Lecture 8: Sequential Networks and Finite State Machines CSE 140: Components and Design

Some useful tasks involving language Find all phone numbers in a text, e.g., occurrences such

1 Finite Automata and Regular Expressions Motivation: Given a pattern (regular expression) for

Feb 15 Lecture MAT309, Winter 2018 Announcements: Problem Set #3 posted on website. Due date:

Computer Architecture Summer 2018 Basics of Logic Design: Finite State Machines Tyler Bletsch

Finite State Machines (FSM) AKA Finite State Automat on State Machines Introduction State

Outline Nondeterminism Regular expressions Elementary reductions Computation,

600.405 Finite-State Methods in NLP Assignment 1: Getting Started Prof. J. Eisner Fall