Tree-sitter @maxbrunsfeld What is Tree-sitter? Why I wrote - - PowerPoint PPT Presentation

▶
tree sitter
SMART_READER_LITE
LIVE PREVIEW

Tree-sitter @maxbrunsfeld What is Tree-sitter? Why I wrote - - PowerPoint PPT Presentation

May 20, 2023 22 likes •774 views

Tree-sitter @maxbrunsfeld What is Tree-sitter? Why I wrote Tree-sitter What were building with Tree-sitter at GitHub Algorithms behind Tree-sitter What is Tree-si tu er? Uniform parsing of di ff erent languages if (a) c();

slide-1
SLIDE 1

Tree-sitter

@maxbrunsfeld

slide-2
SLIDE 2
  • What is Tree-sitter?
  • Why I wrote Tree-sitter
  • What we’re building with Tree-sitter at GitHub
  • Algorithms behind Tree-sitter
slide-3
SLIDE 3

What is Tree-situer?

slide-4
SLIDE 4

Uniform parsing of different languages

slide-5
SLIDE 5

Incremental parsing

if (a) c();

slide-6
SLIDE 6

Incremental parsing

if (a.b) c();

✨ ✨

slide-7
SLIDE 7

Why I wroteTree-situer

slide-8
SLIDE 8

Single-language code analysis tools

slide-9
SLIDE 9

Syntax Highlighting Today

slide-10
SLIDE 10

Syntax Highlighting Today

Why three different colors for types?

slide-11
SLIDE 11

Syntax Highlighting Today

Why three different colors for types?

slide-12
SLIDE 12

Syntax Highlighting Today

Why three different colors for types?

slide-13
SLIDE 13

Why not use standard parsers?

  • Most are too slow for text editors
  • Many require extra information that’s not available
  • Each has its own dependencies
slide-14
SLIDE 14

Why use Tree-situer

  • It’s fast
  • It requires no extra information
  • Parsers have no dependencies
slide-15
SLIDE 15

What we’re doing with Tree-situer

slide-16
SLIDE 16

Syntax Highlighting - Go

slide-17
SLIDE 17

Syntax Highlighting - C

slide-18
SLIDE 18

Syntax Highlighting - C++

slide-19
SLIDE 19

Syntax Highlighting - TypeScript

slide-20
SLIDE 20

Syntax Highlighting - Python

slide-21
SLIDE 21

Syntax Highlighting - Ruby

slide-22
SLIDE 22

Syntax Highlighting - Rust

slide-23
SLIDE 23

Syntax Highlighting - Long Lines

slide-24
SLIDE 24

Syntax Highlighting - Long Lines

slide-25
SLIDE 25

Syntax Highlighting - Performance

$ wc -l react.dev.js 20599 react.dev.js $ tree-sitter parse react.dev.js --time react.dev.js 69ms

slide-26
SLIDE 26

Code Folding

slide-27
SLIDE 27

Code Folding

Indentation doesn’t always match syntax

slide-28
SLIDE 28

Code Folding

slide-29
SLIDE 29

Extend Selection

slide-30
SLIDE 30

Pull Request Table of Contents

slide-31
SLIDE 31

How Tree-situer Works

slide-32
SLIDE 32

Writing a Grammar

grammar.js

slide-33
SLIDE 33

Writing a Grammar

parser.c

slide-34
SLIDE 34

Writing a Grammar

parser.c

slide-35
SLIDE 35

Using the parser

slide-36
SLIDE 36

Algorithms

slide-37
SLIDE 37

Existing Research

Practical Algorithms for Incremental Software Development Environments Tim A. Wagner 1998

slide-38
SLIDE 38

LR Parsing

slide-39
SLIDE 39

LR Parsing

Text: x * y + z

slide-40
SLIDE 40

Text: x * y + z

LR Parsing

Stack:

slide-41
SLIDE 41

Text: x * y + z

LR Parsing

Stack:

variable

slide-42
SLIDE 42

Text: x * y + z

LR Parsing

Stack:

variable *

slide-43
SLIDE 43

Text: x * y + z

LR Parsing

Stack:

variable * variable

slide-44
SLIDE 44

Text: x * y + z

LR Parsing

Stack:

product variable variable *

slide-45
SLIDE 45

Text: x * y + z

LR Parsing

Stack:

product variable variable * +

slide-46
SLIDE 46

Text: x * y + z

LR Parsing

Stack:

product variable variable * + variable

slide-47
SLIDE 47

Text: x * y + z

LR Parsing

Stack:

sum variable product + variable variable *

slide-48
SLIDE 48

LR Parsing

  • Does this work for all languages?
slide-49
SLIDE 49

LR Parsing

  • Does this work for all languages? Nope.

x = (y); // parenthesized expression x = (y) => z; // arrow function

slide-50
SLIDE 50

x = (y); x = (y) => z;

slide-51
SLIDE 51

GLR Parsing

x = (y) => z;

slide-52
SLIDE 52

GLR Parsing

x = (y) => z;

slide-53
SLIDE 53

GLR Parsing

x = (y) => z;

slide-54
SLIDE 54

Error Recovery

slide-55
SLIDE 55

Error Recovery

slide-56
SLIDE 56

Error Recovery

slide-57
SLIDE 57

Error Recovery

slide-58
SLIDE 58

Text: var a = new B(); a.c(); return a;

Incremental Parsing

slide-59
SLIDE 59

Text: var a = new B(); a.c(d); return a;

Incremental Parsing

slide-60
SLIDE 60

Text: var a = new B(); a.c(d); return a;

Incremental Parsing

Stack:

slide-61
SLIDE 61

Text: var a = new B(); a.c(d); return a;

Incremental Parsing

variable declaration

Stack:

slide-62
SLIDE 62

Text: var a = new B(); a.c(d); return a;

Incremental Parsing

variable declaration

Stack:

member expression

slide-63
SLIDE 63

Text: var a = new B(); a.c(d); return a;

Incremental Parsing

variable declaration

Stack:

member expression ‘(‘

slide-64
SLIDE 64

Text: var a = new B(); a.c(d); return a;

Incremental Parsing

variable declaration

Stack:

member expression ‘(‘ identifier

slide-65
SLIDE 65

Text: var a = new B(); a.c(d); return a;

Incremental Parsing

variable declaration

Stack:

member expression ‘(‘ expression

slide-66
SLIDE 66

Text: var a = new B(); a.c(d); return a;

Incremental Parsing

variable declaration

Stack:

member expression ‘(‘ expression ‘)’

slide-67
SLIDE 67

Text: var a = new B(); a.c(d); return a;

Incremental Parsing

variable declaration

Stack:

member expression arguments

slide-68
SLIDE 68

Text: var a = new B(); a.c(d); return a;

Incremental Parsing

variable declaration

Stack:

call expression

slide-69
SLIDE 69

Text: var a = new B(); a.c(d); return a;

Incremental Parsing

variable declaration

Stack:

call expression ‘;’

slide-70
SLIDE 70

Text: var a = new B(); a.c(d); return a;

Incremental Parsing

variable declaration

Stack:

expression statement

slide-71
SLIDE 71

Text: var a = new B(); a.c(d); return a;

Incremental Parsing

Stack:

program repetition

slide-72
SLIDE 72

Text: var a = new B(); a.c(d); return a;

Incremental Parsing

Stack:

program repetition return statement

slide-73
SLIDE 73

Text: var a = new B(); a.c(d); return a;

Incremental Parsing

Stack:

program

slide-74
SLIDE 74

Next Steps

  • Add support for more languages
  • New Atom features using the syntax trees
  • New GitHub.com features using syntax trees
slide-75
SLIDE 75

Thank Y

  • u!

@maxbrunsfeld