Is Sound Gradual Typing Dead? Asumu Takikawa Daniel Feltey Ben - - PowerPoint PPT Presentation

SLIDE 1

Is Sound Gradual Typing Dead?

Asumu Takikawa Daniel Feltey Ben Greenman Max S. New Jan Vitek Matthias Felleisen

SLIDE 2

Gradual typing thesis

• 1. People write untyped code
• 2. Static types help maintain software
• 3. Sound types can be added incrementally
• 4. Types respect existing code & the result is runnable

SLIDE 3

Sound types

SLIDE 4

The meaning of soundness

Unsound Typed

#lang typed/racket/unsound ; fact.rkt (provide fact) (: fact (-> Integer Integer)) (define (fact n) (if (zero? n) 1 (* n (sub1 n))))

Untyped

#lang racket ; use.rkt (require "fact.rkt") (fact "ill-typed call")

SLIDE 5

The meaning of soundness

Unsound Typed

#lang typed/racket/unsound ; fact.rkt (provide fact) (: fact (-> Integer Integer)) (define (fact n) (if (zero? n) 1 (* n (sub1 n))))

Untyped

#lang racket ; use.rkt (require "fact.rkt") (fact "ill-typed call") ; zero?: contract violation ; expected: number? ; given: "ill-typed call" ; [,bt for context]

SLIDE 6

The meaning of soundness

Unsound Typed

#lang typed/racket ; fact.rkt (provide fact) (: fact (-> Integer Integer)) (define (fact n) (if (zero? n) 1 (* n (sub1 n))))

Untyped

#lang racket ; use.rkt (require "fact.rkt") (fact "ill-typed call")

SLIDE 7

The meaning of soundness

Unsound Typed

#lang typed/racket ; fact.rkt (provide fact) (: fact (-> Integer Integer)) (define (fact n) (if (zero? n) 1 (* n (sub1 n))))

Untyped

#lang racket ; use.rkt (require "fact.rkt") (fact "ill-typed call") ; fact: contract violation ; expected: Integer ; given: "ill-typed call" ; in: the 1st argument of ; (-> Integer any) ; contract from: "fact.rkt" ; blaming: "use.rkt"

SLIDE 8

Results are runnable

SLIDE 9

#lang racket/base (provide (struct-out stream) make-stream stream-unfold stream-get stream-take) (struct stream (first rest)) (define (make-stream hd thunk) (stream hd thunk)) (define (stream-unfold st) (values (stream-first st) ((stream-rest st)))) (define (stream-get st i) (define-values (hd tl) (stream-unfold st)) (cond [(= i 0) hd] [else (stream-get tl (sub1 i))])) (define (stream-take st n) (cond [(= n 0) '()] [else (define-values (hd tl) (stream-unfold st)) (cons hd (stream-take tl (sub1 n)))])) #lang racket/base (require "streams.rkt") (define (count-from n) (make-stream n (lambda () (count-from (add1 n))))) (define (sift n st) (define-values (hd tl) (stream-unfold st)) (cond [(= 0 (modulo hd n)) (sift n tl)] [else (make-stream hd (lambda () (sift n tl)))])) (define (sieve st) (define-values (hd tl) (stream-unfold st)) (make-stream hd (lambda () (sieve (sift hd tl))))) (define primes (sieve (count-from 2))) (define (main) (printf "The ~a-th prime number is: ~a\n" 100 (stream-get primes 99))) (time (main))

Prime number sieve

SLIDE 10

#lang racket/base (provide (struct-out stream) make-stream stream-unfold stream-get stream-take) (struct stream (first rest)) (define (make-stream hd thunk) (stream hd thunk)) (define (stream-unfold st) (values (stream-first st) ((stream-rest st)))) (define (stream-get st i) (define-values (hd tl) (stream-unfold st)) (cond [(= i 0) hd] [else (stream-get tl (sub1 i))])) (define (stream-take st n) (cond [(= n 0) '()] [else (define-values (hd tl) (stream-unfold st)) (cons hd (stream-take tl (sub1 n)))])) #lang racket/base (require "streams.rkt") (define (count-from n) (make-stream n (lambda () (count-from (add1 n))))) (define (sift n st) (define-values (hd tl) (stream-unfold st)) (cond [(= 0 (modulo hd n)) (sift n tl)] [else (make-stream hd (lambda () (sift n tl)))])) (define (sieve st) (define-values (hd tl) (stream-unfold st)) (make-stream hd (lambda () (sieve (sift hd tl))))) (define primes (sieve (count-from 2))) (define (main) (printf "The ~a-th prime number is: ~a\n" 100 (stream-get primes 99))) (time (main))

Prime number sieve

SLIDE 11

A Typed Racket demo - prime number sieve

#lang typed/racket/base (provide (struct-out stream) make-stream stream-unfold stream-get stream-take) (struct: stream ([first : Natural] [rest : (-> stream)])) (: make-stream (-> Natural (-> stream) stream)) (define (make-stream hd thunk) (stream hd thunk)) (: stream-unfold (-> stream (values Natural stream))) (define (stream-unfold st) (values (stream-first st) ((stream-rest st)))) (: stream-get (-> stream Natural Natural)) (define (stream-get st i) (define-values (hd tl) (stream-unfold st)) (cond [(= i 0) hd] [else (stream-get tl (sub1 i))])) (: stream-take (-> stream Natural (Listof Natural))) (define (stream-take st n) (cond [(= n 0) '()] [else (define-values (hd tl) (stream-unfold st)) (cons hd (stream-take tl (sub1 n)))])) #lang racket/base (require "streams.rkt") (define (count-from n) (make-stream n (lambda () (count-from (add1 n))))) (define (sift n st) (define-values (hd tl) (stream-unfold st)) (cond [(= 0 (modulo hd n)) (sift n tl)] [else (make-stream hd (lambda () (sift n tl)))])) (define (sieve st) (define-values (hd tl) (stream-unfold st)) (make-stream hd (lambda () (sieve (sift hd tl))))) (define primes (sieve (count-from 2))) (define (main) (printf "The ~a-th prime number is: ~a\n" 100 (stream-get primes 99))) (time (main))

Prime number sieve

SLIDE 12

A Typed Racket demo - prime number sieve

#lang typed/racket/base (provide (struct-out stream) make-stream stream-unfold stream-get stream-take) (struct: stream ([first : Natural] [rest : (-> stream)])) (: make-stream (-> Natural (-> stream) stream)) (define (make-stream hd thunk) (stream hd thunk)) (: stream-unfold (-> stream (values Natural stream))) (define (stream-unfold st) (values (stream-first st) ((stream-rest st)))) (: stream-get (-> stream Natural Natural)) (define (stream-get st i) (define-values (hd tl) (stream-unfold st)) (cond [(= i 0) hd] [else (stream-get tl (sub1 i))])) (: stream-take (-> stream Natural (Listof Natural))) (define (stream-take st n) (cond [(= n 0) '()] [else (define-values (hd tl) (stream-unfold st)) (cons hd (stream-take tl (sub1 n)))])) #lang racket/base (require "streams.rkt") (define (count-from n) (make-stream n (lambda () (count-from (add1 n))))) (define (sift n st) (define-values (hd tl) (stream-unfold st)) (cond [(= 0 (modulo hd n)) (sift n tl)] [else (make-stream hd (lambda () (sift n tl)))])) (define (sieve st) (define-values (hd tl) (stream-unfold st)) (make-stream hd (lambda () (sieve (sift hd tl))))) (define primes (sieve (count-from 2))) (define (main) (printf "The ~a-th prime number is: ~a\n" 100 (stream-get primes 99))) (time (main))

Prime number sieve

SLIDE 13

10x slowdown could make the software undeliverable

SLIDE 14

Anecdotes from users

“The end-product appears to be a 50%-performance hybrid due to boundary contracts”

2x

“At this point, about one-fifth of my code is now typed. Unfortunately, this version is 2.5 times slower”

2.5x

“On my machine, it takes *twelve seconds* ... ... the time taken is 1ms”

12,000x

SLIDE 15

“As a practitioner, there are costs associated with using TR, therefore it has to provide equivalent performance improvements to be worthwhile at all.” — Matthew Butterick

SLIDE 16

Why is it slow?

Bad programming / isolated incidents? Bad implementation / design? Fundamental issue with gradual typing?

SLIDE 17

To answer, we need an evaluation method

SLIDE 18

Contributions of our paper

• Evaluation method for language implementors
• Idea for graphically summarizing evaluation results
• Results of evaluating Typed Racket using the method

SLIDE 19

Key Concepts

SLIDE 20

Programmers add types incrementally so should the evaluation method

SLIDE 21

Suffixtree benchmark with 6 modules

SLIDE 22

SLIDE 23

SLIDE 24

Reminder: incremental addition of types

SLIDE 25

SLIDE 26

SLIDE 27

SLIDE 28

SLIDE 29

SLIDE 30

SLIDE 31

SLIDE 32

SLIDE 33

SLIDE 34

SLIDE 35

SLIDE 36

SLIDE 37

SLIDE 38

SLIDE 39

The performance lattice

Paths in lattice are gradual migration paths

SLIDE 40

Why are the configs useful?

Reveals the cost of boundaries in gradual programs Shows paths from untyped to typed

SLIDE 41

data label lcs main structs ukkonen

231570 ms / 88.72x

data label lcs main structs ukkonen

195755 ms / 75x

Data / Label boundary is costly

SLIDE 42

data label lcs main structs ukkonen

33294 ms / 12.76x

data label lcs main structs ukkonen

22203 ms / 8.51x

When Data / Label have same color, it's more ok

SLIDE 43

The visualization has some limitations

SLIDE 44

Which one is better?

a b c d

8x

a b c d

1.25x

a b c d

10.38x

a b c d

1.25x

a b c d

7x

a b c d

2.63x

a b c d

5.13x

a b c d

5.38x

a b c d

2.75x

a b c d

8.38x

a b c d

11.88x

a b c d

2.88x

a b c d

11.5x

a b c d

5.88x

a b c d

2.25x

a b c d

1x

Version 1

a b c d

0.08x

a b c d

0.71x

a b c d

0.54x

a b c d

1.26x

a b c d

1.12x

a b c d

1.24x

a b c d

0.85x

a b c d

0.59x

a b c d

0.31x

a b c d

0.14x

a b c d

1.1x

a b c d

0.19x

a b c d

1.04x

a b c d

0.87x

a b c d

0.95x

a b c d

1x

Version 2

SLIDE 45

Which one is better?

a b c d

8x

a b c d

1.25x

a b c d

10.38x

a b c d

1.25x

a b c d

7x

a b c d

2.63x

a b c d

5.13x

a b c d

5.38x

a b c d

2.75x

a b c d

8.38x

a b c d

11.88x

a b c d

2.88x

a b c d

11.5x

a b c d

5.88x

a b c d

2.25x

a b c d

1x

Version 1

a b c d

0.08x

a b c d

0.71x

a b c d

0.54x

a b c d

1.26x

a b c d

1.12x

a b c d

1.24x

a b c d

0.85x

a b c d

0.59x

a b c d

0.31x

a b c d

0.14x

a b c d

1.1x

a b c d

0.19x

a b c d

1.04x

a b c d

0.87x

a b c d

0.95x

a b c d

1x

Version 2 Summarize by proportion of “deliverable” configurations

SLIDE 46

A configuration is N-deliverable if its overhead factor ≤ Nx

SLIDE 47 data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen

1.1-deliverable proportion: 6%

SLIDE 48 data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen

3-deliverable proportion: 9%

SLIDE 49 data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen

5-deliverable proportion: 19%

SLIDE 50 data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen

10-deliverable proportion: 22%

SLIDE 51 data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen data label lcs main structs ukkonen

20-deliverable proportion: 38%

Even at 20x, no paths from untyped to typed

SLIDE 52

Visualize N-deliverable parameter with a CDF

SLIDE 53

Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable 1x 1x 1x 1x 1x 1x 1x 1x 1x 6x 6x 6x 6x 6x 6x 6x 6x 6x 10x 10x 10x 10x 10x 10x 10x 10x 10x 15x 15x 15x 15x 15x 15x 15x 15x 15x 20x 20x 20x 20x 20x 20x 20x 20x 20x 20 20 20 20 20 20 20 20 20 40 40 40 40 40 40 40 40 40 60 60 60 60 60 60 60 60 60 80 80 80 80 80 80 80 80 80 100 100 100 100 100 100 100 100 100

Suffixtree CDF

Green line is at 3x-deliverable Shallow slope = bad

SLIDE 54

Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable 1x 1x 1x 1x 1x 1x 1x 1x 1x 6x 6x 6x 6x 6x 6x 6x 6x 6x 10x 10x 10x 10x 10x 10x 10x 10x 10x 15x 15x 15x 15x 15x 15x 15x 15x 15x 20x 20x 20x 20x 20x 20x 20x 20x 20x 20 20 20 20 20 20 20 20 20 40 40 40 40 40 40 40 40 40 60 60 60 60 60 60 60 60 60 80 80 80 80 80 80 80 80 80 100 100 100 100 100 100 100 100 100

Gregor CDF

Steep slope = good

SLIDE 55 a b c d

4.14x

a b c d

2.09x

a b c d

3.09x

a b c d

4.18x

a b c d

3.82x

a b c d

2.86x

a b c d

4.14x

a b c d

0.95x

a b c d

3.14x

a b c d

1.05x

a b c d

2.27x

a b c d

3.27x

a b c d

2.64x

a b c d

1.23x

a b c d

0.32x

a b c d

1x

a b c d

0.64x

a b c d

0.04x

a b c d

0.76x

a b c d

0.33x

a b c d

0.42x

a b c d

0.95x

a b c d

0.65x

a b c d

0.11x

a b c d

0.56x

a b c d

1.02x

a b c d

0.19x

a b c d

0.48x

a b c d

0.37x

a b c d

0.46x

a b c d

0.49x

a b c d

1x

Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable 1x 1x 1x 1x 1x 1x 1x 1x 1x 6x 6x 6x 6x 6x 6x 6x 6x 6x 10x 10x 10x 10x 10x 10x 10x 10x 10x 15x 15x 15x 15x 15x 15x 15x 15x 15x 20x 20x 20x 20x 20x 20x 20x 20x 20x 20 20 20 20 20 20 20 20 20 40 40 40 40 40 40 40 40 40 60 60 60 60 60 60 60 60 60 80 80 80 80 80 80 80 80 80 100 100 100 100 100 100 100 100 100 Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable 1x 1x 1x 1x 1x 1x 1x 1x 1x 6x 6x 6x 6x 6x 6x 6x 6x 6x 10x 10x 10x 10x 10x 10x 10x 10x 10x 15x 15x 15x 15x 15x 15x 15x 15x 15x 20x 20x 20x 20x 20x 20x 20x 20x 20x 20 20 20 20 20 20 20 20 20 40 40 40 40 40 40 40 40 40 60 60 60 60 60 60 60 60 60 80 80 80 80 80 80 80 80 80 100 100 100 100 100 100 100 100 100

SLIDE 56

Summary of approach

• Construct performance lattices for benchmarks
• Inspect lattices manually when feasible
• Compare lattices with N-deliverable CDF

SLIDE 57

Results

SLIDE 58

Measured 12 curated benchmarks on all configs 5 are user-written libraries & programs 5 are educational programs 2 were written for this paper

SLIDE 59

Ran a total of 75844 configurations Took 3 months to run

SLIDE 60

3-deliverable proportions

Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable 1x 1x 1x 1x 1x 1x 1x 1x 1x 6x 6x 6x 6x 6x 6x 6x 6x 6x 10x 10x 10x 10x 10x 10x 10x 10x 10x 15x 15x 15x 15x 15x 15x 15x 15x 15x 20x 20x 20x 20x 20x 20x 20x 20x 20x 20 20 20 20 20 20 20 20 20 40 40 40 40 40 40 40 40 40 60 60 60 60 60 60 60 60 60 80 80 80 80 80 80 80 80 80 100 100 100 100 100 100 100 100 100

Synth

1%

Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable 1x 1x 1x 1x 1x 1x 1x 1x 1x 6x 6x 6x 6x 6x 6x 6x 6x 6x 10x 10x 10x 10x 10x 10x 10x 10x 10x 15x 15x 15x 15x 15x 15x 15x 15x 15x 20x 20x 20x 20x 20x 20x 20x 20x 20x 20 20 20 20 20 20 20 20 20 40 40 40 40 40 40 40 40 40 60 60 60 60 60 60 60 60 60 80 80 80 80 80 80 80 80 80 100 100 100 100 100 100 100 100 100

Snake

2%

Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable 1x 1x 1x 1x 1x 1x 1x 1x 1x 6x 6x 6x 6x 6x 6x 6x 6x 6x 10x 10x 10x 10x 10x 10x 10x 10x 10x 15x 15x 15x 15x 15x 15x 15x 15x 15x 20x 20x 20x 20x 20x 20x 20x 20x 20x 20 20 20 20 20 20 20 20 20 40 40 40 40 40 40 40 40 40 60 60 60 60 60 60 60 60 60 80 80 80 80 80 80 80 80 80 100 100 100 100 100 100 100 100 100

Quad

3%

Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable 1x 1x 1x 1x 1x 1x 1x 1x 1x 6x 6x 6x 6x 6x 6x 6x 6x 6x 10x 10x 10x 10x 10x 10x 10x 10x 10x 15x 15x 15x 15x 15x 15x 15x 15x 15x 20x 20x 20x 20x 20x 20x 20x 20x 20x 20 20 20 20 20 20 20 20 20 40 40 40 40 40 40 40 40 40 60 60 60 60 60 60 60 60 60 80 80 80 80 80 80 80 80 80 100 100 100 100 100 100 100 100 100

Suffixtree

9%

Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable 1x 1x 1x 1x 1x 1x 1x 1x 1x 6x 6x 6x 6x 6x 6x 6x 6x 6x 10x 10x 10x 10x 10x 10x 10x 10x 10x 15x 15x 15x 15x 15x 15x 15x 15x 15x 20x 20x 20x 20x 20x 20x 20x 20x 20x 20 20 20 20 20 20 20 20 20 40 40 40 40 40 40 40 40 40 60 60 60 60 60 60 60 60 60 80 80 80 80 80 80 80 80 80 100 100 100 100 100 100 100 100 100

Tetris

25%

Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable 1x 1x 1x 1x 1x 1x 1x 1x 1x 6x 6x 6x 6x 6x 6x 6x 6x 6x 10x 10x 10x 10x 10x 10x 10x 10x 10x 15x 15x 15x 15x 15x 15x 15x 15x 15x 20x 20x 20x 20x 20x 20x 20x 20x 20x 20 20 20 20 20 20 20 20 20 40 40 40 40 40 40 40 40 40 60 60 60 60 60 60 60 60 60 80 80 80 80 80 80 80 80 80 100 100 100 100 100 100 100 100 100

KCFA

25%

Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable 1x 1x 1x 1x 1x 1x 1x 1x 1x 6x 6x 6x 6x 6x 6x 6x 6x 6x 10x 10x 10x 10x 10x 10x 10x 10x 10x 15x 15x 15x 15x 15x 15x 15x 15x 15x 20x 20x 20x 20x 20x 20x 20x 20x 20x 20 20 20 20 20 20 20 20 20 40 40 40 40 40 40 40 40 40 60 60 60 60 60 60 60 60 60 80 80 80 80 80 80 80 80 80 100 100 100 100 100 100 100 100 100

Sieve

50%

Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable 1x 1x 1x 1x 1x 1x 1x 1x 1x 6x 6x 6x 6x 6x 6x 6x 6x 6x 10x 10x 10x 10x 10x 10x 10x 10x 10x 15x 15x 15x 15x 15x 15x 15x 15x 15x 20x 20x 20x 20x 20x 20x 20x 20x 20x 20 20 20 20 20 20 20 20 20 40 40 40 40 40 40 40 40 40 60 60 60 60 60 60 60 60 60 80 80 80 80 80 80 80 80 80 100 100 100 100 100 100 100 100 100

Zordoz

62%

Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable 1x 1x 1x 1x 1x 1x 1x 1x 1x 6x 6x 6x 6x 6x 6x 6x 6x 6x 10x 10x 10x 10x 10x 10x 10x 10x 10x 15x 15x 15x 15x 15x 15x 15x 15x 15x 20x 20x 20x 20x 20x 20x 20x 20x 20x 20 20 20 20 20 20 20 20 20 40 40 40 40 40 40 40 40 40 60 60 60 60 60 60 60 60 60 80 80 80 80 80 80 80 80 80 100 100 100 100 100 100 100 100 100

Gregor

69%

Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable 1x 1x 1x 1x 1x 1x 1x 1x 1x 6x 6x 6x 6x 6x 6x 6x 6x 6x 10x 10x 10x 10x 10x 10x 10x 10x 10x 15x 15x 15x 15x 15x 15x 15x 15x 15x 20x 20x 20x 20x 20x 20x 20x 20x 20x 20 20 20 20 20 20 20 20 20 40 40 40 40 40 40 40 40 40 60 60 60 60 60 60 60 60 60 80 80 80 80 80 80 80 80 80 100 100 100 100 100 100 100 100 100

MBTA

100%

Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable 1x 1x 1x 1x 1x 1x 1x 1x 1x 6x 6x 6x 6x 6x 6x 6x 6x 6x 10x 10x 10x 10x 10x 10x 10x 10x 10x 15x 15x 15x 15x 15x 15x 15x 15x 15x 20x 20x 20x 20x 20x 20x 20x 20x 20x 20 20 20 20 20 20 20 20 20 40 40 40 40 40 40 40 40 40 60 60 60 60 60 60 60 60 60 80 80 80 80 80 80 80 80 80 100 100 100 100 100 100 100 100 100

LNM

100%

Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable 1x 1x 1x 1x 1x 1x 1x 1x 1x 6x 6x 6x 6x 6x 6x 6x 6x 6x 10x 10x 10x 10x 10x 10x 10x 10x 10x 15x 15x 15x 15x 15x 15x 15x 15x 15x 20x 20x 20x 20x 20x 20x 20x 20x 20x 20 20 20 20 20 20 20 20 20 40 40 40 40 40 40 40 40 40 60 60 60 60 60 60 60 60 60 80 80 80 80 80 80 80 80 80 100 100 100 100 100 100 100 100 100

Morse code

100%

SLIDE 61

1.1-deliverable configs over all benchmarks

283 75844 ≈ 0.4%

SLIDE 62

3-deliverable configs over all benchmarks

7992 75844 ≈ 10.5%

SLIDE 63

Bottom line: most configs not deliverable

Even with liberal 3x-deliverable criterion

SLIDE 64

So, is there hope?

SLIDE 65

Suffixtree improvement

Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable 1x 1x 1x 1x 1x 1x 1x 1x 1x 6x 6x 6x 6x 6x 6x 6x 6x 6x 10x 10x 10x 10x 10x 10x 10x 10x 10x 15x 15x 15x 15x 15x 15x 15x 15x 15x 20x 20x 20x 20x 20x 20x 20x 20x 20x 20 20 20 20 20 20 20 20 20 40 40 40 40 40 40 40 40 40 60 60 60 60 60 60 60 60 60 80 80 80 80 80 80 80 80 80 100 100 100 100 100 100 100 100 100

6.2

Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable 1x 1x 1x 1x 1x 1x 1x 1x 1x 6x 6x 6x 6x 6x 6x 6x 6x 6x 10x 10x 10x 10x 10x 10x 10x 10x 10x 15x 15x 15x 15x 15x 15x 15x 15x 15x 20x 20x 20x 20x 20x 20x 20x 20x 20x 20 20 20 20 20 20 20 20 20 40 40 40 40 40 40 40 40 40 60 60 60 60 60 60 60 60 60 80 80 80 80 80 80 80 80 80 100 100 100 100 100 100 100 100 100

6.4.0.4 9% to 19% improvement in 3-deliverability

SLIDE 66

KCFA improvement

Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable 1x 1x 1x 1x 1x 1x 1x 1x 1x 6x 6x 6x 6x 6x 6x 6x 6x 6x 10x 10x 10x 10x 10x 10x 10x 10x 10x 15x 15x 15x 15x 15x 15x 15x 15x 15x 20x 20x 20x 20x 20x 20x 20x 20x 20x 20 20 20 20 20 20 20 20 20 40 40 40 40 40 40 40 40 40 60 60 60 60 60 60 60 60 60 80 80 80 80 80 80 80 80 80 100 100 100 100 100 100 100 100 100

6.2

Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Overhead (vs. untyped) Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable 1x 1x 1x 1x 1x 1x 1x 1x 1x 6x 6x 6x 6x 6x 6x 6x 6x 6x 10x 10x 10x 10x 10x 10x 10x 10x 10x 15x 15x 15x 15x 15x 15x 15x 15x 15x 20x 20x 20x 20x 20x 20x 20x 20x 20x 20 20 20 20 20 20 20 20 20 40 40 40 40 40 40 40 40 40 60 60 60 60 60 60 60 60 60 80 80 80 80 80 80 80 80 80 100 100 100 100 100 100 100 100 100

6.4.0.4 25% to 29% improvement in 3-deliverability

SLIDE 67

Hope

Evaluation method helps implementors

Helps measure improvements between versions

O verhead (vs. untyped) O verhead (vs. untyped) O verhead (vs. untyped) O verhead (vs. untyped) O verhead (vs. untyped) O verhead (vs. untyped) O verhead (vs. untyped) O verhead (vs. untyped) O verhead (vs. untyped) Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable 1x 1x 1x 1x 1x 1x 1x 1x 1x 6x 6x 6x 6x 6x 6x 6x 6x 6x 10x 10x 10x 10x 10x 10x 10x 10x 10x 15x 15x 15x 15x 15x 15x 15x 15x 15x 20x 20x 20x 20x 20x 20x 20x 20x 20x 20 20 20 20 20 20 20 20 20 40 40 40 40 40 40 40 40 40 60 60 60 60 60 60 60 60 60 80 80 80 80 80 80 80 80 80 100 100 100 100 100 100 100 100 100 O verhead (vs. untyped) O verhead (vs. untyped) O verhead (vs. untyped) O verhead (vs. untyped) O verhead (vs. untyped) O verhead (vs. untyped) O verhead (vs. untyped) O verhead (vs. untyped) O verhead (vs. untyped) Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable Percentage deliverable 1x 1x 1x 1x 1x 1x 1x 1x 1x 6x 6x 6x 6x 6x 6x 6x 6x 6x 10x 10x 10x 10x 10x 10x 10x 10x 10x 15x 15x 15x 15x 15x 15x 15x 15x 15x 20x 20x 20x 20x 20x 20x 20x 20x 20x 20 20 20 20 20 20 20 20 20 40 40 40 40 40 40 40 40 40 60 60 60 60 60 60 60 60 60 80 80 80 80 80 80 80 80 80 100 100 100 100 100 100 100 100 100

Can inspect lattice for bad configs

data label lcs main structs ukkonen

195755 ms / 75x

SLIDE 68

Hope

Tools for avoiding GT performance pitfalls

Initial steps: contract profiler [St-Amour et al 2015]

SLIDE 69

Hope

Evaluation method helps GT system implementors Tools for avoiding GT performance pitfalls

Paper & Datasets:

http://www.ccs.neu.edu/racket/pubs/#popl16-tfgnvf

SLIDE 70

Hope

Evaluation method helps GT system implementors Tools for avoiding GT performance pitfalls

Paper & Datasets:

http://www.ccs.neu.edu/racket/pubs/#popl16-tfgnvf

Thank you!

SLIDE 71

SLIDE 72

Other research implementations of gradual typing

2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 Safe TypeScript StrongScript Gradualtalk Reticulated Python Typed Racket

Challenge: adapt this method to your chosen sound GT system