How to Evaluate the Performance of Gradual Type Systems Ben Greenman - - PowerPoint PPT Presentation

▶
how to evaluate the performance of gradual type systems
SMART_READER_LITE
LIVE PREVIEW

How to Evaluate the Performance of Gradual Type Systems Ben Greenman - - PowerPoint PPT Presentation

Mar 05, 2023 225 likes •701 views

How to Evaluate the Performance of Gradual Type Systems Ben Greenman Asumu Takikawa Max S. New Daniel Feltey Robert Bruce Findler Jan Vitek Matthias Felleisen 1 Contribution: Our paper presents the first systematic method to measure the

slide-1
SLIDE 1

How to Evaluate the Performance of Gradual Type Systems

Ben Greenman Max S. New Robert Bruce Findler Matthias Felleisen Asumu Takikawa Daniel Feltey Jan Vitek

1

slide-2
SLIDE 2

Contribution:

Our paper presents the first systematic method to measure the performance implications

  • f a gradual typing system.

2

slide-3
SLIDE 3

Notation

Program Component Dependency

3

slide-4
SLIDE 4

Gradual Typing

Mixed-Typed Program Statically-typed Component Dynamically-typed Component Type Boundary

4

slide-5
SLIDE 5

Typed-Untyped Interaction

5

slide-6
SLIDE 6

Typed-Untyped Interaction

need Integer

6

slide-7
SLIDE 7

Typed-Untyped Interaction

need Integer 42

7

slide-8
SLIDE 8

Typed-Untyped Interaction

need Integer 42

8

slide-9
SLIDE 9

Typed-Untyped Interaction

9

slide-10
SLIDE 10

Typed-Untyped Interaction

need Integer

1

slide-11
SLIDE 11

Typed-Untyped Interaction

need Integer "NaN"

11

slide-12
SLIDE 12

Typed-Untyped Interaction

need Integer "NaN"

12

slide-13
SLIDE 13

Typed-Untyped Interaction

13

slide-14
SLIDE 14

Typed-Untyped Interaction

need Listof(String)

14

slide-15
SLIDE 15

Typed-Untyped Interaction

need Listof(String) (list "A" "B" 27)

15

slide-16
SLIDE 16

Typed-Untyped Interaction

need Listof(String) (list "A" "B" 27)

16

slide-17
SLIDE 17

Typed-Untyped Interaction

17

slide-18
SLIDE 18

Typed-Untyped Interaction

need Bool->Bool

18

slide-19
SLIDE 19

Typed-Untyped Interaction

need Bool->Bool #<procedure>

19

slide-20
SLIDE 20

Typed-Untyped Interaction

need Bool->Bool #<procedure> Bool?

2

slide-21
SLIDE 21

Typed-Untyped Interaction

Type boundaries impose a run-time cost!

(Some mixed-typed languages do not enforce

  • types. For these languages, the performance
  • f type boundaries is not an issue.)

21

slide-22
SLIDE 22

22

slide-23
SLIDE 23
  • Q. What is the overall cost
  • f boundaries in a gradual

typing system?

23

slide-24
SLIDE 24
  • Q. What is the overall cost
  • f boundaries in a gradual

typing system? Need a method to measure and evaluate the performance implications of a gradual typing system

24

slide-25
SLIDE 25

The Method

25

slide-26
SLIDE 26

26

slide-27
SLIDE 27

548 ms 560 ms 716,637 ms 721,454 709,770 ms 711,000 ms 963 ms 891 ms 592 ms 575 ms 829,779 ms 821,285 ms 829,048 ms 602 ms 529 ms

27

slide-28
SLIDE 28

1x

28

slide-29
SLIDE 29

1x 1x 1,307x 1,294x 1,297x 1x 1x 1x 1x 1,513x 1,498x 1,527x 1,512x 1x 0.97x

29

slide-30
SLIDE 30

1x 1x 1,307x 1,294x 1,297x 1x 1x 1x 1x 1,513x 1,498x 1,527x 1,512x 1x 0.97x

3

slide-31
SLIDE 31

D-deliverable

A configuration is D-deliverable if its performance is no worse than a factor

  • f D slowdown compared to the baseline

and D

+

31

slide-32
SLIDE 32

Method: exhaustive perf. eval.

  • 1. Typed program
  • 2. Measure all configurations
  • 3. Count D-deliverable cfgs.

+

Repeat for other programs

32

slide-33
SLIDE 33

A Method for Presenting the Data

33

slide-34
SLIDE 34

+

D

34

slide-35
SLIDE 35

+

D 100% 0% X>1 1

35

slide-36
SLIDE 36

+

D 100% 0% 20 1

36

slide-37
SLIDE 37

+

D 100% 0% 20 1

37

slide-38
SLIDE 38

Scaling the Method

38

slide-39
SLIDE 39

Exponential Blowup

39

slide-40
SLIDE 40

Exponential Blowup

N components => 2N configurations

4

slide-41
SLIDE 41

Simple Random Sampling

41

slide-42
SLIDE 42

Simple Random Sampling

  • 1. Sample O(N) configurations

N = number of components

42

slide-43
SLIDE 43

Simple Random Sampling

  • 1. Sample O(N) configurations
  • 2. Count D-deliverable cfgs.

in the sample

+

N = number of components

43

slide-44
SLIDE 44

More in Paper

44

slide-45
SLIDE 45
  • justification for O(N) sampling
  • exhaustive method applied to Typed Racket
  • comparison: TR v6.2, v6.3, & v6.4
  • discussion of pathologies

∘ N = number of components ∘ 20 benchmarks, docs.racket-lang.org/gtp-benchmarks ∘ the method quantifies improvements

45

slide-46
SLIDE 46

Thank you

Sam Tobin-Hochstadt

For Typed Racket, and for significant improvements to v6.3, v6.4, and beyond.

46

slide-47
SLIDE 47

How to Evaluate the Performance of Gradual Type Systems

Ben Greenman * Max S. New Robert Bruce Findler Matthias Felleisen Asumu Takikawa Daniel Feltey Jan Vitek

47