To Type or Not to Type: Quantifying Detectable Bugs in JavaScript - - PowerPoint PPT Presentation

▶
to type or not to type quantifying detectable bugs in
SMART_READER_LITE
LIVE PREVIEW

To Type or Not to Type: Quantifying Detectable Bugs in JavaScript - - PowerPoint PPT Presentation

Jun 10, 2023 118 likes •807 views

To Type or Not to Type: Quantifying Detectable Bugs in JavaScript Zheng Gao , Christian Bird , Earl Barr University College London, Microsoft Research Static Typing vs. Dynamic Typing Static Typing vs. Dynamic Typing

slide-1
SLIDE 1

To Type or Not to Type: Quantifying Detectable Bugs in JavaScript

Zheng Gao✢, Christian Bird✵, Earl Barr✢

✢University College London, ✵Microsoft Research

slide-2
SLIDE 2

Static Typing Dynamic Typing vs.

slide-3
SLIDE 3

Static Typing

3 ,91 933 13 33 11 53 53 13

Dynamic Typing vs.

slide-4
SLIDE 4

Static Typing

13

Dynamic Typing vs.

slide-5
SLIDE 5

Engine of the Web

is dynamically typed; has a large set of long-running projects.

{

slide-6
SLIDE 6

Engine of the Web

is dynamically typed; has a large set of long-running projects.

5539

{

slide-7
SLIDE 7

Static Typing for JavaScript

slide-8
SLIDE 8

Static Typing for JavaScript

slide-9
SLIDE 9

Had Static Typing been Used …

…

slide-10
SLIDE 10

…

  • Had Static Typing been Used …
slide-11
SLIDE 11

…

  • Had Static Typing been Used …
slide-12
SLIDE 12

…

  • Had Static Typing been Used …
slide-13
SLIDE 13

Central Finding

slide-14
SLIDE 14

Central Finding

15%

slide-15
SLIDE 15

Central Finding

15%

slide-16
SLIDE 16

Bug Life Cycle

slide-17
SLIDE 17

Bug Life Cycle

edit-time bugs

slide-18
SLIDE 18

Bug Life Cycle

edit-time bugs saved bugs

slide-19
SLIDE 19

Bug Life Cycle

edit-time bugs saved bugs team bugs

slide-20
SLIDE 20

Bug Life Cycle

edit-time bugs saved bugs team bugs field bugs

slide-21
SLIDE 21

Bug Life Cycle

team bugs field bugs

slide-22
SLIDE 22

public bugs

Bug Life Cycle

team bugs field bugs

slide-23
SLIDE 23

Type System Detectable

Definition (ts-detectable): Given a static type system ts, a bug is ts-detectable when

  • 1. adding or changing type annotations causes the

program containing the bug to fail to type check

  • n a line a fix changes.
slide-24
SLIDE 24

Problem

var a = b + 1; var a:boolean = b + 1;

When the type of b is nullable number, annotating to “trivially” triggers a type error.

slide-25
SLIDE 25

Consistency

Definition (Consistency): The added or changed type annotations are consistent with a fixed version of the program containing the bug f, if they carried to f type check, and the type of every annotated term is a supertype of that term’s type when an oracle precisely annotates it in f.

slide-26
SLIDE 26

Type System Detectable

Definition (ts-detectable): Given a static type system ts, a bug is ts-detectable when

  • 1. adding or changing type annotations causes the

program containing the bug to fail to type check

  • n a line a fix changes;
  • 2. the new annotations are consistent with a fixed

version of the program containing the bug.

slide-27
SLIDE 27

Example of Detection

TypeScript throws the following error: Error-free in JavaScript, and unexpectedly displays an string, 30

slide-28
SLIDE 28

Research Question

What percentage of public bugs are detectable under Flow or TypeScript?

slide-29
SLIDE 29

Experiment Overview

patch

slide-30
SLIDE 30

patch

Corpus Collection

slide-31
SLIDE 31

patch

Corpus Collection

  • What is the sample size?
  • How to identify public bugs?
slide-32
SLIDE 32

Corpus Collection

  • What is the sample size?
  • How to identify public bugs?
slide-33
SLIDE 33

Sample Size Calculation

3,910,969 closed bug reports 384 bugs

s: sample size X2: a constant for the confidence level of 95% N: population size, 3910969 P: population proportion, 0.5 d: degree of accuracy, 0.05

slide-34
SLIDE 34

Corpus Collection

  • What is the sample size?
  • How to identify public bugs?
slide-35
SLIDE 35

Bug Identification

Bug Identification

slide-36
SLIDE 36

Bug Identification

Bug Identification Fix Identification

slide-37
SLIDE 37

Bug Identification

Parents are buggy

Bug Identification Fix Identification

slide-38
SLIDE 38

Fix Identification

slide-39
SLIDE 39

Fix Identification

These candidates may include commits that add features or refactor.

slide-40
SLIDE 40

Subjects

general bugs ts-detectable bugs public bugs

slide-41
SLIDE 41

Subjects

general bugs ts-detectable bugs public bugs

slide-42
SLIDE 42

Subjects

general bugs ts-detectable bugs public bugs fixed public bugs

slide-43
SLIDE 43

Subjects

general bugs ts-detectable bugs public bugs fixed public bugs

slide-44
SLIDE 44

Size Statistics of the Corpus

The sizes are in lines of code.

slide-45
SLIDE 45

Methodology

  • Researcher

Project Version History

slide-46
SLIDE 46

Methodology

  • buggy

version fixed version Researcher

Project Version History

slide-47
SLIDE 47

Methodology

  • buggy

version fixed version travel back in time

Project Version History

slide-48
SLIDE 48

Methodology

  • Check out pi-1

Project Version History

slide-49
SLIDE 49

Methodology

  • Gradually annotate pi-1

Project Version History

slide-50
SLIDE 50

Methodology

  • a is the annotation function

Project Version History

Type check a(pi-1)

slide-51
SLIDE 51

patch

Annotation

slide-52
SLIDE 52

patch

Annotation

justify undetectable

N Y Y

type check possibly detectable? explain annotations gradually add annotation detected?

N

slide-53
SLIDE 53

patch

Annotation

justify undetectable

N Y Y

type check possibly detectable? explain annotations gradually add annotation We do not fully annotate the program; we rely on gradual typing to locally, minimally annotate the patched region. detected?

N

slide-54
SLIDE 54

Annotation Sources

bug fixes bug reports project documentation

slide-55
SLIDE 55

Expert Source

slide-56
SLIDE 56

Problem

var t = {x:0, z:1}; t.x = t.y; // the error is y does not exist on t t.x = t.z;

slide-57
SLIDE 57

Problem

var t = {x:0, z:1}; t.x = t.y; // the error is y does not exist on t t.x = t.z;

What is the type of variable t?

slide-58
SLIDE 58

Problem

var t = {x:0, z:1}; t.x = t.y; // the error is y does not exist on t t.x = t.z;

What is the type of variable t? Seems to be {x: number, z: number}?

slide-59
SLIDE 59

Problem

var t = {x:0, z:1}; t.x = t.y; // the error is y does not exist on t t.x = t.z;

What is the type of variable t? Seems to be {x: number, z: number}? Not necessarily!

slide-60
SLIDE 60

Problem

var t = {x:0, z:1}; t.x = t.y; // the error is y does not exist on t t.x = t.z;

What is the type of variable t?

… … … t.x = “a”;

Seems to be {x: number, z: number}? Not necessarily!

slide-61
SLIDE 61

Problem

var t = {x:0, z:1}; t.x = t.y; // the error is y does not exist on t t.x = t.z;

What is the type of variable t?

… … … t.x = “a”;

Seems to be {x: number, z: number}? Not necessarily! Now becomes {x: number | string, z: number}.

slide-62
SLIDE 62

Type Shims

A set of type bindings for the free identifiers that 1). is consistent with but 2). may not exist in a fixed version of the program containing the bug.

slide-63
SLIDE 63

Shim Example

var t = {x:0, z:1}; t.x = t.y; // y does not exist on t t.x = t.z; interface T { x:any; z:any; } var t:T = {x:0, z:1}; t.x = t.y; t.x = t.z;

This shim is consistent, as T must be the supertype.

annotate

slide-64
SLIDE 64

Annotation Quality

84% of the annotated fixed versions type check.

  • we add the same annotations to pi

Project Version History

Type check a(pi)

slide-65
SLIDE 65

Results

Both Flow and TypeScript detect 15% of the collected bugs; the confidence range is [11.5%,18.5%], at a 95% confidence level.

slide-66
SLIDE 66

“That’s shocking. If you could make a change to the way we do development that would reduce the number of bugs being checked in by 10% or more

  • vernight, that’s a no-brainer. Unless it doubles

development time or something, we’d do it.”

  • An engineering manager at Microsoft

Implications

slide-67
SLIDE 67

Experimental Artefacts

http://ttendency.cs.ucl.ac.uk/projects/type_study/index.html

slide-68
SLIDE 68

http://ttendency.cs.ucl.ac.uk/projects/type_study/index.html