Translating Code Comments to Procedure Specifications Arianna Blasi - - PowerPoint PPT Presentation

translating code comments to procedure specifications
SMART_READER_LITE
LIVE PREVIEW

Translating Code Comments to Procedure Specifications Arianna Blasi - - PowerPoint PPT Presentation

Translating Code Comments to Procedure Specifications Arianna Blasi Alberto Goffi Konstantin Kuznetsov Alessandra Gorla Michael D. Ernst Mauro Pezz Sergio Delgado USI Universit della Svizzera


slide-1
SLIDE 1

Translating Code Comments to Procedure Specifications

Arianna Blasi♦♥ · Alberto Goffi♦ · Konstantin Kuznetsov♣ Alessandra Gorla♥ · Michael D. Ernst♠ · Mauro Pezzè♦ · Sergio Delgado♥

♦USI Università della Svizzera italiana,

Switzerland

♥IMDEA Software Institute,

Spain

♣Saarland University/CISPA,

Germany

♠University of Washington Seattle,

WA, USA

slide-2
SLIDE 2

public static boolean removeAll(List myList){…}

slide-3
SLIDE 3

public static boolean removeAll(List myList){…}

slide-4
SLIDE 4

public static boolean removeAll(List myList){…}

@Test public void testRemoveAll(){ }

slide-5
SLIDE 5

public static boolean removeAll(List myList){…}

@Test public void testRemoveAll(){ } List<Integer> list = Arrays.asList(1, 2, 3);

slide-6
SLIDE 6

public static boolean removeAll(List myList){…}

@Test public void testRemoveAll(){ } List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeAll(list);

slide-7
SLIDE 7

public static boolean removeAll(List myList){…}

@Test public void testRemoveAll(){ } List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeAll(list);

assert ?

slide-8
SLIDE 8

public static boolean removeAll(List myList){…}

@Test public void testRemoveAll(){ } List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeAll(list);

We have no idea

assert ?

slide-9
SLIDE 9
slide-10
SLIDE 10
slide-11
SLIDE 11

Where is the oracle?

slide-12
SLIDE 12

Where is the oracle? Where is the specification?

slide-13
SLIDE 13
slide-14
SLIDE 14

Why not generating executable specifications automatically?

slide-15
SLIDE 15

Why not generating executable specifications automatically? Let’s rely on Javadoc comments!

slide-16
SLIDE 16

/** * @return true if the list is empty */ public static boolean removeAll(List myList) {…}

Our solution

slide-17
SLIDE 17

/** * @return true if the list is empty */ public static boolean removeAll(List myList) {…}

myList.isEmpty() ? result==true

Our solution

slide-18
SLIDE 18

/** * @return true if the list is empty */ public static boolean removeAll(List myList) {…}

myList.isEmpty() ? result==true

Our solution

slide-19
SLIDE 19

/** * @return true if the list is empty */ public static boolean removeAll(List myList) {…}

@Test public void testRemoveAll(){ } List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeAll(list);

myList.isEmpty() ? result==true

Our solution

slide-20
SLIDE 20

/** * @return true if the list is empty */ public static boolean removeAll(List myList) {…}

@Test public void testRemoveAll(){ } List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeAll(list);

myList.isEmpty() ? result==true

if(list.isEmpty()) assert result==true;

Our solution

slide-21
SLIDE 21

/** * @return true if the list is empty */ public static boolean removeAll(List myList) {…}

@Test public void testRemoveAll(){ } List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeAll(list);

myList.isEmpty() ? result==true

if(list.isEmpty()) assert result==true;

Our solution

slide-22
SLIDE 22

The idea is not new...

slide-23
SLIDE 23

The idea is not new... @tComment

Tan et al. ICST 2012

slide-24
SLIDE 24

The idea is not new... @tComment Toradocu

Tan et al. ICST 2012 Goffi et al. ISSTA 2016

slide-25
SLIDE 25

The idea is not new... @tComment Toradocu ALICS

Tan et al. ICST 2012 Goffi et al. ISSTA 2016 Pandita et al. ICSE 2012

slide-26
SLIDE 26

The idea is not new... @tComment Toradocu ALICS

Preconditions Normal Postconditions Exceptional Postconditions

Tan et al. ICST 2012 Goffi et al. ISSTA 2016 Pandita et al. ICSE 2012

slide-27
SLIDE 27

The idea is not new... @tComment Toradocu ALICS

Preconditions Normal Postconditions Exceptional Postconditions

Tan et al. ICST 2012 Goffi et al. ISSTA 2016 Pandita et al. ICSE 2012

slide-28
SLIDE 28

The idea is not new... @tComment Toradocu ALICS

Preconditions Normal Postconditions Exceptional Postconditions

Tan et al. ICST 2012 Goffi et al. ISSTA 2016 Pandita et al. ICSE 2012

Jdoctor

slide-29
SLIDE 29

Source

Javadoc
 Extractor Comment
 Translator

Specs

Jdoctor

slide-30
SLIDE 30

Source

Javadoc
 Extractor

Specs

Comment
 Translator

slide-31
SLIDE 31

Source Source

Javadoc
 Extractor

Specs

Comment
 Translator

  • 1. Expected result
  • 2. Condition (Text)
slide-32
SLIDE 32

/** * @return true * if the list is empty */ public boolean removeAll(List list){…}

  • 1. true
  • 2. “if the list is empty”

Source

Javadoc
 Extractor

Specs

Comment
 Translator

  • 1. Expected result
  • 2. Condition (Text)

Source

slide-33
SLIDE 33

/** * @return true * if the list is empty */ public boolean removeAll(List list){…}

  • 1. true
  • 2. “if the list is empty”

Source

Javadoc
 Extractor

Specs

Comment
 Translator

  • 1. Expected result
  • 2. Condition (Text)
slide-34
SLIDE 34

Source

Javadoc
 Extractor

Specs

Comment
 Translator

  • 1. Expected result
  • 2. Condition (Text)
slide-35
SLIDE 35
  • 1. Expected result
  • 2. Condition (Code)
  • 1. Expected result
  • 2. Condition (Text)

Source

Javadoc
 Extractor

Specs

Comment
 Translator

slide-36
SLIDE 36
  • 1. Expected result
  • 2. Condition (Code)
  • 1. Expected result
  • 2. Condition (Text)
  • 1. true
  • 2. “if the list is empty”
  • 1. true
  • 2. myList.isEmpty()

Source

Javadoc
 Extractor

Specs

Comment
 Translator

slide-37
SLIDE 37
  • 1. Expected result
  • 2. Condition (Code)
  • 1. Expected result
  • 2. Condition (Text)
  • 1. true
  • 2. “if the list is empty”
  • 1. true
  • 2. myList.isEmpty()

Source

Javadoc
 Extractor

Specs

Comment
 Translator

myList.isEmpty() ? result==true

slide-38
SLIDE 38

Java Specification Natural Language Specification

Comment
 Translator

slide-39
SLIDE 39

Comment Translator

Subject Matcher NL Parser Predicate
 Matcher Java Specification Natural Language Specification

slide-40
SLIDE 40

Subject Matcher Predicate
 Matcher NL Parser “if the list is empty”

slide-41
SLIDE 41

“if the list is empty” Subject Matcher Predicate
 Matcher NL Parser

Subject Predicate “list” “is empty”

slide-42
SLIDE 42

Predicate
 Matcher NL Parser Subject Matcher

Subject: “list”

slide-43
SLIDE 43

Predicate
 Matcher NL Parser Subject Matcher

Subject: “list” Candidates

Formal Parameters Class Name Methods Fields

slide-44
SLIDE 44

Predicate
 Matcher NL Parser Subject Matcher

Subject: “list”

Candidate Distance

myList

2

isEmpty

7

iterator

8

size

4 …

Candidates

Formal Parameters Class Name Methods Fields

slide-45
SLIDE 45

Predicate
 Matcher NL Parser Subject Matcher

Subject: “list”

Candidate Distance

myList

2

isEmpty

7

iterator

8

size

4 …

Candidates

Formal Parameters Class Name Methods Fields myList 2

slide-46
SLIDE 46

Predicate
 Matcher NL Parser Subject Matcher

Candidate Distance

myList

2

isEmpty

7

iterator

8

size

4 …

Candidates

Formal Parameters Class Name Methods Fields

Subject: “list” => myList

myList 2

slide-47
SLIDE 47

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

slide-48
SLIDE 48

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

slide-49
SLIDE 49

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

1: Exact pattern match?

slide-50
SLIDE 50

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

is null == null

1: Exact pattern match?

slide-51
SLIDE 51

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

is null == null is negative < 0

1: Exact pattern match?

slide-52
SLIDE 52

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

is null == null is negative < 0 is positive > 0

1: Exact pattern match?

slide-53
SLIDE 53

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

is null == null is negative < 0 is positive > 0 ...

1: Exact pattern match?

slide-54
SLIDE 54

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

is null == null is negative < 0 is positive > 0 ...

1: Exact pattern match?

ALICS Toradocu @tComment

slide-55
SLIDE 55

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

is null == null is negative < 0 is positive > 0 ...

"is empty" not found...

ALICS Toradocu @tComment

slide-56
SLIDE 56

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

slide-57
SLIDE 57

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

2: Syntax match?

Toradocu

slide-58
SLIDE 58

Candidate Distance

myList

9

isEmpty

1

iterator

6

size

7 …

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

2: Syntax match?

Toradocu

slide-59
SLIDE 59

Candidate Distance

myList

9

isEmpty

1

iterator

6

size

7 …

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

2: Syntax match?

Toradocu

isEmpty 1

slide-60
SLIDE 60

Pattern and syntax match are not enough

slide-61
SLIDE 61

Pattern and syntax match are not enough

Return true if the list is empty

slide-62
SLIDE 62

Pattern and syntax match are not enough

Candidate Distance

clear

7

isEmpty

1

iterator

6

size

7 …

Return true if the list is empty

slide-63
SLIDE 63

Pattern and syntax match are not enough

Return true if the list is empty contains no element

slide-64
SLIDE 64

Pattern and syntax match are not enough

Return true if the list is empty contains no element has no content

slide-65
SLIDE 65

Pattern and syntax match are not enough

Return true if the list is empty

Candidate Distance

myList isEmpty iterator size

contains no element has no content

slide-66
SLIDE 66

Pattern and syntax match are not enough

Return true if the list is empty

Candidate Distance

myList isEmpty iterator size

contains no element has no content

slide-67
SLIDE 67

Pattern and syntax match are not enough

Return true if the list is empty

Candidate Distance

myList isEmpty iterator size

contains no element has no content

ALICS Toradocu @tComment

slide-68
SLIDE 68

Analyzing complex comments

/** * Checks if there is a credible threatening unit to * this unit within a range of moves. * * @param moves list of moves * @return True if a threat was found. */ public boolean isInDanger(List<Object> moves)

slide-69
SLIDE 69

Analyzing complex comments

/** * Checks if there is a credible threatening unit to * this unit within a range of moves. * * @param moves list of moves * @return True if a threat was found. */ public boolean isInDanger(List<Object> moves)

slide-70
SLIDE 70

Analyzing complex comments

searchForDanger(List<Object>)

/** * Checks if there is a credible threatening unit to * this unit within a range of moves. * * @param moves list of moves * @return True if a threat was found. */ public boolean isInDanger(List<Object> moves)

slide-71
SLIDE 71

threat was found

search for danger

slide-72
SLIDE 72

threat was found

search for danger

slide-73
SLIDE 73
slide-74
SLIDE 74

“search” is not displayed as a result

slide-75
SLIDE 75

“search” is not displayed as a result We need more than semantic equivalence

slide-76
SLIDE 76

Word embedded in vector space: GloVe, Word2Vec

man king woman queen

Word embedding

slide-77
SLIDE 77

Word embedding

danger threat found search

Word embedded in vector space: GloVe, Word2Vec

slide-78
SLIDE 78

Word embedding

danger threat found search

Word mover's distance (Matt J. Kusner et al., 2015)

slide-79
SLIDE 79

Word embedding

danger threat found search

Word mover's distance (Matt J. Kusner et al., 2015)

slide-80
SLIDE 80

In Chicago the president greets the press Obama speaks in Illinois.

Word Mover's distance

slide-81
SLIDE 81

In Chicago the president greets the press Obama speaks in Illinois.

Word Mover's distance

slide-82
SLIDE 82

In Chicago the president greets the press Obama speaks in Illinois.

Word Mover's distance

slide-83
SLIDE 83

In Chicago the president greets the press Obama speaks in Illinois.

Word Mover's distance

slide-84
SLIDE 84

Search For Danger Threat was found

Word Mover's distance

slide-85
SLIDE 85

Search For Danger Threat was found

Word Mover's distance

slide-86
SLIDE 86

Search For Danger Threat was found

Word Mover's distance

slide-87
SLIDE 87
slide-88
SLIDE 88

How accurate is Jdoctor?

slide-89
SLIDE 89

Popular open-source Java systems

Experimental setup

6

slide-90
SLIDE 90

Analyzed Java methods

Experimental setup

563

slide-91
SLIDE 91

Manually-written Java conditions

Experimental setup

829

slide-92
SLIDE 92
slide-93
SLIDE 93

92 %

Precision

slide-94
SLIDE 94

92 % 83 %

Precision Recall

slide-95
SLIDE 95

Exceptional Postconditions

Precision Recall

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

@tComment Toradocu Jdoctor

slide-96
SLIDE 96

Preconditions

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

@tComment Toradocu Jdoctor

Precision Recall

slide-97
SLIDE 97

Normal Postconditions

Precision Recall

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

@tComment Toradocu Jdoctor

slide-98
SLIDE 98

0% 20% 40% 60% 80% 100%

@tComment Toradocu Jdoctor

Precision Recall

Overall accuracy

slide-99
SLIDE 99
slide-100
SLIDE 100

How can you use Jdoctor?

slide-101
SLIDE 101

Automatic Test Case Generator Jdoctor

+

slide-102
SLIDE 102

Jdoctor

+

slide-103
SLIDE 103

Randoop+Jdoctor reclassification

passing passing failing invalid

slide-104
SLIDE 104

Randoop+Jdoctor reclassification

passing passing failing invalid

slide-105
SLIDE 105

Randoop+Jdoctor reclassification

passing passing failing invalid Jdoctor

+

slide-106
SLIDE 106

Randoop+Jdoctor reclassification

passing invalid passing failing invalid Jdoctor

+

slide-107
SLIDE 107

Randoop+Jdoctor reclassification

passing invalid = invalid passing failing invalid Jdoctor

+

slide-108
SLIDE 108

Randoop+Jdoctor reclassification

passing invalid = invalid passing failing failing invalid Jdoctor

+

slide-109
SLIDE 109

Randoop+Jdoctor reclassification

passing invalid = invalid passing failing = missed alarm failing invalid Jdoctor

+

slide-110
SLIDE 110

Randoop+Jdoctor reclassification

passing invalid = invalid passing failing = missed alarm failing passing invalid Jdoctor

+

slide-111
SLIDE 111

Randoop+Jdoctor reclassification

passing invalid = invalid passing failing = missed alarm failing passing = false alarm invalid Jdoctor

+

slide-112
SLIDE 112

Randoop+Jdoctor reclassification

passing invalid = invalid passing failing = missed alarm failing passing = false alarm invalid passing Jdoctor

+

slide-113
SLIDE 113

Randoop+Jdoctor reclassification

passing invalid = invalid passing failing = missed alarm failing passing = false alarm invalid passing = new test Jdoctor

+

slide-114
SLIDE 114

Missed alarm

public static boolean removeAll(List myList){…}

slide-115
SLIDE 115

@Test public void testRemoveAll(){ } List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeAll(list);

Missed alarm

public static boolean removeAll(List myList){…}

slide-116
SLIDE 116

Missed alarm

public static boolean removeAll(List myList){…}

@Test public void testRemoveAll(){ } List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeAll(list);

/** * @return true if the list is empty */ public static boolean removeAll(List myList){…}

+ Jdoctor

slide-117
SLIDE 117

Missed alarm

public static boolean removeAll(List myList){…}

@Test public void testRemoveAll(){ } List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeAll(list); if(list.isEmpty()) assert result==true;

/** * @return true if the list is empty */ public static boolean removeAll(List myList){…}

+ Jdoctor

slide-118
SLIDE 118

Missed alarm

public static boolean removeAll(List myList){…}

@Test public void testRemoveAll(){ } List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeAll(list); if(list.isEmpty()) assert result==true;

/** * @return true if the list is empty */ public static boolean removeAll(List myList){…}

+ Jdoctor

slide-119
SLIDE 119

public static float max(float[] floatArray){…}

Invalid test

slide-120
SLIDE 120

public static float max(float[] floatArray){…}

@Test public void testRemoveAll(){ float[] floatArray0 = null; float float1 = max(floatArray0); }

Invalid test

slide-121
SLIDE 121

/** * @param floatArray array of floats, not null */ public static float max(float[] floatArray){…}

@Test public void testRemoveAll(){ float[] floatArray0 = null; float float1 = max(floatArray0); }

Invalid test

+ Jdoctor

slide-122
SLIDE 122

/** * @param floatArray array of floats, not null */ public static float max(float[] floatArray){…}

@Test public void testRemoveAll(){ float[] floatArray0 = null; float float1 = max(floatArray0); }

Invalid test INVALID

+ Jdoctor

slide-123
SLIDE 123

Reclassification

False alarm Missed alarm Invalid test New test

slide-124
SLIDE 124

Source

Javadoc
 Extractor Comment
 Translator

Specs

Jdoctor

slide-125
SLIDE 125

Source

Javadoc
 Extractor Comment
 Translator

Specs

Jdoctor

Preconditions

slide-126
SLIDE 126

Source

Javadoc
 Extractor Comment
 Translator

Specs

Jdoctor

Preconditions Normal Postconditions

slide-127
SLIDE 127

Source

Javadoc
 Extractor Comment
 Translator

Specs

Jdoctor

Preconditions Exceptional Postconditions Normal Postconditions

slide-128
SLIDE 128

Source

Javadoc
 Extractor Comment
 Translator

Specs

Jdoctor

Preconditions Exceptional Postconditions Normal Postconditions Pattern Match

slide-129
SLIDE 129

Source

Javadoc
 Extractor Comment
 Translator

Specs

Jdoctor

Preconditions Exceptional Postconditions Normal Postconditions Pattern Match Syntax Match

slide-130
SLIDE 130

Source

Javadoc
 Extractor Comment
 Translator

Specs

Jdoctor

Preconditions Exceptional Postconditions Normal Postconditions Pattern Match Syntax Match Semantic Match

slide-131
SLIDE 131

Source

Javadoc
 Extractor Comment
 Translator

Specs

Jdoctor

Preconditions Exceptional Postconditions Normal Postconditions

https://github.com/albertogoffi/toradocu

Pattern Match Syntax Match Semantic Match