Translating Code Comments to Procedure Specifications Arianna Blasi - - PowerPoint PPT Presentation

▶
translating code comments to procedure specifications
SMART_READER_LITE
LIVE PREVIEW

Translating Code Comments to Procedure Specifications Arianna Blasi - - PowerPoint PPT Presentation

Jun 18, 2023 399 likes •1.73k views

Translating Code Comments to Procedure Specifications Arianna Blasi Alberto Goffi Konstantin Kuznetsov Alessandra Gorla Michael D. Ernst Mauro Pezz Sergio Delgado USI Universit della Svizzera

slide-1
SLIDE 1

Translating Code Comments to Procedure Specifications

Arianna Blasi♦♥ · Alberto Goffi♦ · Konstantin Kuznetsov♣ Alessandra Gorla♥ · Michael D. Ernst♠ · Mauro Pezzè♦ · Sergio Delgado♥

♦USI Università della Svizzera italiana,

Switzerland

♥IMDEA Software Institute,

Spain

♣Saarland University/CISPA,

Germany

♠University of Washington Seattle,

WA, USA

slide-2
SLIDE 2

public static boolean removeAll(List myList){…}

slide-3
SLIDE 3

public static boolean removeAll(List myList){…}

slide-4
SLIDE 4

public static boolean removeAll(List myList){…}

@Test public void testRemoveAll(){ }

slide-5
SLIDE 5

public static boolean removeAll(List myList){…}

@Test public void testRemoveAll(){ } List<Integer> list = Arrays.asList(1, 2, 3);

slide-6
SLIDE 6

public static boolean removeAll(List myList){…}

@Test public void testRemoveAll(){ } List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeAll(list);

slide-7
SLIDE 7

public static boolean removeAll(List myList){…}

@Test public void testRemoveAll(){ } List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeAll(list);

assert ?

slide-8
SLIDE 8

public static boolean removeAll(List myList){…}

@Test public void testRemoveAll(){ } List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeAll(list);

We have no idea

assert ?

slide-9
SLIDE 9
slide-10
SLIDE 10
slide-11
SLIDE 11

Where is the oracle?

slide-12
SLIDE 12

Where is the oracle? Where is the specification?

slide-13
SLIDE 13
slide-14
SLIDE 14

Why not generating executable specifications automatically?

slide-15
SLIDE 15

Why not generating executable specifications automatically? Let’s rely on Javadoc comments!

slide-16
SLIDE 16

/** * @return true if the list is empty */ public static boolean removeAll(List myList) {…}

Our solution

slide-17
SLIDE 17

/** * @return true if the list is empty */ public static boolean removeAll(List myList) {…}

myList.isEmpty() ? result==true

Our solution

slide-18
SLIDE 18

/** * @return true if the list is empty */ public static boolean removeAll(List myList) {…}

myList.isEmpty() ? result==true

Our solution

slide-19
SLIDE 19

/** * @return true if the list is empty */ public static boolean removeAll(List myList) {…}

@Test public void testRemoveAll(){ } List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeAll(list);

myList.isEmpty() ? result==true

Our solution

slide-20
SLIDE 20

/** * @return true if the list is empty */ public static boolean removeAll(List myList) {…}

@Test public void testRemoveAll(){ } List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeAll(list);

myList.isEmpty() ? result==true

if(list.isEmpty()) assert result==true;

Our solution

slide-21
SLIDE 21

/** * @return true if the list is empty */ public static boolean removeAll(List myList) {…}

@Test public void testRemoveAll(){ } List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeAll(list);

myList.isEmpty() ? result==true

if(list.isEmpty()) assert result==true;

Our solution

slide-22
SLIDE 22

The idea is not new...

slide-23
SLIDE 23

The idea is not new... @tComment

Tan et al. ICST 2012

slide-24
SLIDE 24

The idea is not new... @tComment Toradocu

Tan et al. ICST 2012 Goffi et al. ISSTA 2016

slide-25
SLIDE 25

The idea is not new... @tComment Toradocu ALICS

Tan et al. ICST 2012 Goffi et al. ISSTA 2016 Pandita et al. ICSE 2012

slide-26
SLIDE 26

The idea is not new... @tComment Toradocu ALICS

Preconditions Normal Postconditions Exceptional Postconditions

Tan et al. ICST 2012 Goffi et al. ISSTA 2016 Pandita et al. ICSE 2012

slide-27
SLIDE 27

The idea is not new... @tComment Toradocu ALICS

Preconditions Normal Postconditions Exceptional Postconditions

Tan et al. ICST 2012 Goffi et al. ISSTA 2016 Pandita et al. ICSE 2012

slide-28
SLIDE 28

The idea is not new... @tComment Toradocu ALICS

Preconditions Normal Postconditions Exceptional Postconditions

Tan et al. ICST 2012 Goffi et al. ISSTA 2016 Pandita et al. ICSE 2012

Jdoctor

slide-29
SLIDE 29

Source

Javadoc
 Extractor Comment
 Translator

Specs

Jdoctor

slide-30
SLIDE 30

Source

Javadoc
 Extractor

Specs

Comment
 Translator

slide-31
SLIDE 31

Source Source

Javadoc
 Extractor

Specs

Comment
 Translator

  • 1. Expected result
  • 2. Condition (Text)
slide-32
SLIDE 32

/** * @return true * if the list is empty */ public boolean removeAll(List list){…}

  • 1. true
  • 2. “if the list is empty”

Source

Javadoc
 Extractor

Specs

Comment
 Translator

  • 1. Expected result
  • 2. Condition (Text)

Source

slide-33
SLIDE 33

/** * @return true * if the list is empty */ public boolean removeAll(List list){…}

  • 1. true
  • 2. “if the list is empty”

Source

Javadoc
 Extractor

Specs

Comment
 Translator

  • 1. Expected result
  • 2. Condition (Text)
slide-34
SLIDE 34

Source

Javadoc
 Extractor

Specs

Comment
 Translator

  • 1. Expected result
  • 2. Condition (Text)
slide-35
SLIDE 35
  • 1. Expected result
  • 2. Condition (Code)
  • 1. Expected result
  • 2. Condition (Text)

Source

Javadoc
 Extractor

Specs

Comment
 Translator

slide-36
SLIDE 36
  • 1. Expected result
  • 2. Condition (Code)
  • 1. Expected result
  • 2. Condition (Text)
  • 1. true
  • 2. “if the list is empty”
  • 1. true
  • 2. myList.isEmpty()

Source

Javadoc
 Extractor

Specs

Comment
 Translator

slide-37
SLIDE 37
  • 1. Expected result
  • 2. Condition (Code)
  • 1. Expected result
  • 2. Condition (Text)
  • 1. true
  • 2. “if the list is empty”
  • 1. true
  • 2. myList.isEmpty()

Source

Javadoc
 Extractor

Specs

Comment
 Translator

myList.isEmpty() ? result==true

slide-38
SLIDE 38

Java Specification Natural Language Specification

Comment
 Translator

slide-39
SLIDE 39

Comment Translator

Subject Matcher NL Parser Predicate
 Matcher Java Specification Natural Language Specification

slide-40
SLIDE 40

Subject Matcher Predicate
 Matcher NL Parser “if the list is empty”

slide-41
SLIDE 41

“if the list is empty” Subject Matcher Predicate
 Matcher NL Parser

Subject Predicate “list” “is empty”

slide-42
SLIDE 42

Predicate
 Matcher NL Parser Subject Matcher

Subject: “list”

slide-43
SLIDE 43

Predicate
 Matcher NL Parser Subject Matcher

Subject: “list” Candidates

Formal Parameters Class Name Methods Fields

slide-44
SLIDE 44

Predicate
 Matcher NL Parser Subject Matcher

Subject: “list”

Candidate Distance

myList

2

isEmpty

7

iterator

8

size

4 …

Candidates

Formal Parameters Class Name Methods Fields

slide-45
SLIDE 45

Predicate
 Matcher NL Parser Subject Matcher

Subject: “list”

Candidate Distance

myList

2

isEmpty

7

iterator

8

size

4 …

Candidates

Formal Parameters Class Name Methods Fields myList 2

slide-46
SLIDE 46

Predicate
 Matcher NL Parser Subject Matcher

Candidate Distance

myList

2

isEmpty

7

iterator

8

size

4 …

Candidates

Formal Parameters Class Name Methods Fields

Subject: “list” => myList

myList 2

slide-47
SLIDE 47

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

slide-48
SLIDE 48

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

slide-49
SLIDE 49

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

1: Exact pattern match?

slide-50
SLIDE 50

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

is null == null

1: Exact pattern match?

slide-51
SLIDE 51

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

is null == null is negative < 0

1: Exact pattern match?

slide-52
SLIDE 52

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

is null == null is negative < 0 is positive > 0

1: Exact pattern match?

slide-53
SLIDE 53

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

is null == null is negative < 0 is positive > 0 ...

1: Exact pattern match?

slide-54
SLIDE 54

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

is null == null is negative < 0 is positive > 0 ...

1: Exact pattern match?

ALICS Toradocu @tComment

slide-55
SLIDE 55

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

is null == null is negative < 0 is positive > 0 ...

"is empty" not found...

ALICS Toradocu @tComment

slide-56
SLIDE 56

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

slide-57
SLIDE 57

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

2: Syntax match?

Toradocu

slide-58
SLIDE 58

Candidate Distance

myList

9

isEmpty

1

iterator

6

size

7 …

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

2: Syntax match?

Toradocu

slide-59
SLIDE 59

Candidate Distance

myList

9

isEmpty

1

iterator

6

size

7 …

Subject Matcher NL Parser Predicate
 Matcher

Subject Predicate “list” “is empty”

2: Syntax match?

Toradocu

isEmpty 1

slide-60
SLIDE 60

Pattern and syntax match are not enough

slide-61
SLIDE 61

Pattern and syntax match are not enough

Return true if the list is empty

slide-62
SLIDE 62

Pattern and syntax match are not enough

Candidate Distance

clear

7

isEmpty

1

iterator

6

size

7 …

Return true if the list is empty

slide-63
SLIDE 63

Pattern and syntax match are not enough

Return true if the list is empty contains no element

slide-64
SLIDE 64

Pattern and syntax match are not enough

Return true if the list is empty contains no element has no content

slide-65
SLIDE 65

Pattern and syntax match are not enough

Return true if the list is empty

Candidate Distance

myList isEmpty iterator size

…

contains no element has no content

slide-66
SLIDE 66

Pattern and syntax match are not enough

Return true if the list is empty

Candidate Distance

myList isEmpty iterator size

…

contains no element has no content

slide-67
SLIDE 67

Pattern and syntax match are not enough

Return true if the list is empty

Candidate Distance

myList isEmpty iterator size

…

contains no element has no content

ALICS Toradocu @tComment

slide-68
SLIDE 68

Analyzing complex comments

/** * Checks if there is a credible threatening unit to * this unit within a range of moves. * * @param moves list of moves * @return True if a threat was found. */ public boolean isInDanger(List<Object> moves)

slide-69
SLIDE 69

Analyzing complex comments

/** * Checks if there is a credible threatening unit to * this unit within a range of moves. * * @param moves list of moves * @return True if a threat was found. */ public boolean isInDanger(List<Object> moves)

slide-70
SLIDE 70

Analyzing complex comments

searchForDanger(List<Object>)

/** * Checks if there is a credible threatening unit to * this unit within a range of moves. * * @param moves list of moves * @return True if a threat was found. */ public boolean isInDanger(List<Object> moves)

slide-71
SLIDE 71

threat was found

search for danger

slide-72
SLIDE 72

threat was found

search for danger

slide-73
SLIDE 73
slide-74
SLIDE 74

“search” is not displayed as a result

slide-75
SLIDE 75

“search” is not displayed as a result We need more than semantic equivalence

slide-76
SLIDE 76

Word embedded in vector space: GloVe, Word2Vec

man king woman queen

Word embedding

slide-77
SLIDE 77

Word embedding

danger threat found search

Word embedded in vector space: GloVe, Word2Vec

slide-78
SLIDE 78

Word embedding

danger threat found search

Word mover's distance (Matt J. Kusner et al., 2015)

slide-79
SLIDE 79

Word embedding

danger threat found search

Word mover's distance (Matt J. Kusner et al., 2015)

slide-80
SLIDE 80

In Chicago the president greets the press Obama speaks in Illinois.

Word Mover's distance

slide-81
SLIDE 81

In Chicago the president greets the press Obama speaks in Illinois.

Word Mover's distance

slide-82
SLIDE 82

In Chicago the president greets the press Obama speaks in Illinois.

Word Mover's distance

slide-83
SLIDE 83

In Chicago the president greets the press Obama speaks in Illinois.

Word Mover's distance

slide-84
SLIDE 84

Search For Danger Threat was found

Word Mover's distance

slide-85
SLIDE 85

Search For Danger Threat was found

Word Mover's distance

slide-86
SLIDE 86

Search For Danger Threat was found

Word Mover's distance

slide-87
SLIDE 87
slide-88
SLIDE 88

How accurate is Jdoctor?

slide-89
SLIDE 89

Popular open-source Java systems

Experimental setup

6

slide-90
SLIDE 90

Analyzed Java methods

Experimental setup

563

slide-91
SLIDE 91

Manually-written Java conditions

Experimental setup

829

slide-92
SLIDE 92
slide-93
SLIDE 93

92 %

Precision

slide-94
SLIDE 94

92 % 83 %

Precision Recall

slide-95
SLIDE 95

Exceptional Postconditions

Precision Recall

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

@tComment Toradocu Jdoctor

slide-96
SLIDE 96

Preconditions

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

@tComment Toradocu Jdoctor

Precision Recall

slide-97
SLIDE 97

Normal Postconditions

Precision Recall

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

@tComment Toradocu Jdoctor

slide-98
SLIDE 98

0% 20% 40% 60% 80% 100%

@tComment Toradocu Jdoctor

Precision Recall

Overall accuracy

slide-99
SLIDE 99
slide-100
SLIDE 100

How can you use Jdoctor?

slide-101
SLIDE 101

Automatic Test Case Generator Jdoctor

+

slide-102
SLIDE 102

Jdoctor

+

slide-103
SLIDE 103

Randoop+Jdoctor reclassification

passing passing failing invalid

slide-104
SLIDE 104

Randoop+Jdoctor reclassification

passing passing failing invalid

slide-105
SLIDE 105

Randoop+Jdoctor reclassification

passing passing failing invalid Jdoctor

+

slide-106
SLIDE 106

Randoop+Jdoctor reclassification

passing invalid passing failing invalid Jdoctor

+

slide-107
SLIDE 107

Randoop+Jdoctor reclassification

passing invalid = invalid passing failing invalid Jdoctor

+

slide-108
SLIDE 108

Randoop+Jdoctor reclassification

passing invalid = invalid passing failing failing invalid Jdoctor

+

slide-109
SLIDE 109

Randoop+Jdoctor reclassification

passing invalid = invalid passing failing = missed alarm failing invalid Jdoctor

+

slide-110
SLIDE 110

Randoop+Jdoctor reclassification

passing invalid = invalid passing failing = missed alarm failing passing invalid Jdoctor

+

slide-111
SLIDE 111

Randoop+Jdoctor reclassification

passing invalid = invalid passing failing = missed alarm failing passing = false alarm invalid Jdoctor

+

slide-112
SLIDE 112

Randoop+Jdoctor reclassification

passing invalid = invalid passing failing = missed alarm failing passing = false alarm invalid passing Jdoctor

+

slide-113
SLIDE 113

Randoop+Jdoctor reclassification

passing invalid = invalid passing failing = missed alarm failing passing = false alarm invalid passing = new test Jdoctor

+

slide-114
SLIDE 114

Missed alarm

public static boolean removeAll(List myList){…}

slide-115
SLIDE 115

@Test public void testRemoveAll(){ } List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeAll(list);

Missed alarm

public static boolean removeAll(List myList){…}

slide-116
SLIDE 116

Missed alarm

public static boolean removeAll(List myList){…}

@Test public void testRemoveAll(){ } List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeAll(list);

/** * @return true if the list is empty */ public static boolean removeAll(List myList){…}

+ Jdoctor

slide-117
SLIDE 117

Missed alarm

public static boolean removeAll(List myList){…}

@Test public void testRemoveAll(){ } List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeAll(list); if(list.isEmpty()) assert result==true;

/** * @return true if the list is empty */ public static boolean removeAll(List myList){…}

+ Jdoctor

slide-118
SLIDE 118

Missed alarm

public static boolean removeAll(List myList){…}

@Test public void testRemoveAll(){ } List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeAll(list); if(list.isEmpty()) assert result==true;

/** * @return true if the list is empty */ public static boolean removeAll(List myList){…}

+ Jdoctor

slide-119
SLIDE 119

public static float max(float[] floatArray){…}

Invalid test

slide-120
SLIDE 120

public static float max(float[] floatArray){…}

@Test public void testRemoveAll(){ float[] floatArray0 = null; float float1 = max(floatArray0); }

Invalid test

slide-121
SLIDE 121

/** * @param floatArray array of floats, not null */ public static float max(float[] floatArray){…}

@Test public void testRemoveAll(){ float[] floatArray0 = null; float float1 = max(floatArray0); }

Invalid test

+ Jdoctor

slide-122
SLIDE 122

/** * @param floatArray array of floats, not null */ public static float max(float[] floatArray){…}

@Test public void testRemoveAll(){ float[] floatArray0 = null; float float1 = max(floatArray0); }

Invalid test INVALID

+ Jdoctor

slide-123
SLIDE 123

Reclassification

False alarm Missed alarm Invalid test New test

slide-124
SLIDE 124

Source

Javadoc
 Extractor Comment
 Translator

Specs

Jdoctor

slide-125
SLIDE 125

Source

Javadoc
 Extractor Comment
 Translator

Specs

Jdoctor

Preconditions

slide-126
SLIDE 126

Source

Javadoc
 Extractor Comment
 Translator

Specs

Jdoctor

Preconditions Normal Postconditions

slide-127
SLIDE 127

Source

Javadoc
 Extractor Comment
 Translator

Specs

Jdoctor

Preconditions Exceptional Postconditions Normal Postconditions

slide-128
SLIDE 128

Source

Javadoc
 Extractor Comment
 Translator

Specs

Jdoctor

Preconditions Exceptional Postconditions Normal Postconditions Pattern Match

slide-129
SLIDE 129

Source

Javadoc
 Extractor Comment
 Translator

Specs

Jdoctor

Preconditions Exceptional Postconditions Normal Postconditions Pattern Match Syntax Match

slide-130
SLIDE 130

Source

Javadoc
 Extractor Comment
 Translator

Specs

Jdoctor

Preconditions Exceptional Postconditions Normal Postconditions Pattern Match Syntax Match Semantic Match

slide-131
SLIDE 131

Source

Javadoc
 Extractor Comment
 Translator

Specs

Jdoctor

Preconditions Exceptional Postconditions Normal Postconditions

https://github.com/albertogoffi/toradocu

Pattern Match Syntax Match Semantic Match