On the Diffusion of Test Smells in Automatically Generated Test - - PowerPoint PPT Presentation

▶
on the diffusion of test smells in automatically
SMART_READER_LITE
LIVE PREVIEW

On the Diffusion of Test Smells in Automatically Generated Test - - PowerPoint PPT Presentation

Feb 08, 2023 110 likes •611 views

On the Diffusion of Test Smells in Automatically Generated Test Code: An Empirical Study Fabio Palomba * , Dario Di Nucci * , Annibale Panichella + , Rocco Oliveto , Andrea De Lucia * SBST 2016 May, 16th 2016 Austin, TX, USA * University of

slide-1
SLIDE 1

On the Diffusion of Test Smells in Automatically Generated Test Code: An Empirical Study

Fabio Palomba*, Dario Di Nucci*, Annibale Panichella+, Rocco Oliveto°, Andrea De Lucia*

* University of Salerno, Italy +Delft University of Technology, The Netherlands ° University of Molise, Italy

SBST 2016 May, 16th 2016 Austin, TX, USA

slide-2
SLIDE 2
slide-3
SLIDE 3

Automatic Generation of Test Code

method1() method2() … methodN() Production Class test_method1() test_method2() … test_methodN() Test Suite

slide-4
SLIDE 4

Automatic Generation of Test Code

[Arcuri and Fraser - SSBSE’14]

Effectiveness of Whole Suite Test Case Generation

[Panichella et al. - ICSE’16]

Impact of Test Case Summaries

  • n Bug Fixing Performance

[Shamshiri et al. - ASE’15]

Effectiveness of Generated Test Cases on Effectiveness

slide-5
SLIDE 5

Automatic Generation of Test Code

[Rojas et al. - ISSTA’15]

Usability of Automatic Generation Tools in Practice

slide-6
SLIDE 6

Automatic Generation of Test Code

What about the characteristics of test code produced by such tools?

slide-7
SLIDE 7

Test Smells in Test Code

[Van Deursen et al. - XP 2001]

“Test Smells represent a set of a poor design solutions to write tests ”

11

test smells related to the way developers write test fixtures and test cases

slide-8
SLIDE 8

Test Smells in Test Code

public void test12 () throws Throwable { JSTerm jSTerm0 = new JSTerm();
 jSTerm0.makeVariable () ; jSTerm0.add((Object) ””); jSTerm0.matches(jSTerm0);
 assertEquals (false, jSTerm0.isGround ()); assertEquals(true, jSTerm0.isVariable()); }

slide-9
SLIDE 9

Test Smells in Test Code

public void test12 () throws Throwable { JSTerm jSTerm0 = new JSTerm();
 jSTerm0.makeVariable () ; jSTerm0.add((Object) ””); jSTerm0.matches(jSTerm0);
 assertEquals (false, jSTerm0.isGround ()); assertEquals(true, jSTerm0.isVariable()); }

The test method checks the production method isGround()

slide-10
SLIDE 10

Test Smells in Test Code

public void test12 () throws Throwable { JSTerm jSTerm0 = new JSTerm();
 jSTerm0.makeVariable () ; jSTerm0.add((Object) ””); jSTerm0.matches(jSTerm0);
 assertEquals (false, jSTerm0.isGround ()); assertEquals(true, jSTerm0.isVariable()); }

But also the production method isVariable()

slide-11
SLIDE 11

Test Smells in Test Code

public void test12 () throws Throwable { JSTerm jSTerm0 = new JSTerm();
 jSTerm0.makeVariable () ; jSTerm0.add((Object) ””); jSTerm0.matches(jSTerm0);
 assertEquals (false, jSTerm0.isGround ()); assertEquals(true, jSTerm0.isVariable()); }

This is an Eager Test, namely a test which checks more than

  • ne method of the class to be tested, making difficult

the comprehension of the actual test target.

slide-12
SLIDE 12

Test Smells in Test Code

A test case is affected by a Resource Optimism when it makes assumptions about the state or the existence

  • f external resources, providing a non-deterministic

result that depend on the state of the resources. An Assertion Roulette comes from having a number of assertions in a test method that have no explanation. If an assertion fails, the identification of the assert that failed can be difficult.

slide-13
SLIDE 13

Who cares about Test Smells? Test Cases can be re-generated!

slide-14
SLIDE 14

Who cares about Test Smells? Test Cases can be re-generated!

True

slide-15
SLIDE 15

Who cares about Test Smells? Test Cases can be re-generated!

True BUT

slide-16
SLIDE 16

Who cares about Test Smells? Test Cases can be re-generated!

Developers modify and remove test code Developers add tests when automatic tools leave uncovered branches Developers combine generated with manually written tests

[Rojas et al. - ISSTA’15]

Usability of Automatic Generation Tools in Practice

True BUT

slide-17
SLIDE 17

On the Diffusion of Test Smells in Automatically Generated Test Code: An Empirical Study Empirical Study Design

slide-18
SLIDE 18

Empirical Study Design

8

test smell types

“Refactoring Test Code”

[Van Deursen et al. - XP 2001]

slide-19
SLIDE 19

Empirical Study Design

110

software projects

8

test smell types

[Fraser and Arcuri - TOSEM 2014]

“A Large Scale Evaluation of Automated Unit Test Generation using Evosuite” “Refactoring Test Code”

[Van Deursen et al. - XP 2001]

slide-20
SLIDE 20

Empirical Study Design

110

software projects

8

test smell types

[Fraser and Arcuri - TOSEM 2014]

“A Large Scale Evaluation of Automated Unit Test Generation using Evosuite” “Refactoring Test Code”

[Van Deursen et al. - XP 2001]

slide-21
SLIDE 21

Empirical Study Design

16,603

JUnit classes

[Fraser and Arcuri - TOSEM 2014]

“A Large Scale Evaluation of Automated Unit Test Generation using Evosuite”

slide-22
SLIDE 22

Data Extraction

test_method1() test_method2() … test_methodN() Test Suite

[Bavota et al. - EMSE 2015]

“Are Test Smells Harmful? An Empirical Study”

Test Smell Detector

slide-23
SLIDE 23

Data Extraction

test_method1() test_method2() … test_methodN() Test Suite

[Bavota et al. - EMSE 2015]

“Are Test Smells Harmful? An Empirical Study”

Test Smell Detector

75%

precision 100% recall

Sample size: 378 JUnit classes

slide-24
SLIDE 24

Research Questions

RQ1: To What Extent Test Smells are Spread in Automatically Generated Test Classes? ?

slide-25
SLIDE 25

RQ1: To What Extent Test Smells are Spread in Automatically Generated Test Classes? RQ2: Which Test Smells Occur More Frequently in Automatically Generated Test Classes??

Research Questions

slide-26
SLIDE 26

RQ1: To What Extent Test Smells are Spread in Automatically Generated Test Classes? RQ2: Which Test Smells Occur More Frequently in Automatically Generated Test Classes? RQ3: Which Test Smells Co-Occur Together??

Research Questions

slide-27
SLIDE 27

RQ1: To What Extent Test Smells are Spread in Automatically Generated Test Classes? RQ2: Which Test Smells Occur More Frequently in Automatically Generated Test Classes? RQ3: Which Test Smells Co-Occur Together? RQ4: Is There a Relationship Between the Presence

  • f Test Smells and the Project Characteristics??

Research Questions

slide-28
SLIDE 28

On the Diffusion of Test Smells in Automatically Generated Test Code: An Empirical Study Analysis of the Results

slide-29
SLIDE 29

Results of the Study

RQ1: To What Extent Test Smells are Spread in Automatically Generated Test Classes? !

13,791

smelly JUnit classes

slide-30
SLIDE 30

Results of the Study

RQ1: To What Extent Test Smells are Spread in Automatically Generated Test Classes? !

83%

  • f the JUnit classes analyzed
slide-31
SLIDE 31

Results of the Study

RQ1: To What Extent Test Smells are Spread in Automatically Generated Test Classes? ! RQ1 Test Smells are highly diffused in the automatically generated test suites

slide-32
SLIDE 32

RQ2: Which Test Smells Occur More Frequently in Automatically Generated Test Classes?

Results of the Study

!

Assertion Roulette

54%

slide-33
SLIDE 33

RQ2: Which Test Smells Occur More Frequently in Automatically Generated Test Classes?

Results of the Study

!

Assertion Roulette

54%

Test Code Duplication

33%

slide-34
SLIDE 34

RQ2: Which Test Smells Occur More Frequently in Automatically Generated Test Classes?

Results of the Study

!

Assertion Roulette

54%

Test Code Duplication Eager Test

33% 29%

slide-35
SLIDE 35

RQ2: Which Test Smells Occur More Frequently in Automatically Generated Test Classes?

Results of the Study

!

public void test8 () throws Throwable { Document document0 = new Document();
 assertNotNull(document0); document0.procText.add((Character) ”s”); String string0 = document0.stringify();
 assertEquals (“s”, document0.stringify()); 
 assertNotNull(string0);
 assertEquals(“s”, string0); } Assertion Roulette

slide-36
SLIDE 36

RQ2: Which Test Smells Occur More Frequently in Automatically Generated Test Classes?

Results of the Study

!

Assertion Roulette

W h a t i s t h e b e h a v i

  • r

u n d e r t e s t ? Are the generated assertions valid?

slide-37
SLIDE 37

RQ2: Which Test Smells Occur More Frequently in Automatically Generated Test Classes?

Results of the Study

!

Assertion Roulette

[Panichella et al. - ICSE’16]

The Impact of Test Case Summaries on Bug Fixing Performance An Empirical Investigation

These problems have a huge impact on developers’ ability to find faults

slide-38
SLIDE 38

RQ2: Which Test Smells Occur More Frequently in Automatically Generated Test Classes?

Results of the Study

!

public void test8 () throws Throwable { GenericProperties generic0 = new GenericProperties();
 boolean boolean0 = generic0.isValidClassname(); … }

Test Code Duplication

public void test9 () throws Throwable { GenericProperties generic0 = new GenericProperties();
 boolean boolean0 = generic0.isValidClassname(); … }

slide-39
SLIDE 39

RQ2: Which Test Smells Occur More Frequently in Automatically Generated Test Classes?

Results of the Study

!

Test Code Duplication This problem can be avoided by generating test fixtures!

slide-40
SLIDE 40

Results of the Study

!

RQ3: Which Test Smells Co-Occur Together?

Assertion Roulette Eager Test Assertion Roulette Sensitive Equality Resource Optimism Mystery Guest

slide-41
SLIDE 41

Results of the Study

!

RQ3: Which Test Smells Co-Occur Together?

Assertion Roulette Eager Test Assertion Roulette Sensitive Equality Resource Optimism Mystery Guest

Automatic tools have as main goal that of maximize coverage, without considering test code quality

slide-42
SLIDE 42

RQ4: Is There a Relationship Between the Presence

  • f Test Smells and the Project Characteristics?

Results of the Study

!

Yes! The higher the LOC to be tested, the higher the probability to produce a smelly test case!

slide-43
SLIDE 43

RQ4: Is There a Relationship Between the Presence

  • f Test Smells and the Project Characteristics?

Results of the Study

!

Yes! The higher the LOC to be tested, the higher the probability to produce a smelly test case! The higher the LOC of the JUnit class, the higher the probability to produce a smelly test case!

slide-44
SLIDE 44

Summarizing

Lesson 1: Current implementations of search-based algorithms for automatic test case generation do not consider code quality, increasing the probability to introduce smells!

slide-45
SLIDE 45

Summarizing

Lesson 1: Current implementations of search-based algorithms for automatic test case generation do not consider code quality, increasing the probability to introduce smells!

NB: Considering test code quality is important not only to avoid the introduction of smells, but also because the coverage can be increased!

[F. Palomba, A. Panichella, A. Zaidman, R. Oliveto, A. De Lucia - ISSTA’16]

Automatic Test Case Generation: What If Test Code Quality Matters?

slide-46
SLIDE 46

Summarizing

Lesson 2: Automatic test case generation tools do not produce text fixtures during their computation, and this implies the introduction of several code clones in the resulting JUnit classes.

Future research should spend effort in the automatic generation of test fixtures!

slide-47
SLIDE 47

From now on…

?

Challenge 1: Evaluating Test Smells in Test Cases automatically generated by other tools

slide-48
SLIDE 48

From now on…

?

Challenge 1: Evaluating Test Smells in Test Cases automatically generated by other tools Challenge 2: Defining new algorithms able to solve the design problems analyzed (e.g., test fixtures).

slide-49
SLIDE 49

On the Diffusion of Test Smells in Automatically Generated Test Code: An Empirical Study

Fabio Palomba*, Dario Di Nucci*, Annibale Panichella+, Rocco Oliveto°, Andrea De Lucia*

* University of Salerno, Italy +Delft University of Technology, The Netherlands ° University of Molise, Italy

SBST 2016 May, 16th 2016 Austin, TX, USA