Mutation Testing Meets Approximate Computing Milos Gligoric 1 , - - PowerPoint PPT Presentation

▶
mutation testing meets
SMART_READER_LITE
LIVE PREVIEW

Mutation Testing Meets Approximate Computing Milos Gligoric 1 , - - PowerPoint PPT Presentation

Dec 11, 2023 126 likes •450 views

Mutation Testing Meets Approximate Computing Milos Gligoric 1 , Sarfraz Khurshid 1 , Sasa Misailovic 2 , August Shi 2 ICSE NIER 2017 Buenos Aires, Argentina May 24, 2017 1 2 CCF-1409423, CCF-1421503, CCF-1566363, CCF-1629431, CCF-1319688,

slide-1
SLIDE 1

Mutation Testing Meets Approximate Computing

Milos Gligoric1, Sarfraz Khurshid1, Sasa Misailovic2, August Shi2 ICSE NIER 2017 Buenos Aires, Argentina May 24, 2017

1

CCF-1409423, CCF-1421503, CCF-1566363, CCF-1629431, CCF-1319688, CNS-1239498

1 2

slide-2
SLIDE 2

Mutation Testing

  • Goal: Evaluate quality of test suites
  • How: Apply transformations (mutation operators) on

the code and run tests to see if they can detect the changes

  • Example:

x = x + 1

2

slide-3
SLIDE 3

Mutation Testing

  • Goal: Evaluate quality of test suites
  • How: Apply transformations (mutation operators) on

the code and run tests to see if they can detect the changes

  • Example:

x = x + 2

  • Problems:
  • Evaluation of quality limited by mutation operators
  • Too slow

3

slide-4
SLIDE 4

Approximate Computing

  • Goal: Improve performance of code
  • How: Apply transformations that may lead to

(slightly) inaccurate results

  • Example:

for (i = 0; i < n; i = i + 1)

4

slide-5
SLIDE 5

Approximate Computing

  • Goal: Improve performance of code
  • How: Apply transformations that may lead to

(slightly) inaccurate results

  • Example:

for (i = 0; i < n; i = i + 2)

  • Problems:
  • Not sure where in exact code to apply approximations
  • Unclear how to check quality of tests on already

approximate code

5

slide-6
SLIDE 6

How can Mu Mutatio ion Testin ing and Approxim imate Comp mputin ing improve one another?

6

slide-7
SLIDE 7

Improving One Another

  • Approximate computing to provide new mutation
  • perators for evaluating quality of tests
  • Approximate computing to improve speed of

mutation testing

  • Mutation testing to point out opportunities for

applying approximations on exact code

  • Mutation testing to evaluate quality of tests on

(already) approximate code

7

slide-8
SLIDE 8

Example Code: Commons-Math

8

// MathArrays.java static double[] unique(double[] data) { TreeSet<Double> values = new TreeSet<>(); for (int i = 0; i < data.length; i++) { 0 values.add(data[i]); } int count = values.size(); double[] out = new double[count]; Iterator<Double> iterator = values.descendingIterator(); int i = 0; while (iterator.hasNext()) {

  • ut[i++] = iterator.next();

} return out; } // MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2}; double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArrayEquals(values, MathArrays.unique(x), 0); }

Test Passes

slide-9
SLIDE 9

// MathArrays.java static double[] unique(double[] data) { TreeSet<Double> values = new TreeSet<>(); for (int i = 0; i < data.length; i++) { values.add(data[i]); } int count = values.size(); double[] out = new double[count]; Iterator<Double> iterator = values.descendingIterator(); int i = 1; while (iterator.hasNext()) {

  • ut[i++] = iterator.next();

} return out; }

Mutant: Constant Replacement

9

Mutant Killed Replace 0 with 1 Test Fails

// MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2}; double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArrayEquals(values, MathArrays.unique(x), 0); }

slide-10
SLIDE 10

// MathArrays.java static double[] unique(double[] data) { TreeSet<Double> values = new TreeSet<>(); for (int i = 1; i < data.length; i++) { 0 values.add(data[i]); } int count = values.size(); double[] out = new double[count]; Iterator<Double> iterator = values.descendingIterator(); int i = 0; while (iterator.hasNext()) {

  • ut[i++] = iterator.next();

} return out; }

Mutant: Constant Replacement

10

Mutant Survived Replace 0 with 1

// MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2}; double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArrayEquals(values, MathArrays.unique(x), 0); }

Test Passes

slide-11
SLIDE 11

// MathArrays.java static double[] unique(double[] data) { TreeSet<Double> values = new TreeSet<>(); for (int i = 0; i < data.length; i+=2) values.add(data[i]); } int count = values.size(); double[] out = new double[count]; Iterator<Double> iterator = values.descendingIterator(); int i = 0; while (iterator.hasNext()) {

  • ut[i++] = iterator.next();

} return out; }

Approximate: Loop Perforation

11

Skip every other iteration 68% of runtime is in this loop

// MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2}; double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArrayEquals(values, MathArrays.unique(x), 0); }

Test Fails

slide-12
SLIDE 12

Approximate: Loop Perforation

12

Test Passes

// MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2}; double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArraySubset(values, MathArrays.unique(x), 0); }

Modify assertion Approximation is Acceptable

// MathArrays.java static double[] unique(double[] data) { TreeSet<Double> values = new TreeSet<>(); for (int i = 0; i < data.length; i+=2) values.add(data[i]); } int count = values.size(); double[] out = new double[count]; Iterator<Double> iterator = values.descendingIterator(); int i = 0; while (iterator.hasNext()) {

  • ut[i++] = iterator.next();

} return out; }

Skip every other iteration 68% of runtime is in this loop

slide-13
SLIDE 13

Comparison of Transformation Results

13

Failing Test Passing Test Mutation Testing Approximate Computing

slide-14
SLIDE 14

Comparison of Transformation Results

14

Failing Test Passing Test Mutation Testing Approximate Computing

slide-15
SLIDE 15
  • Approx. Transformation as Operator

15

// MathArrays.java static double[] unique(double[] data) { TreeSet<Double> values = new TreeSet<>(); for (int i = 0; i < data.length; i++) values.add(data[i]); } int count = values.size(); double[] out = new double[count]; Iterator<Double> iterator = values.descendingIterator(); int i = 0; while (iterator.hasNext()) {

  • ut[i++] = iterator.next();

} return out; } // MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2}; double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArrayEquals(values, MathArrays.unique(x), 0); }

slide-16
SLIDE 16
  • Approx. Transformation as Operator

16

// MathArrays.java static double[] unique(double[] data) { TreeSet<Double> values = new TreeSet<>(); for (int i = 0; i < data.length; i+=2) values.add(data[i]); } int count = values.size(); double[] out = new double[count]; Iterator<Double> iterator = values.descendingIterator(); int i = 0; while (iterator.hasNext()) {

  • ut[i++] = iterator.next();

} return out; } // MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2}; double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArrayEquals(values, MathArrays.unique(x), 0); }

Replace i++ with i+=2 Perforates loop Mutant Killed Test Fails

slide-17
SLIDE 17

Questions for Approx. Operators

  • Do killed approximate mutants indicate different

strengths? Do surviving approximate mutants indicate new weaknesses in the test suite?

  • How do mutants generated by approximate

computing differ from traditional mutants?

  • Are approximate mutants faster than traditional

mutants?

17

slide-18
SLIDE 18

Mutants Find Approx. Opportunities

18

// MathArrays.java static double[] unique(double[] data) { TreeSet<Double> values = new TreeSet<>(); for (int i = 0; i < data.length; i++) values.add(data[i]); } int count = values.size(); double[] out = new double[count]; Iterator<Double> iterator = values.descendingIterator(); int i = 0; while (iterator.hasNext()) {

  • ut[i++] = iterator.next();

} return out; } // MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2}; double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArrayEquals(values, MathArrays.unique(x), 0); }

slide-19
SLIDE 19

// MathArrays.java static double[] unique(double[] data) { TreeSet<Double> values = new TreeSet<>(); for (int i = 1; i < data.length; i++) values.add(data[i]); } int count = values.size(); double[] out = new double[count]; Iterator<Double> iterator = values.descendingIterator(); int i = 0; while (iterator.hasNext()) {

  • ut[i++] = iterator.next();

} return out; } // MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2}; double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArrayEquals(values, MathArrays.unique(x), 0); }

Mutants Find Approx. Opportunities

19

Replace 0 with 1 Approximable? Test Passes

slide-20
SLIDE 20

Questions for Approx. Opportunities

  • How can we classify surviving mutants?

Are they good for approximate computing?

  • What approximations are applicable for which

surviving mutants?

  • How can we tailor mutants for the purpose of

finding approximate computing opportunities?

20

slide-21
SLIDE 21

Improving One Another

  • Approximate computing to provide new mutation
  • perators for evaluating quality of tests
  • Approximate computing to improve speed of

mutation testing

  • Mutation testing to point out opportunities for

applying approximations on exact code

  • Mutation testing to evaluate quality of tests on

(already) approximate code

21

More in paper!

slide-22
SLIDE 22

Conclusions

  • Approximate computing can provide new

mutation operators

  • Mutation testing can show opportunities for

approximate computing on exact code

  • There is so much more we can do

(More directions in the paper)

22

August Shi: awshi2@illinois.edu

slide-23
SLIDE 23

BACKUP

23

slide-24
SLIDE 24

Mutating Approximate Code?

24

// MathArrays.java static double[] unique(double[] data) { TreeSet<Double> values = new TreeSet<>(); for (int i = 1; i < data.length; i++) { if (i%2 != 0) continue; values.add(data[i]); } return values.toArray(); }

Test Passes Replace 0 with 1

// MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2} double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArrayEquals(values, MathArrays.unique(x), 0); }

slide-25
SLIDE 25

Loop Perforation (Line 04) Skip every 2nd iteration 100 Skip every 4th iteration 95 Only execute every 4th iteration 100

Mutation testing approximate code?

Approximate Transformation Instance % Mutants Killed Exact Version N/A 95

25

What precisely do these changes in percentages mean?

slide-26
SLIDE 26

Directions for Research

  • If approximate version is proxy of exact version, is

mutation score of approximate version also proxy of mutation score of exact version?

  • Do I get same confidence in quality of tests at cheaper

cost?

  • If so, what are the exact conditions where they are

good proxies?

26

slide-27
SLIDE 27

Approximate Code to Speed up Testing?

27

// MathArrays.java

  • 01. static double[] unique(double[] data) {

02. TreeSet values = 03. new TreeSet<>(); 04. for (int i = 0; 05. i < data.length; i++) { 06. if (i%2 != 0) continue; 07. values.add(data[i]); 08. } 09. int count = values.size() 10. double[] out = new double[count]; 11. Iterator iterator = 12. values.descendingIterator(); 13. int i = 0; 14. while (iterator.hasNext()) { 15. if (i == data.length / 2 + 1) break; 16.

  • ut[i++] = iterator.next();

17. } 19. return out;

  • 20. }

// MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2} double[] values = {11, 3, 0, -1}; assertArrayEquals(values, MathArrays.unique(x), 0); }

Loop (line 04) takes 68% of runtime Perforation can cut time in half

slide-28
SLIDE 28

// MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2} double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertApproxEquals(values, MathArrays.unique(x), 0 0.8); }

Approximate Code to Speed up Testing?

28

// MathArrays.java

  • 01. static double[] unique(double[] data) {

02. TreeSet values = 03. new TreeSet<>(); 04. for (int i = 0; 05. i < data.length; i++) { 06. if (i%2 != 0) continue; 07. values.add(data[i]); 08. } 09. int count = values.size() 10. double[] out = new double[count]; 11. Iterator iterator = 12. values.descendingIterator(); 13. int i = 0; 14. while (iterator.hasNext()) { 15. if (i == data.length / 2 + 1) break; 16.

  • ut[i++] = iterator.next();

17. } 19. return out;

  • 20. }

Introduce new assertions?

slide-29
SLIDE 29

Example: Mutation Testing (SURVIVED)

29

// MathArrays.java static double[] unique(double[] data) { TreeSet values = new TreeSet<>(); for (int i = 0; i < data.length; i++) { if (i%2 != 0) continue; values.add(data[i]); } int count = values.size() double[] out = new double[count]; … return out; }

Constant Replacement

Original Mutant

Test passes on original Test passes on mutant => Mutant SURVIVED

// MathArrays.java static double[] unique(double[] data) { TreeSet values = new TreeSet<>(); for (int i = 1; i < data.length; i++) { if (i%2 != 0) continue; values.add(data[i]); } int count = values.size() double[] out = new double[count]; … return out; } // MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2} double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArrayEquals(values, MathArrays.unique(x), 0); }

slide-30
SLIDE 30

Example: Mutation Testing (KILLED)

30

// MathArrays.java static double[] unique(double[] data) { TreeSet values = new TreeSet<>(); for (int i = 0; i < data.length; i++) { if (i%2 != 0) continue; values.add(data[i]); } int count = values.size() double[] out = new double[count]; … return out; }

Boundary Mutator

Original Mutant

Test passes on original Test passes on mutant => Mutant KILLED

// MathArrays.java static double[] unique(double[] data) { TreeSet values = new TreeSet<>(); for (int i = 0; i <= data.length; i++) { if (i%2 != 0) continue; values.add(data[i]); } int count = values.size() double[] out = new double[count]; … return out; } // MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2} double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArrayEquals(values, MathArrays.unique(x), 0); }

slide-31
SLIDE 31

Example: Loop Perforation

31

// MathArrays.java static double[] unique(double[] data) { TreeSet values = new TreeSet<>(); for (int i = 0; i < data.length; i++) { if (i%2 != 0) continue; values.add(data[i]); } int count = values.size() double[] out = new double[count]; … return out; }

Skip Every Other Iteration

Original Mutant

// MathArrays.java static double[] unique(double[] data) { TreeSet values = new TreeSet<>(); for (int i = 0; i < data.length; i+=2) { if (i%2 != 0) continue; values.add(data[i]); } int count = values.size() double[] out = new double[count]; … return out; }