Automated Test Case Generation for CTRL using Pex: Lessons Learned - - PowerPoint PPT Presentation

▶
automated test case generation for ctrl using pex lessons
SMART_READER_LITE
LIVE PREVIEW

Automated Test Case Generation for CTRL using Pex: Lessons Learned - - PowerPoint PPT Presentation

Mar 21, 2024 221 likes •709 views

1 University of Geneva, Switzerland 2 CERN, Geneva, Switzerland Automated Test Case Generation for CTRL using Pex: Lessons Learned David PY Lawrence 1 Stefan Klikovits 1 , 2 Manuel Gonzalez-Berges 2 Didier Buchs 1 Stefan Klikovits 1

slide-1
SLIDE 1

1University of Geneva, Switzerland 2CERN, Geneva, Switzerland

Automated Test Case Generation for CTRL using Pex: Lessons Learned

David PY Lawrence 1 Manuel Gonzalez-Berges 2 Didier Buchs 1 Stefan Klikovits 1,2

Stefan Klikovits 1 s.k@cern.ch

slide-2
SLIDE 2

What are we doing?

  • 1 MLOC code
  • no automated unit testing until two years ago
  • frequent changes in execution environment
  • (mostly) manual verification
  • big expenses (time) on QA side

Stefan Klikovits 2 s.k@cern.ch

slide-3
SLIDE 3

What are we doing?

  • 1 MLOC code
  • no automated unit testing until two years ago
  • frequent changes in execution environment
  • (mostly) manual verification
  • big expenses (time) on QA side

code test cases

Stefan Klikovits 2 s.k@cern.ch

slide-4
SLIDE 4

How are we doing it?

IDE SP engine CTRL test gen. test driver

code SP data test cases results

CTRL TC Gen source code translator test input translator

SP CTRL CTRL test inputs SP tool code test inputs

TI generator

ITEC workflow

Stefan Klikovits 3 s.k@cern.ch

slide-5
SLIDE 5

How are we doing it?

IDE SP engine CTRL test gen. test driver

code SP data test cases results

CTRL TC Gen source code translator test input translator

SP CTRL CTRL test inputs SP tool code test inputs

TI generator

ITEC workflow

Considering Execution Environment Resilience: A White-Box Approach

Klikovits et. al. SERENE 2015

Stefan Klikovits 3 s.k@cern.ch

slide-6
SLIDE 6

Recap semi-purification

  • replace dependencies with parameters

1 f ( x ){ 2 i f GLOBAL_VAR : 3 return dbGet(x) 4 e l s e : 5 return −1 6 }

A non-pure function

1 f_sp ( x , a ,b){ 2 i f a : 3 return b 4 e l s e : 5 return −1 6 }

Semi-purified f(x)

1 test_f_sp (){ 2 x = f ( " t e s t " , True , 5 ) // act 3 a s s e r t ( x == 5) // a s s e r t 4 }

Test case

Stefan Klikovits 4 s.k@cern.ch

slide-7
SLIDE 7

From Pex to test cases

Pex SP parameter vals parameter vals

  • bservation vals

Test Input Set mock spec call to CUT asserts // Mock( func ) r e t u r n (. . . ) void test_case_1 (){ param1 = . . . param2 = . . . r e s = CUT( param1 , param2 ) a s s e r t ( " check " , r e s == . . . ) } Test case

Test case generation from Pex output

Stefan Klikovits 5 s.k@cern.ch

slide-8
SLIDE 8

Test case generation: results

CTRL Functions 1111 166 Unsupported 159 SP Err Semi-purification SP Functions 786 184 Translation Err Translation C# Code 602 Test Inputs 3972 ATCG (Pex) 294 TCGen Err Test Cases 3678 TCGen 29 Exec Error 2465 matching oracles 1184 mismatching oracles

Stefan Klikovits 6 s.k@cern.ch

slide-9
SLIDE 9

Test case generation: update

CTRL Functions 1191 ✘✘ 1111 249 ✟

✟

166 Unsupported 67 ✟

✟

159 SP Err Semi-purification SP Functions 875 ✟

✟

786 199 ✟

✟

184 Translation Err Translation C# Code 676 ✟

✟

602 Test Inputs 3610 ✘✘ 3972 ATCG (Pex) 493 ✟

✟

294 TCGen Err Test Cases 3117 ✘✘ 3678 TCGen 15 ✚

✚

29 Exec Error 2613 ✘✘ 2464 matching oracles 489 ✘✘ 1184 mismatching oracles

Stefan Klikovits 7 s.k@cern.ch

slide-10
SLIDE 10

Number of test cases

1 - 3 4 - 7 8 - 15 > 15 50 100 150 200 44 295 231 87 19 # Test cases # Functions

Stefan Klikovits 8 s.k@cern.ch

slide-11
SLIDE 11

Coverages

100% 99% - 75% 74% - 50% 49% - 1% 0% 50 100 150 200 230 129 110 118 15 Coverage # Functions

Stefan Klikovits 9 s.k@cern.ch

slide-12
SLIDE 12

Coverages: updated

100% 99% - 75% 74% - 50% 49% - 1% 0% 50 100 150 200 230 129 110 118 15 406 88 58 54 70 Coverage # Functions

Stefan Klikovits 9 s.k@cern.ch

slide-13
SLIDE 13

Coverages: matching oracles

100% 99% - 75% 74% - 50% 49% - 1% 0% 50 100 150 200 406 88 58 54 70 377 83 74 52 90 Coverage # Functions

Stefan Klikovits 9 s.k@cern.ch

slide-14
SLIDE 14

Lessons learned

  • not everything can be translated (easily)

Stefan Klikovits 10 s.k@cern.ch

slide-15
SLIDE 15

Lessons learned

  • not everything can be translated (easily)
  • not all features should be supported

Stefan Klikovits 10 s.k@cern.ch

slide-16
SLIDE 16

Lessons learned

  • not everything can be translated (easily)
  • not all features should be supported
  • C# is no silver bullet

Stefan Klikovits 10 s.k@cern.ch

slide-17
SLIDE 17

Lessons learned

  • not everything can be translated (easily)
  • not all features should be supported
  • C# is no silver bullet
  • improving the quality of test cases ?

Stefan Klikovits 10 s.k@cern.ch

slide-18
SLIDE 18

Lessons learned

  • not everything can be translated (easily)
  • not all features should be supported
  • C# is no silver bullet
  • improving the quality of test cases ?
  • tools have “features”

Stefan Klikovits 10 s.k@cern.ch

slide-19
SLIDE 19

How are we doing it?

IDE SP engine CTRL test gen. test driver

code SP data test cases results

CTRL TC Gen source code translator test input translator

SP CTRL CTRL test inputs SP tool code test inputs

TI generator

ITEC workflow

Stefan Klikovits 11 s.k@cern.ch

slide-20
SLIDE 20

How are we doing it?

IDE SP engine CTRL test gen. test driver

code SP data test cases results

CTRL TC Gen source code translator test input translator

SP CTRL CTRL test inputs SP tool code test inputs

TI generator

ITEC workflow

Stefan Klikovits 11 s.k@cern.ch

slide-21
SLIDE 21

Why test the translation?

  • DySyEx: execute code, cover max. paths
  • small differences – big impacts

Stefan Klikovits 12 s.k@cern.ch

slide-22
SLIDE 22

Why test the translation?

  • DySyEx: execute code, cover max. paths
  • small differences – big impacts

http://samcnitt.tumblr.com/ Stefan Klikovits 12 s.k@cern.ch

slide-23
SLIDE 23

Why not test the translator?

  • No formal semantics?
  • Changing language?
  • Effort to adapt the solution?

Stefan Klikovits 13 s.k@cern.ch

slide-24
SLIDE 24

Why not test the translator?

  • No formal semantics?
  • Changing language?
  • Effort to adapt the solution?

http://asterix.wikia.com/wiki/Asterix_and_Cleopatra

Stefan Klikovits 13 s.k@cern.ch

slide-25
SLIDE 25

How to test?

Stefan Klikovits 14 s.k@cern.ch

slide-26
SLIDE 26

How to test?

Divide

http://chapleau.us/Img/caesar_asterix.gif

Stefan Klikovits 14 s.k@cern.ch

slide-27
SLIDE 27

How to test?

Divide

http://chapleau.us/Img/caesar_asterix.gif

Anonymise

https://www.youtube.com/watch?v=UF6E-4G4n_M

Stefan Klikovits 14 s.k@cern.ch

slide-28
SLIDE 28

How to test?

Divide

http://chapleau.us/Img/caesar_asterix.gif

Analyse Blocks

https://en.gamigo.com/game/asterix

Anonymise

https://www.youtube.com/watch?v=UF6E-4G4n_M

Stefan Klikovits 14 s.k@cern.ch

slide-29
SLIDE 29

How to test?

Divide

http://chapleau.us/Img/caesar_asterix.gif

Analyse Blocks

https://en.gamigo.com/game/asterix

Anonymise

https://www.youtube.com/watch?v=UF6E-4G4n_M

Conquer

https://www.pinterest.com/pin/336784878358770673/

Stefan Klikovits 14 s.k@cern.ch

slide-30
SLIDE 30

How to test?

1 i n t func ( i n t a , i n t b ) { 2 a++ 3 a++ 4 b = b+2 5 i f ( a > b ){ 6 return a % b 7 } e l s e { 8 return a + b 9 } 10 }

Divide

Analyse Blocks

https://en.gamigo.com/game/asterix

Anonymise

https://www.youtube.com/watch?v=UF6E-4G4n_M

Conquer

https://www.pinterest.com/pin/336784878358770673/

Stefan Klikovits 14 s.k@cern.ch

slide-31
SLIDE 31

How to test?

1 i n t func ( i n t a , i n t b ) { 2 a++ 3 a++ 4 b = b+2 5 i f ( a > b ){ 6 return a % b 7 } e l s e { 8 return a + b 9 } 10 }

Divide

Analyse Blocks

https://en.gamigo.com/game/asterix

1 i n t func ( int , i n t ){ 2 i n t++ 3 i n t++ 4 i n t = i n t + i n t 5 i f ( i n t > i n t ) { 6 return i n t % i n t 7 } e l s e { 8 return i n t + i n t 9 } 10 }

Anonymise

Conquer

https://www.pinterest.com/pin/336784878358770673/

Stefan Klikovits 14 s.k@cern.ch

slide-32
SLIDE 32

How to test?

1 i n t func ( i n t a , i n t b ) { 2 a++ 3 a++ 4 b = b+2 5 i f ( a > b ){ 6 return a % b 7 } e l s e { 8 return a + b 9 } 10 }

Divide

1 i n t func ( i n t a , i n t b ) { 2 i n t++ 1 3 i n t++ 1 4 i n t = i n t+i n t 1 5 i f ( i n t > i n t ){ 6 return i n t % i n t 7 } e l s e { 8 return i n t + i n t 1 9 } 10 }

Analyse Blocks

1 i n t func ( int , i n t ){ 2 i n t++ 3 i n t++ 4 i n t = i n t + i n t 5 i f ( i n t > i n t ) { 6 return i n t % i n t 7 } e l s e { 8 return i n t + i n t 9 } 10 }

Anonymise

Conquer

https://www.pinterest.com/pin/336784878358770673/

Stefan Klikovits 14 s.k@cern.ch

slide-33
SLIDE 33

How to test?

1 i n t func ( i n t a , i n t b ) { 2 a++ 3 a++ 4 b = b+2 5 i f ( a > b ){ 6 return a % b 7 } e l s e { 8 return a + b 9 } 10 }

Divide

1 i n t func ( i n t a , i n t b ) { 2 i n t++ 1 3 i n t++ 1 4 i n t = i n t+i n t 1 5 i f ( i n t > i n t ){ 6 return i n t % i n t 7 } e l s e { 8 return i n t + i n t 1 9 } 10 }

Analyse Blocks

1 i n t func ( int , i n t ){ 2 i n t++ 3 i n t++ 4 i n t = i n t + i n t 5 i f ( i n t > i n t ) { 6 return i n t % i n t 7 } e l s e { 8 return i n t + i n t 9 } 10 }

Anonymise φ =

φ(Li)

|L| Conquer

Stefan Klikovits 14 s.k@cern.ch

slide-34
SLIDE 34

Anonymisation & basic blocks

  • Equivalence classes
  • exhaustive testing of features
  • confidence in basic blocks
  • calculate confidence for CUT

Stefan Klikovits 15 s.k@cern.ch

slide-35
SLIDE 35

The quality metric

For translated source code conf (c) =

n

  • i=1

φ(anoni) n , anoni ∈ {Anon}

Stefan Klikovits 16 s.k@cern.ch

slide-36
SLIDE 36

The quality metric

For translated source code conf (c) =

n

  • i=1

φ(anoni) n , anoni ∈ {Anon} For test cases conftc(c, σ) =

  • ∀si∈S|σ

φ(anonymize(si))

Stefan Klikovits 16 s.k@cern.ch

slide-37
SLIDE 37

Example calculation

1 i n t func ( i n t a , i n t b ) { 2 i n t++ 1 3 i n t++ 1 4 i n t = i n t+i n t 1 5 i f ( i n t > i n t ){ 6 return i n t % i n t 7 } e l s e { 8 return i n t + i n t 1 9 } 10 }

φ(func) = 1L2 + 1L4 + 0L5 + 0L6 4 = 0.5

Stefan Klikovits 17 s.k@cern.ch

slide-38
SLIDE 38

Example calculation

1 i n t func ( i n t a , i n t b ) { 2 a++ 1 3 a++ 1 4 b = b+2 1 5 i f ( a > b ){ 6 return a % b 7 } e l s e { 8 return a + b 1 9 } 10 }

φ(func) = 1L2 + 1L4 + 0L5 + 0L6 4 = 0.5 φ(func, 3, 5) = 1L2 ∗ 1L3 ∗ 1L4 ∗ 0L5 ∗ 1L8 = 0

Stefan Klikovits 17 s.k@cern.ch

slide-39
SLIDE 39

Lessons learned

  • how to find exhaustive test suites ?

Stefan Klikovits 18 s.k@cern.ch

slide-40
SLIDE 40

Lessons learned

  • how to find exhaustive test suites ?
  • granularity of basic blocks ?

Stefan Klikovits 18 s.k@cern.ch

slide-41
SLIDE 41

Lessons learned

  • how to find exhaustive test suites ?
  • granularity of basic blocks ?
  • there are thousands (!!) of possibilities

Stefan Klikovits 18 s.k@cern.ch

slide-42
SLIDE 42

Lessons learned

  • how to find exhaustive test suites ?
  • granularity of basic blocks ?
  • there are thousands (!!) of possibilities
  • automation ?

Stefan Klikovits 18 s.k@cern.ch

slide-43
SLIDE 43

What next?

  • expand TC generation
  • exhaustive testing for basic blocks
  • representative study for quality metric
  • trade-off complexity vs. usefulness
  • research unsupported features

Stefan Klikovits 19 s.k@cern.ch

slide-44
SLIDE 44

Conclusion

Stefan Klikovits 20 s.k@cern.ch

slide-45
SLIDE 45

Conclusion

Stefan Klikovits 20 s.k@cern.ch

slide-46
SLIDE 46

Conclusion

Stefan Klikovits 20 s.k@cern.ch

slide-47
SLIDE 47

Conclusion

Stefan Klikovits 20 s.k@cern.ch

slide-48
SLIDE 48

1University of Geneva, Switzerland 2CERN, Geneva, Switzerland

Automated Test Case Generation for CTRL using Pex: Lessons Learned

David PY Lawrence 1 Manuel Gonzalez-Berges 2 Didier Buchs 1 Stefan Klikovits 1,2

Stefan Klikovits 21 s.k@cern.ch