[PPT] - Towards Language Independent (Dynamic) Symbolic Execution Manuel PowerPoint Presentation

SLIDE 1

1University of Geneva, Switzerland 2CERN, Geneva, Switzerland

Towards Language Independent (Dynamic) Symbolic Execution

Manuel Gonzalez-Berges 2 Didier Buchs 1 Stefan Klikovits 1,2

Stefan Klikovits 1 stefan.klikovits@unige.ch

SLIDE 2

What are we doing?

1 MLOC code (Control)
no automated unit testing until three ago
frequent changes in execution environment
(mostly) manual verification
big expenses (time) on QA side

Stefan Klikovits 2 stefan.klikovits@unige.ch

SLIDE 3

What are we doing?

1 MLOC code (Control)
no automated unit testing until three ago
frequent changes in execution environment
(mostly) manual verification
big expenses (time) on QA side

code test cases

Stefan Klikovits 2 stefan.klikovits@unige.ch

SLIDE 4

Language Independent Test Case Generation

https://thriftytraveller.files.wordpress.com/2013/11/asterix_obelix3.gif

Stefan Klikovits 3 stefan.klikovits@unige.ch

SLIDE 5

Language Independent Test Case Generation

https://thriftytraveller.files.wordpress.com/2013/11/asterix_obelix3.gif

Stefan Klikovits 3 stefan.klikovits@unige.ch

SLIDE 6

Language Independent Test Case Generation

1. Develop generic tool

Stefan Klikovits 3 stefan.klikovits@unige.ch

SLIDE 7

Language Independent Test Case Generation

1. Develop generic tool

http://asterix.wikia.com/wiki/Asterix_and_Cleopatra

Stefan Klikovits 3 stefan.klikovits@unige.ch

SLIDE 8

Language Independent Test Case Generation

1. Develop generic tool
2. Modify parser and execution

Stefan Klikovits 3 stefan.klikovits@unige.ch

SLIDE 9

Language Independent Test Case Generation

1. Develop generic tool
2. Modify parser and execution

https://www.efiliale.de/efiliale/images/aktionen/asterix/5624_Troubadix.png

Stefan Klikovits 3 stefan.klikovits@unige.ch

SLIDE 10

Language Independent Test Case Generation

1. Develop generic tool
2. Modify parser and execution
3. Translate to existing tool

language

http://www.asterix.com/asterix-de-a-a-z/les-personnages/perso/a43b.gif

Stefan Klikovits 3 stefan.klikovits@unige.ch

SLIDE 11

Semantics, semantics, semantics

small differences – big impacts

http://samcnitt.tumblr.com/ Stefan Klikovits 4 stefan.klikovits@unige.ch

SLIDE 12

How do we generate TCs?

IDE SP engine CTRL test gen. test driver

code SP data test cases results

CTRL TC Gen source code translator test input translator

SP CTRL CTRL test inputs SP tool code test inputs

TI generator

ITEC workflow

Stefan Klikovits 5 stefan.klikovits@unige.ch

SLIDE 13

How do we generate TCs?

IDE SP engine CTRL test gen. test driver

code SP data test cases results

CTRL TC Gen source code translator test input translator

SP CTRL CTRL test inputs SP tool code test inputs

TI generator

ITEC workflow Considering Execution Environment Resilience: A White-Box Approach

Klikovits et. al., Proc. SERENE 2015, Paris

Stefan Klikovits 5 stefan.klikovits@unige.ch

SLIDE 14

Semi-purification

replace dependencies with parameters

1 f ( x ){ 2 i f GLOBAL_VAR : 3 return dbGet(x) 4 e l s e : 5 return −1 6 }

A non-pure function

1 f_sp ( x , a ,b){ 2 i f a : 3 return b 4 e l s e : 5 return −1 6 }

Semi-purified f(x)

1 test_f_sp (){ 2 x = f ( " t e s t " , True , 5 ) // act 3 a s s e r t ( x == 5) // a s s e r t 4 }

Test case

Stefan Klikovits 6 stefan.klikovits@unige.ch

SLIDE 15

CUT translation & TC Gen

Pex (Microsoft Research)
Dynamic Symbolic Execution
translate CUT, generate PUT
manually create Pex factories, data types, built-in functions

https://sites.google.com/site/diedruidenmt/_/rsrc/1367838067499/miraculix/Miraculix.jpg

Stefan Klikovits 7 stefan.klikovits@unige.ch

SLIDE 16

From Pex to test cases

Pex SP parameter vals parameter vals

bservation vals

Test Input Set mock spec call to CUT asserts // Mock( func ) r e t u r n (. . . ) void test_case_1 (){ param1 = . . . param2 = . . . r e s = CUT( param1 , param2 ) a s s e r t ( " check " , r e s == . . . ) } Test case

Test case generation from Pex output

Stefan Klikovits 8 stefan.klikovits@unige.ch

SLIDE 17

How are we doing it?

IDE SP engine CTRL test gen. test driver

code SP data test cases results

CTRL TC Gen source code translator test input translator

SP CTRL CTRL test inputs SP tool code test inputs

TI generator

ITEC workflow

Stefan Klikovits 9 stefan.klikovits@unige.ch

SLIDE 18

How are we doing it?

IDE SP engine CTRL test gen. test driver

code SP data test cases results

CTRL TC Gen source code translator test input translator

SP CTRL CTRL test inputs SP tool code test inputs

TI generator

ITEC workflow

Stefan Klikovits 9 stefan.klikovits@unige.ch

SLIDE 19

How are we doing it?

IDE SP engine CTRL test gen. test driver

code SP data test cases results

CTRL TC Gen source code translator test input translator

SP CTRL CTRL test inputs SP tool code test inputs

TI generator

ITEC workflow Automated Test Case Generation for CTRL using Pex: Lessons Learned

Klikovits et. al., Proc. SERENE 2016, Gothenburg

Stefan Klikovits 9 stefan.klikovits@unige.ch

SLIDE 20

How to test translation?

Stefan Klikovits 10 stefan.klikovits@unige.ch

SLIDE 21

How to test translation?

Divide

http://chapleau.us/Img/caesar_asterix.gif

Stefan Klikovits 10 stefan.klikovits@unige.ch

SLIDE 22

How to test translation?

Divide

http://chapleau.us/Img/caesar_asterix.gif

Anonymise

https://www.youtube.com/watch?v=UF6E-4G4n_M

Stefan Klikovits 10 stefan.klikovits@unige.ch

SLIDE 23

How to test translation?

Divide

http://chapleau.us/Img/caesar_asterix.gif

Analyse Blocks

https://en.gamigo.com/game/asterix

Anonymise

https://www.youtube.com/watch?v=UF6E-4G4n_M

Stefan Klikovits 10 stefan.klikovits@unige.ch

SLIDE 24

How to test translation?

Divide

http://chapleau.us/Img/caesar_asterix.gif

Analyse Blocks

https://en.gamigo.com/game/asterix

Anonymise

https://www.youtube.com/watch?v=UF6E-4G4n_M

Conquer

https://www.pinterest.com/pin/336784878358770673/

Stefan Klikovits 10 stefan.klikovits@unige.ch

SLIDE 25

How to test translation?

1 i n t func ( i n t a , i n t b ) { 2 a++ 3 a++ 4 b = b+2 5 i f ( a > b ){ 6 return a % b 7 } e l s e { 8 return a + b 9 } 10 }

Divide

Analyse Blocks

https://en.gamigo.com/game/asterix

Anonymise

https://www.youtube.com/watch?v=UF6E-4G4n_M

Conquer

https://www.pinterest.com/pin/336784878358770673/

Stefan Klikovits 10 stefan.klikovits@unige.ch

SLIDE 26

How to test translation?

1 i n t func ( i n t a , i n t b ) { 2 a++ 3 a++ 4 b = b+2 5 i f ( a > b ){ 6 return a % b 7 } e l s e { 8 return a + b 9 } 10 }

Divide

Analyse Blocks

https://en.gamigo.com/game/asterix

1 i n t func ( int , i n t ){ 2 i n t++ 3 i n t++ 4 i n t = i n t + i n t 5 i f ( i n t > i n t ) { 6 return i n t % i n t 7 } e l s e { 8 return i n t + i n t 9 } 10 }

Anonymise

Conquer

https://www.pinterest.com/pin/336784878358770673/

Stefan Klikovits 10 stefan.klikovits@unige.ch

SLIDE 27

How to test translation?

1 i n t func ( i n t a , i n t b ) { 2 a++ 3 a++ 4 b = b+2 5 i f ( a > b ){ 6 return a % b 7 } e l s e { 8 return a + b 9 } 10 }

Divide

1 i n t func ( i n t a , i n t b ) { 2 i n t++ 1 3 i n t++ 1 4 i n t = i n t+i n t 1 5 i f ( i n t > i n t ){ 6 return i n t % i n t 7 } e l s e { 8 return i n t + i n t 1 9 } 10 }

Analyse Blocks

1 i n t func ( int , i n t ){ 2 i n t++ 3 i n t++ 4 i n t = i n t + i n t 5 i f ( i n t > i n t ) { 6 return i n t % i n t 7 } e l s e { 8 return i n t + i n t 9 } 10 }

Anonymise

Conquer

https://www.pinterest.com/pin/336784878358770673/

Stefan Klikovits 10 stefan.klikovits@unige.ch

SLIDE 28

How to test translation?

1 i n t func ( i n t a , i n t b ) { 2 a++ 3 a++ 4 b = b+2 5 i f ( a > b ){ 6 return a % b 7 } e l s e { 8 return a + b 9 } 10 }

Divide

1 i n t func ( i n t a , i n t b ) { 2 i n t++ 1 3 i n t++ 1 4 i n t = i n t+i n t 1 5 i f ( i n t > i n t ){ 6 return i n t % i n t 7 } e l s e { 8 return i n t + i n t 1 9 } 10 }

Analyse Blocks

1 i n t func ( int , i n t ){ 2 i n t++ 3 i n t++ 4 i n t = i n t + i n t 5 i f ( i n t > i n t ) { 6 return i n t % i n t 7 } e l s e { 8 return i n t + i n t 9 } 10 }

Anonymise φ =

φ(Li)

|L| Conquer

Stefan Klikovits 10 stefan.klikovits@unige.ch

SLIDE 29

Test case generation: results

CTRL Functions 1521 499 Unsupported Feature 45 SP Error Semi-purification SP Functions 977 186 Translation Error Translation C# Code 791 Test Inputs 5060 ATCG (Pex) Test Cases 5060 TCGen 24 Exec Error 4138 matching oracles 898 mismatching oracles

Stefan Klikovits 11 stefan.klikovits@unige.ch

SLIDE 30

Number of Test Cases

1 - 3 4 - 7 8 - 14 15 - 30 > 30 5 10 15 20 25 30 35 40 45

% Functions

0.8% 42.1% 32.9% 17.1% 5.9% 1.3%

# TCs per function (n = 791)

10 10 20 30 40 50 60 70 80 90 100 110 120

Box plot distribution

Stefan Klikovits 12 stefan.klikovits@unige.ch

SLIDE 31

Coverage

no report 0% 1% - 49% 50% - 75% 75% - 99% 100%

Coverage % Functions

0.1% 1.0% 5.8% 7.2% 9.9% 76.0% 76.0%

n = 791

Stefan Klikovits 13 stefan.klikovits@unige.ch

SLIDE 32

Lessons learned

not everything can be translated (easily)
nor should it ... (?)
C# is no silver bullet
improving the quality of test cases ?
tools have “features”

Stefan Klikovits 14 stefan.klikovits@unige.ch

SLIDE 33

Summary

Stefan Klikovits 15 stefan.klikovits@unige.ch

SLIDE 34

1University of Geneva, Switzerland 2CERN, Geneva, Switzerland

Towards Language Independent (Dynamic) Symbolic Execution

Manuel Gonzalez-Berges 2 Didier Buchs 1 Stefan Klikovits 1,2

Stefan Klikovits 16 stefan.klikovits@unige.ch

SLIDE 35

What next?

expand TC generation
other/different use cases
trade-off complexity vs. usefulness
research unsupported features

Stefan Klikovits 17 stefan.klikovits@unige.ch