Cloud9 Parallel Symbolic Execution for Automated Real-World - - PowerPoint PPT Presentation

▶

cloud9

Cloud9 Parallel Symbolic Execution for Automated Real-World - - PowerPoint PPT Presentation

Dec 29, 2022 640 likes •1.25k views

Cloud9 Parallel Symbolic Execution for Automated Real-World Software Testing Stefan Bucur, Vlad Ureche, Cristian Zamfir, George Candea School of Computer and Communication Sciences Automated Software Testing Automated Industrial Techniques

slide-1

SLIDE 1

Parallel Symbolic Execution for Automated Real-World Software Testing

Stefan Bucur, Vlad Ureche, Cristian Zamfir, George Candea

Cloud9

School of Computer and Communication Sciences

slide-2

SLIDE 2

Automated Techniques

Automated Software Testing

2

λ

Symbolic Execution Model Checking

Industrial SW Testing

Manual Testing Static Analysis Fuzzing Scalability Applicability Usability

slide-3

SLIDE 3

Cloud9 - The Big Picture

Parallel symbolic execution
Linear scalability on commodity clusters
Full symbolic POSIX support
Applicable on real-world systems
Platform for writing test cases
Easy-to-use platform API

3

slide-4

SLIDE 4

Automated Systems Testing

4

[*] C. Cadar, D. Dunbar, D. Engler, “KLEE: Unassisted and automatic generation

f high-coverage tests for complex systems programs”, OSDI 2008
Promising for systems testing:

KLEE [*]

High-coverage test cases
Found new bugs
... But applied only on small

programs

λ

Symbolic Execution

slide-5

SLIDE 5

5

Memcached GNU Coreutils Apache

slide-6

SLIDE 6

void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... }

Symbolic Execution in a Nutshell

[C9 A0 ... ]

6

slide-7

SLIDE 7

void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... }

Symbolic Execution in a Nutshell

[C9 A0 ... ]

6

slide-8

SLIDE 8

pkt->magic != 0xC9 void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... }

Symbolic Execution in a Nutshell

[C9 A0 ... ]

6

slide-9

SLIDE 9

pkt->cmd == GET pkt->magic != 0xC9 void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... }

Symbolic Execution in a Nutshell

[C9 A0 ... ]

6

slide-10

SLIDE 10

pkt->cmd == GET pkt->magic != 0xC9 void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... }

Symbolic Execution in a Nutshell

[C9 A0 ... ]

6

slide-11

SLIDE 11

void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... }

Symbolic Execution in a Nutshell

7

λ

slide-12

SLIDE 12

λ.magic == 0xC9 λ.magic != 0xC9 void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... }

Symbolic Execution in a Nutshell

7

λ

slide-13

SLIDE 13

λ.cmd == GET λ.cmd != GET λ.magic == 0xC9 λ.magic != 0xC9 void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... }

Symbolic Execution in a Nutshell

7

λ

slide-14

SLIDE 14

λ.cmd == GET λ.cmd != GET λ.magic == 0xC9 λ.magic != 0xC9 void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... }

Symbolic Execution in a Nutshell

7

∼2 paths

λ program size

slide-15

SLIDE 15

8

CPU Bottleneck Memory Exhaustion

slide-16

SLIDE 16

W1 W2 W3

Parallel Tree Exploration

8

slide-17

SLIDE 17

W1 W2 W3

Parallel Tree Exploration

8

Key research problem: Scalable parallel exploration

slide-18

SLIDE 18

Linear Solution to Exponential Problem

9

Program Size Time to Test

slide-19

SLIDE 19

Linear Solution to Exponential Problem

9

Program Size Time to Test

Testing target 1 worker

slide-20

SLIDE 20

Linear Solution to Exponential Problem

9

Program Size Time to Test

Testing target

Bring testing time down to practical values

1 worker 2 workers 4 workers 8 workers

slide-21

SLIDE 21

Throw Hardware at the Problem

10

slide-22

SLIDE 22

Scalability Challenges

Tree structure not known a priori

? ?? ?

?

?

?

? ? ?

11

slide-23

SLIDE 23

Scalability Challenges

Static Allocation

12

slide-24

SLIDE 24

Scalability Challenges

12

slide-25

SLIDE 25

Scalability Challenges

Anticipate Allocation

13

slide-26

SLIDE 26

Scalability Challenges

13

slide-27

SLIDE 27

Outline

Scalable Parallel Symbolic Execution
POSIX Environment Model
Evaluation

14

slide-28

SLIDE 28

Cloud9 Architecture

15

Global Symbolic Tree

slide-29

SLIDE 29

Cloud9 Architecture

15

W1’s Local Tree W2’s Local Tree W3’s Local Tree Each worker runs a local sequential symbolic execution engine (KLEE)

slide-30

SLIDE 30

Cloud9 Architecture

16

Candidate nodes Fence nodes

Candidate nodes are selected for

exploration

Fence nodes bound the local tree

slide-31

SLIDE 31

Load Balancing

LB

W1 W2 W3

17

Hybrid distributed system: centralized reports, P2P work transfer

slide-32

SLIDE 32

Load Balancing

LB

W1 W2 W3

17

Hybrid distributed system: centralized reports, P2P work transfer

slide-33

SLIDE 33

Load Balancing

LB

W1 W2 W3

17

Hybrid distributed system: centralized reports, P2P work transfer

slide-34

SLIDE 34

Work Transfer

W1

18

Candidate Fence

slide-35

SLIDE 35

Work Transfer

W1 W2

18

Candidate Fence

slide-36

SLIDE 36

Work Transfer

W1 W2

Virtual

18

Candidate Fence

slide-37

SLIDE 37

Work Transfer

W1 W2

Virtual

18

Candidate Fence

slide-38

SLIDE 38

Work Transfer

W1 W2

Materialized

18

Candidate Fence

slide-39

SLIDE 39

Work Transfer

W1 W2

18

Exploration disjointness + completeness

Candidate Fence

slide-40

SLIDE 40

1 1 1 1 1

Path-based Encoding

19

Nodes are encoded as paths in tree
Compact binary representation
Two paths can share common prefix
Small encoding size
For a tree of 2100 leaves, a path fits in

<128 bits (16 bytes)

slide-41

SLIDE 41

Load Balancing in Practice

20

LB stops after 1 min LB stops after 4 min Continuous load balancing Work done [% of total instructions] Time [minutes]

10 20 30 40 50 60 70 80 90 100 2 4 6 8 10

Load balancing necessary to ensure scalability

slide-42

SLIDE 42

Outline

Scalable Parallel Symbolic Execution
POSIX Environment Model
Evaluation

21

slide-43

SLIDE 43

Calls into the Environment

22

if (fork() == 0) { ... if ((res = recv(sock, buff, size, 0)) > 0) { pthread_mutex_lock(&mutex); memcpy(gBuff, buff, res); pthread_mutex_unlock(&mutex); } ... } else { ... pid_t pid = wait(&stat); ... }

slide-44

SLIDE 44

fork()

Program Under Test Environment

(C Library / OS)

Environment Model

23

Cannot directly execute symbolically

slide-45

SLIDE 45

fork()

Program Under Test Environment

(C Library / OS)

Environment Model

23

Model Code

Symbolic Execution Engine

Equivalent functionality Executable symbolically

slide-46

SLIDE 46

Starting Point

24

Symbolic Execution Engine

Network

Stubs

Files

POSIX

S i n g l e

t

h r e a d e d i s

l

a t e d n

d

e s S i n g l e

t

h r e a d e d u t i l i t i e s

slide-47

SLIDE 47

POSIX Environment Model

25

Symbolic Execution Engine

Network

TCP/UDP/UNIX

Files Pipes Threads

pthread_*

Processes

POSIX

M e s s a g e p a s s i n g S e r v e r s a n d c l i e n t s M u l t i

t

h r e a d e d p r

g

r a m s D i s t r i b u t e d s y s t e m s

Signals

A s y n c h r

n
u

s e v e n t s , I P C S i n g l e

t

h r e a d e d u t i l i t i e s

slide-48

SLIDE 48

Key Changes in Symbolic Execution

Multithreading and Scheduling

Deterministic or symbolic scheduling
Non-preemptive execution model

Address Space Isolation

Copy on Write (CoW) between processes
CoW domains for memory sharing

26

slide-49

SLIDE 49

Symbolic Engine System Calls

Symbolic engine support

needed for threads/processes

1. Thread/process lifecycle
2. Synchronization
3. Shared memory

27

Symbolic Engine System Calls

thread_create thread_terminate process_fork process_terminate get_context thread_preempt thread_sleep thread_notify get_wait_list make_shared

1 2 3

slide-50

SLIDE 50

Outline

Scalable Parallel Symbolic Execution
POSIX Environment Model
Evaluation

28

slide-51

SLIDE 51

Testing Real-World Software

29

Memcached GNU Coreutils Apache

slide-52

SLIDE 52

Time to Reach Target Coverage

30

printf Faster time-to-cover, higher coverage values

60% coverage 70% coverage 80% coverage 90% coverage 10 20 30 40 50 60 1 4 8 24 48 Time to achieve target coverage [minutes] Number of workers

slide-53

SLIDE 53

Increase in Code Coverage

10 20 30 40 50 10 20 30 40 50 60 70 80 90 Additional code covered [ % of program LOC ] Index of tested Coreutil (sorted by additional coverage)

31

Coreutils suite (12 workers, 10 min.) Consistent code coverage increase

slide-54

SLIDE 54

Exhaustive Exploration

32

1 2 3 4 5 6 2 4 6 12 24 48 Time to complete exhaustive test [hours] Number of workers

Scalability of exhaustive path exploration memcached (7.4×104 paths)

slide-55

SLIDE 55

Instruction Throughput

33

0.0e+00 2.0e+09 4.0e+09 6.0e+09 8.0e+09 1.0e+10 1.2e+10 1.4e+10 1.6e+10 1.8e+10 1 4 6 12 24 48 Useful work done [ # of instructions ] Number of workers 4 minutes 6 minutes 8 minutes 10 minutes

memcached Linear scalability with number of workers

slide-56

SLIDE 56

Execute the “whole world” symbolically Symbolic State

Experimental Setup

34

Client Process memcached/ Apache/ lighttpd

TCP Stream Symbolic cmd.

Srv. response

slide-57

SLIDE 57

Symbolic Test Cases

Easy-to-use API for developers to write

symbolic test cases

Basic symbolic memory support
POSIX extensions for environment control
Network conditions, fault injection, symbolic

scheduler

35

slide-58

SLIDE 58

Symbolic Test Cases

36

Testing HTTP header extension

make_symbolic(hdrData); // Append symbolic header to request strcat(req, “X-NewExtension: “); strcat(req, hdrData); // Enable fault injection on socket ioctl(ssock, SIO_FAULT_INJ, RD | WR); // Symbolic stream fragmentation ioctl(ssock, SIO_PKT_FRAGMENT, RD);

slide-59

SLIDE 59

Conclusions

Parallel symbolic execution
Linear scalability on commodity clusters
Full POSIX environment model
Real-world systems testing
Use cases
Increasing coverage
Exhaustive path exploration
Bug patch verification

37