Interoperability-Guided Testing of QUIC Implementations using - - PowerPoint PPT Presentation

Interoperability-Guided Testing

• f QUIC Implementations

using Symbolic Execution

Felix Rath, Daniel Schemmel, Klaus Wehrle

https://comsys.rwth-aachen.de EPIQ Workshop, Heraklion, Greece, 2018-12-04

Motivation

QUANT? Mozquic? mvfst? picoquic? AppleQUIC

?

Are the implementations interoperable?

2

Felix Rath

Motivation

QUANT

?

Mozquic? mvfst? picoquic? AppleQUIC

?

Are the implementations interoperable?

2

Felix Rath

Motivation

QUANT

?

Mozquic

?

mvfst? picoquic? AppleQUIC

?

Are the implementations interoperable?

2

Felix Rath

Motivation

QUANT

?

Mozquic

?

mvfst

?

picoquic? AppleQUIC

?

Are the implementations interoperable?

2

Felix Rath

Motivation

QUANT

?

Mozquic

?

mvfst

?

picoquic

?

AppleQUIC

?

Are the implementations interoperable?

2

Felix Rath

Motivation

QUANT

?

Mozquic

?

mvfst

?

picoquic

?

AppleQUIC

?

Are the implementations interoperable?

2

Felix Rath

Motivation

QUANT? Mozquic? mvfst? picoquic? AppleQUIC

?

Are the implementations interoperable?

2

Felix Rath

Results Summary

• State-of-the-Art software testing approaches work

Can analyze complex communications Uncovered subtle, hard-to-detect bugs

But: Deep interoperability testing requires more insight into implementations Our idea: Comparing belief states of endpoints

Based on a common defjnition Provided by implementations

We would like to propose the development of a comon way to query implementations for their current belief state.

3

Felix Rath

Results Summary

• State-of-the-Art software testing approaches work

▶ Can analyze complex communications Uncovered subtle, hard-to-detect bugs

But: Deep interoperability testing requires more insight into implementations Our idea: Comparing belief states of endpoints

Based on a common defjnition Provided by implementations

We would like to propose the development of a comon way to query implementations for their current belief state.

3

Felix Rath

Results Summary

• State-of-the-Art software testing approaches work

▶ Can analyze complex communications ▶ Uncovered subtle, hard-to-detect bugs

But: Deep interoperability testing requires more insight into implementations Our idea: Comparing belief states of endpoints

Based on a common defjnition Provided by implementations

We would like to propose the development of a comon way to query implementations for their current belief state.

3

Felix Rath

Results Summary

• State-of-the-Art software testing approaches work

▶ Can analyze complex communications ▶ Uncovered subtle, hard-to-detect bugs

• But: Deep interoperability testing requires more insight into

implementations Our idea: Comparing belief states of endpoints

Based on a common defjnition Provided by implementations

We would like to propose the development of a comon way to query implementations for their current belief state.

3

Felix Rath

Results Summary

• State-of-the-Art software testing approaches work

▶ Can analyze complex communications ▶ Uncovered subtle, hard-to-detect bugs

• But: Deep interoperability testing requires more insight into

implementations

• Our idea: Comparing belief states of endpoints

Based on a common defjnition Provided by implementations

We would like to propose the development of a comon way to query implementations for their current belief state.

3

Felix Rath

Results Summary

• State-of-the-Art software testing approaches work

▶ Can analyze complex communications ▶ Uncovered subtle, hard-to-detect bugs

• But: Deep interoperability testing requires more insight into

implementations

• Our idea: Comparing belief states of endpoints

▶ Based on a common defjnition Provided by implementations

We would like to propose the development of a comon way to query implementations for their current belief state.

3

Felix Rath

Results Summary

• State-of-the-Art software testing approaches work

▶ Can analyze complex communications ▶ Uncovered subtle, hard-to-detect bugs

• But: Deep interoperability testing requires more insight into

implementations

• Our idea: Comparing belief states of endpoints

▶ Based on a common defjnition ▶ Provided by implementations

We would like to propose the development of a comon way to query implementations for their current belief state.

3

Felix Rath

Results Summary

• State-of-the-Art software testing approaches work

▶ Can analyze complex communications ▶ Uncovered subtle, hard-to-detect bugs

• But: Deep interoperability testing requires more insight into

implementations

• Our idea: Comparing belief states of endpoints

▶ Based on a common defjnition ▶ Provided by implementations

We would like to propose the development of a comon way to query implementations for their current belief state.

3

Felix Rath

Testing Interoperability

Observations:

• Checking an implementation for standard compliance is problematic

Requires a new “implementation” in the form of a specifjcation Even then: Verifjcation of implementations almost impossible Many implementations already available Test multiple implementations against each other Interoperability-issues occur when “things go wrong” Result: Either non-compliance or bug in standard

4

Felix Rath

Testing Interoperability

Observations:

• Checking an implementation for standard compliance is problematic
• Requires a new “implementation” in the form of a specifjcation

Even then: Verifjcation of implementations almost impossible Many implementations already available Test multiple implementations against each other Interoperability-issues occur when “things go wrong” Result: Either non-compliance or bug in standard

4

Felix Rath

Testing Interoperability

Observations:

• Checking an implementation for standard compliance is problematic
• Requires a new “implementation” in the form of a specifjcation
• Even then: Verifjcation of implementations almost impossible

Many implementations already available Test multiple implementations against each other Interoperability-issues occur when “things go wrong” Result: Either non-compliance or bug in standard

4

Felix Rath

Testing Interoperability

Observations:

• Checking an implementation for standard compliance is problematic
• Requires a new “implementation” in the form of a specifjcation
• Even then: Verifjcation of implementations almost impossible
• Many implementations already available

Test multiple implementations against each other Interoperability-issues occur when “things go wrong” Result: Either non-compliance or bug in standard

4

Felix Rath

Testing Interoperability

Observations:

• Checking an implementation for standard compliance is problematic
• Requires a new “implementation” in the form of a specifjcation
• Even then: Verifjcation of implementations almost impossible
• Many implementations already available
• → Test multiple implementations against each other

Interoperability-issues occur when “things go wrong” Result: Either non-compliance or bug in standard

4

Felix Rath

Testing Interoperability

Observations:

• Checking an implementation for standard compliance is problematic
• Requires a new “implementation” in the form of a specifjcation
• Even then: Verifjcation of implementations almost impossible
• Many implementations already available
• → Test multiple implementations against each other
• Interoperability-issues occur when “things go wrong”

Result: Either non-compliance or bug in standard

4

Felix Rath

Testing Interoperability

Observations:

• Checking an implementation for standard compliance is problematic
• Requires a new “implementation” in the form of a specifjcation
• Even then: Verifjcation of implementations almost impossible
• Many implementations already available
• → Test multiple implementations against each other
• Interoperability-issues occur when “things go wrong”
• Result: Either non-compliance or bug in standard

4

Felix Rath

“Things Going Wrong”

Example 1:

4 open streams + 1 closed stream 5 open streams

Example 2:

malformed (?) packet

5

Felix Rath

“Things Going Wrong”

Example 1:

4 open streams + 1 closed stream 5 open streams

Example 2:

malformed (?) packet

5

Felix Rath

“Things Going Wrong”

Example 1:

4 open streams + 1 closed stream 5 open streams

Example 2:

malformed (?) packet

5

Felix Rath

“Things Going Wrong”

Example 1:

4 open streams + 1 closed stream 5 open streams

Example 2:

malformed (?) packet

5

Felix Rath

“Things Going Wrong”

Example 1:

4 open streams + 1 closed stream 5 open streams

Example 2:

malformed (?) packet

5

Felix Rath

“Things Going Wrong”

Example 1:

4 open streams + 1 closed stream 5 open streams

Example 2:

malformed (?) packet

5

Felix Rath

“Things Going Wrong”

Example 1:

4 open streams + 1 closed stream 5 open streams

Example 2:

malformed (?) packet

5

Felix Rath

“Things Going Wrong”

Example 1:

4 open streams + 1 closed stream 5 open streams

Example 2:

malformed (?) packet

5

Felix Rath

Belief State

Observations:

• Endpoints track state of the connection

Adjusted due to received and sent packets, time, QUIC is standardized, so this state should be similar State can be compared across implementations Belief State

In practice:

Unclear what comprises the belief state of a QUIC connection Extraction requires deep understanding of each implementation

6

Felix Rath

Belief State

Observations:

• Endpoints track state of the connection
• Adjusted due to received and sent packets, time, . . .

QUIC is standardized, so this state should be similar State can be compared across implementations Belief State

In practice:

Unclear what comprises the belief state of a QUIC connection Extraction requires deep understanding of each implementation

6

Felix Rath

Belief State

Observations:

• Endpoints track state of the connection
• Adjusted due to received and sent packets, time, . . .
• QUIC is standardized, so this state should be similar

State can be compared across implementations Belief State

In practice:

Unclear what comprises the belief state of a QUIC connection Extraction requires deep understanding of each implementation

6

Felix Rath

Belief State

Observations:

• Endpoints track state of the connection
• Adjusted due to received and sent packets, time, . . .
• QUIC is standardized, so this state should be similar
• → State can be compared across implementations

Belief State

In practice:

Unclear what comprises the belief state of a QUIC connection Extraction requires deep understanding of each implementation

6

Felix Rath

Belief State

Observations:

• Endpoints track state of the connection
• Adjusted due to received and sent packets, time, . . .
• QUIC is standardized, so this state should be similar
• → State can be compared across implementations
• → Belief State

In practice:

Unclear what comprises the belief state of a QUIC connection Extraction requires deep understanding of each implementation

6

Felix Rath

Belief State

Observations:

• Endpoints track state of the connection
• Adjusted due to received and sent packets, time, . . .
• QUIC is standardized, so this state should be similar
• → State can be compared across implementations
• → Belief State

In practice:

• Unclear what comprises the belief state of a QUIC connection

Extraction requires deep understanding of each implementation

6

Felix Rath

Belief State

Observations:

• Endpoints track state of the connection
• Adjusted due to received and sent packets, time, . . .
• QUIC is standardized, so this state should be similar
• → State can be compared across implementations
• → Belief State

In practice:

• Unclear what comprises the belief state of a QUIC connection
• Extraction requires deep understanding of each implementation

6

Felix Rath

Testing Robustness

Observations:

• Protocol implementations are often exposed to arbitrary input

Should be stable and safe as few bugs as possible Therefore: Also test implementations for robustness Main attack vector: Packet handling code Test receiver functions as thoroughly as possible Detect cases that lead to errors

7

Felix Rath

Testing Robustness

Observations:

• Protocol implementations are often exposed to arbitrary input
• Should be stable and safe → as few bugs as possible

Therefore: Also test implementations for robustness Main attack vector: Packet handling code Test receiver functions as thoroughly as possible Detect cases that lead to errors

7

Felix Rath

Testing Robustness

Observations:

• Protocol implementations are often exposed to arbitrary input
• Should be stable and safe → as few bugs as possible
• Therefore: Also test implementations for robustness

Main attack vector: Packet handling code Test receiver functions as thoroughly as possible Detect cases that lead to errors

7

Felix Rath

Testing Robustness

Observations:

• Protocol implementations are often exposed to arbitrary input
• Should be stable and safe → as few bugs as possible
• Therefore: Also test implementations for robustness
• Main attack vector: Packet handling code

Test receiver functions as thoroughly as possible Detect cases that lead to errors

7

Felix Rath

Testing Robustness

Observations:

• Protocol implementations are often exposed to arbitrary input
• Should be stable and safe → as few bugs as possible
• Therefore: Also test implementations for robustness
• Main attack vector: Packet handling code
• Test receiver functions as thoroughly as possible

Detect cases that lead to errors

7

Felix Rath

Testing Robustness

Observations:

• Protocol implementations are often exposed to arbitrary input
• Should be stable and safe → as few bugs as possible
• Therefore: Also test implementations for robustness
• Main attack vector: Packet handling code
• Test receiver functions as thoroughly as possible
• Detect cases that lead to errors

7

Felix Rath

Symbolic Execution

1 int parse_packet 2 (char* data, size_t len){ 3 if(len >= 1) 4 if(data[1] == 'x') 5 return 1; 6 return 0; 7 }

8

Felix Rath

Symbolic Execution

1 int parse_packet 2 (char* data, size_t len){ 3 if(len >= 1) 4 if(data[1] == 'x') 5 return 1; 6 return 0; 7 } data = symbolic() len = symbolic()

8

Felix Rath

Symbolic Execution

1 int parse_packet 2 (char* data, size_t len){ 3 if(len >= 1) 4 if(data[1] == 'x') 5 return 1; 6 return 0; 7 } data = symbolic() len = symbolic() if(len >= 1)

8

Felix Rath

Symbolic Execution

1 int parse_packet 2 (char* data, size_t len){ 3 if(len >= 1) 4 if(data[1] == 'x') 5 return 1; 6 return 0; 7 } data = symbolic() len = symbolic() if(len >= 1) data[1] len ≥ 1 return 0; ¬(len ≥ 1)

8

Felix Rath

Symbolic Execution

1 int parse_packet 2 (char* data, size_t len){ 3 if(len >= 1) 4 if(data[1] == 'x') 5 return 1; 6 return 0; 7 } data = symbolic() len = symbolic() if(len >= 1) data[1] len ≥ 1 return 0; ¬(len ≥ 1) Out-of-bounds if(data[1] == 'x')

8

Felix Rath

Symbolic Execution

1 int parse_packet 2 (char* data, size_t len){ 3 if(len >= 1) 4 if(data[1] == 'x') 5 return 1; 6 return 0; 7 } data = symbolic() len = symbolic() if(len >= 1) data[1] len ≥ 1 return 0; ¬(len ≥ 1) Out-of-bounds if(data[1] == 'x') return 1; s = 1 return 0; ¬(s = 1)

8

Felix Rath

Symbolic Execution

1 int parse_packet 2 (char* data, size_t len){ 3 if(len >= 1) 4 if(data[1] == 'x') 5 return 1; 6 return 0; 7 } data = symbolic() len = symbolic() if(len >= 1) data[1] len ≥ 1 return 0; ¬(len ≥ 1) Out-of-bounds if(data[1] == 'x') return 1; s = 1 return 0; ¬(s = 1)

Exhaustive exploration

8

Felix Rath

Symbolic Execution

1 int parse_packet 2 (char* data, size_t len){ 3 if(len >= 1) 4 if(data[1] == 'x') 5 return 1; 6 return 0; 7 } data = symbolic() len = symbolic() if(len >= 1) data[1] len ≥ 1 return 0; ¬(len ≥ 1) Out-of-bounds if(data[1] == 'x') return 1; s = 1 return 0; ¬(s = 1)

Exhaustive exploration Find bugs/assertions

8

Felix Rath

SymEx of QUIC Implementations

SymEx requires. . .

• single process and single entry point (e.g., main(), parse_packet())

symbolically running executed code

QUIC implementations

are libraries (no explicit entry point) use a lot of external/kernel functionality (TLS, networking, ) Challenge: How to bridge these gaps?

9

Felix Rath

SymEx of QUIC Implementations

SymEx requires. . .

• single process and single entry point (e.g., main(), parse_packet())
• symbolically running executed code

QUIC implementations

are libraries (no explicit entry point) use a lot of external/kernel functionality (TLS, networking, ) Challenge: How to bridge these gaps?

9

Felix Rath

SymEx of QUIC Implementations

SymEx requires. . .

• single process and single entry point (e.g., main(), parse_packet())
• symbolically running executed code

QUIC implementations . . .

• are libraries (no explicit entry point)

use a lot of external/kernel functionality (TLS, networking, ) Challenge: How to bridge these gaps?

9

Felix Rath

SymEx of QUIC Implementations

SymEx requires. . .

• single process and single entry point (e.g., main(), parse_packet())
• symbolically running executed code

QUIC implementations . . .

• are libraries (no explicit entry point)
• use a lot of external/kernel functionality (TLS, networking, . . .)

Challenge: How to bridge these gaps?

9

Felix Rath

SymEx of QUIC Implementations

SymEx requires. . .

• single process and single entry point (e.g., main(), parse_packet())
• symbolically running executed code

QUIC implementations . . .

• are libraries (no explicit entry point)
• use a lot of external/kernel functionality (TLS, networking, . . .)

Challenge: How to bridge these gaps?

9

Felix Rath

Symbolic Test Scenarios

Solution 1

Defjne concrete test scenarios that use symbolic input.

• pen_conn

+

• pen_stream

+

send_request

+

send_response

+

close_conn

+

10

Felix Rath

Symbolic Test Scenarios

Solution 1

Defjne concrete test scenarios that use symbolic input.

• pen_conn

+

• pen_stream

+

send_request

+

send_response

+

close_conn

+

10

Felix Rath

Symbolic Test Scenarios

Solution 1

Defjne concrete test scenarios that use symbolic input.

• pen_conn

+

• pen_stream

+

send_request

+

send_response

+

close_conn

+

10

Felix Rath

Symbolic Test Scenarios

Solution 1

Defjne concrete test scenarios that use symbolic input.

• pen_conn

+

• pen_stream

+

send_request

+

send_response

+

close_conn

+

10

Felix Rath

Symbolic Test Scenarios

Solution 1

Defjne concrete test scenarios that use symbolic input.

• pen_conn

+

• pen_stream

+

send_request

+

send_response

+

close_conn

+

10

Felix Rath

Symbolic Test Scenarios

Solution 1

Defjne concrete test scenarios that use symbolic input.

• pen_conn

+

• pen_stream

+

send_request

+

send_response

+

close_conn

+

10

Felix Rath

Symbolic Test Scenarios

Solution 1

Defjne concrete test scenarios that use symbolic input.

• pen_conn

+

• pen_stream

+

send_request

+

send_response

+

close_conn

+

10

Felix Rath

Symbolic Test Scenarios

Solution 1

Defjne concrete test scenarios that use symbolic input.

• pen_conn + λ
• pen_stream + λ

send_request + λ send_response + λ close_conn + λ 10

Felix Rath

Mock Libraries

Solution 2

Replace external functionality with mock implementations.

For this work:

OS (UNIX Sockets, ...): enable symbolic packet data Crypto (OpenSSL, ...): make encryption transparent Networking (Libev, ...): enable applications

11

Felix Rath

Mock Libraries

Solution 2

Replace external functionality with mock implementations.

For this work:

• OS (UNIX Sockets, ...): enable symbolic packet data

Crypto (OpenSSL, ...): make encryption transparent Networking (Libev, ...): enable applications

11

Felix Rath

Mock Libraries

Solution 2

Replace external functionality with mock implementations.

For this work:

• OS (UNIX Sockets, ...): enable symbolic packet data
• Crypto (OpenSSL, ...): make encryption transparent

Networking (Libev, ...): enable applications

11

Felix Rath

Mock Libraries

Solution 2

Replace external functionality with mock implementations.

For this work:

• OS (UNIX Sockets, ...): enable symbolic packet data
• Crypto (OpenSSL, ...): make encryption transparent
• Networking (Libev, ...): enable applications

11

Felix Rath

Mock Libraries

Solution 2

Replace external functionality with mock implementations.

For this work:

• OS (UNIX Sockets, ...): enable symbolic packet data
• Crypto (OpenSSL, ...): make encryption transparent
• Networking (Libev, ...): enable applications

Server Client Libev Mock

OpenSSL Mock OpenSSL Mock Socket Mock

• sym. drop
• sym. mod

11

Felix Rath

Case Study

Setup

Picoquic client QUANT server Per-library frontends Three test scenarios Interoperability: Failed connections + Timeouts Robustness: Symbolic packet modifjcations + drops

12

Felix Rath

Case Study

Setup

• Picoquic client

QUANT server Per-library frontends Three test scenarios Interoperability: Failed connections + Timeouts Robustness: Symbolic packet modifjcations + drops

12

Felix Rath

Case Study

Setup

• Picoquic client
• QUANT server

Per-library frontends Three test scenarios Interoperability: Failed connections + Timeouts Robustness: Symbolic packet modifjcations + drops

12

Felix Rath

Case Study

Setup

• Picoquic client
• QUANT server
• Per-library frontends

Three test scenarios Interoperability: Failed connections + Timeouts Robustness: Symbolic packet modifjcations + drops

12

Felix Rath

Case Study

Setup

• Picoquic client
• QUANT server
• Per-library frontends
• Three test scenarios

Interoperability: Failed connections + Timeouts Robustness: Symbolic packet modifjcations + drops

12

Felix Rath

Case Study

Setup

• Picoquic client
• QUANT server
• Per-library frontends
• Three test scenarios
• Interoperability: Failed connections + Timeouts

Robustness: Symbolic packet modifjcations + drops

12

Felix Rath

Case Study

Setup

• Picoquic client
• QUANT server
• Per-library frontends
• Three test scenarios
• Interoperability: Failed connections + Timeouts
• Robustness: Symbolic packet modifjcations + drops

12

Felix Rath

Evaluation

Setup

• Draft 14
• Six confjgurations
• 8h per confjguration
• 32GB memory limit

Confjguration Instrs/s Time[h] ICov[%] BCov[%] TSolver[%] MaxMem[GB] Unique errors sym-stream 1725742 0:01 38.96 24.81 0.06 0.16 2 sym-version 232139 0:25 38.87 24.83 83.83 0.15 1 sym-drop 432753 8:00 38.85 25.31 0.02 11.97 1 sym-mod-1 380751 8:00 41.10 27.04 0.79 32.44 sym-mod-5 241116 7:00 40.11 26.11 8.35 33.02 sym-mod-10 4118 8:01 32.11 18.79 78.78 5.34 1

13

Felix Rath

Sym-Stream

Confjguration Instrs/s Time[h] ICov[%] BCov[%] TSolver[%] MaxMem[GB] Unique errors sym-stream 1725742 0:01 38.96 24.81 0.06 0.16 2 sym-version 232139 0:25 38.87 24.83 83.83 0.15 1 sym-drop 432753 8:00 38.85 25.31 0.02 11.97 1 sym-mod-1 380751 8:00 41.10 27.04 0.79 32.44 sym-mod-5 241116 7:00 40.11 26.11 8.35 33.02 sym-mod-10 4118 8:01 32.11 18.79 78.78 5.34 1

Symbolic Input

No stream Stream without response Stream with 1 byte response

Bugs detected

Interoperability bug (known beforehand) Use-after-free

14

Felix Rath

Sym-Stream

Confjguration Instrs/s Time[h] ICov[%] BCov[%] TSolver[%] MaxMem[GB] Unique errors sym-stream 1725742 0:01 38.96 24.81 0.06 0.16 2 sym-version 232139 0:25 38.87 24.83 83.83 0.15 1 sym-drop 432753 8:00 38.85 25.31 0.02 11.97 1 sym-mod-1 380751 8:00 41.10 27.04 0.79 32.44 sym-mod-5 241116 7:00 40.11 26.11 8.35 33.02 sym-mod-10 4118 8:01 32.11 18.79 78.78 5.34 1

Symbolic Input

• No stream
• Stream without response
• Stream with 1 byte response

Bugs detected

Interoperability bug (known beforehand) Use-after-free

14

Felix Rath

Sym-Stream

Confjguration Instrs/s Time[h] ICov[%] BCov[%] TSolver[%] MaxMem[GB] Unique errors sym-stream 1725742 0:01 38.96 24.81 0.06 0.16 2 sym-version 232139 0:25 38.87 24.83 83.83 0.15 1 sym-drop 432753 8:00 38.85 25.31 0.02 11.97 1 sym-mod-1 380751 8:00 41.10 27.04 0.79 32.44 sym-mod-5 241116 7:00 40.11 26.11 8.35 33.02 sym-mod-10 4118 8:01 32.11 18.79 78.78 5.34 1

Symbolic Input

• No stream
• Stream without response
• Stream with 1 byte response

Bugs detected

• Interoperability bug (known beforehand)

Use-after-free

14

Felix Rath

Sym-Stream

Confjguration Instrs/s Time[h] ICov[%] BCov[%] TSolver[%] MaxMem[GB] Unique errors sym-stream 1725742 0:01 38.96 24.81 0.06 0.16 2 sym-version 232139 0:25 38.87 24.83 83.83 0.15 1 sym-drop 432753 8:00 38.85 25.31 0.02 11.97 1 sym-mod-1 380751 8:00 41.10 27.04 0.79 32.44 sym-mod-5 241116 7:00 40.11 26.11 8.35 33.02 sym-mod-10 4118 8:01 32.11 18.79 78.78 5.34 1

Symbolic Input

• No stream
• Stream without response
• Stream with 1 byte response

Bugs detected

• Interoperability bug (known beforehand)
• Use-after-free

14

Felix Rath

Sym-Version

Confjguration Instrs/s Time[h] ICov[%] BCov[%] TSolver[%] MaxMem[GB] Unique errors sym-stream 1725742 0:01 38.96 24.81 0.06 0.16 2 sym-version 232139 0:25 38.87 24.83 83.83 0.15 1 sym-drop 432753 8:00 38.85 25.31 0.02 11.97 1 sym-mod-1 380751 8:00 41.10 27.04 0.79 32.44 sym-mod-5 241116 7:00 40.11 26.11 8.35 33.02 sym-mod-10 4118 8:01 32.11 18.79 78.78 5.34 1

Symbolic Input

Version announced by the picoquic client

Bugs detected

No connection when version is 0xbabababa

15

Felix Rath

Sym-Version

Confjguration Instrs/s Time[h] ICov[%] BCov[%] TSolver[%] MaxMem[GB] Unique errors sym-stream 1725742 0:01 38.96 24.81 0.06 0.16 2 sym-version 232139 0:25 38.87 24.83 83.83 0.15 1 sym-drop 432753 8:00 38.85 25.31 0.02 11.97 1 sym-mod-1 380751 8:00 41.10 27.04 0.79 32.44 sym-mod-5 241116 7:00 40.11 26.11 8.35 33.02 sym-mod-10 4118 8:01 32.11 18.79 78.78 5.34 1

Symbolic Input

• Version announced by the picoquic client

Bugs detected

No connection when version is 0xbabababa

15

Felix Rath

Sym-Version

Confjguration Instrs/s Time[h] ICov[%] BCov[%] TSolver[%] MaxMem[GB] Unique errors sym-stream 1725742 0:01 38.96 24.81 0.06 0.16 2 sym-version 232139 0:25 38.87 24.83 83.83 0.15 1 sym-drop 432753 8:00 38.85 25.31 0.02 11.97 1 sym-mod-1 380751 8:00 41.10 27.04 0.79 32.44 sym-mod-5 241116 7:00 40.11 26.11 8.35 33.02 sym-mod-10 4118 8:01 32.11 18.79 78.78 5.34 1

Symbolic Input

• Version announced by the picoquic client

Bugs detected

• No connection when version is 0xbabababa

15

Felix Rath

Sym-Drop

Confjguration Instrs/s Time[h] ICov[%] BCov[%] TSolver[%] MaxMem[GB] Unique errors sym-stream 1725742 0:01 38.96 24.81 0.06 0.16 2 sym-version 232139 0:25 38.87 24.83 83.83 0.15 1 sym-drop 432753 8:00 38.85 25.31 0.02 11.97 1 sym-mod-1 380751 8:00 41.10 27.04 0.79 32.44 sym-mod-5 241116 7:00 40.11 26.11 8.35 33.02 sym-mod-10 4118 8:01 32.11 18.79 78.78 5.34 1

Symbolic Input

Every packet dropped symbolically

Bugs detected

Nullpointer-deref when certain packets are dropped

Testcase drops 4th, 5th, 7th, error on 9th Points toward a hard-to-detect issue

16

Felix Rath

Sym-Drop

Confjguration Instrs/s Time[h] ICov[%] BCov[%] TSolver[%] MaxMem[GB] Unique errors sym-stream 1725742 0:01 38.96 24.81 0.06 0.16 2 sym-version 232139 0:25 38.87 24.83 83.83 0.15 1 sym-drop 432753 8:00 38.85 25.31 0.02 11.97 1 sym-mod-1 380751 8:00 41.10 27.04 0.79 32.44 sym-mod-5 241116 7:00 40.11 26.11 8.35 33.02 sym-mod-10 4118 8:01 32.11 18.79 78.78 5.34 1

Symbolic Input

• Every packet dropped symbolically

Bugs detected

Nullpointer-deref when certain packets are dropped

Testcase drops 4th, 5th, 7th, error on 9th Points toward a hard-to-detect issue

16

Felix Rath

Sym-Drop

Confjguration Instrs/s Time[h] ICov[%] BCov[%] TSolver[%] MaxMem[GB] Unique errors sym-stream 1725742 0:01 38.96 24.81 0.06 0.16 2 sym-version 232139 0:25 38.87 24.83 83.83 0.15 1 sym-drop 432753 8:00 38.85 25.31 0.02 11.97 1 sym-mod-1 380751 8:00 41.10 27.04 0.79 32.44 sym-mod-5 241116 7:00 40.11 26.11 8.35 33.02 sym-mod-10 4118 8:01 32.11 18.79 78.78 5.34 1

Symbolic Input

• Every packet dropped symbolically

Bugs detected

• Nullpointer-deref when certain packets are dropped

Testcase drops 4th, 5th, 7th, error on 9th Points toward a hard-to-detect issue

16

Felix Rath

Sym-Drop

Confjguration Instrs/s Time[h] ICov[%] BCov[%] TSolver[%] MaxMem[GB] Unique errors sym-stream 1725742 0:01 38.96 24.81 0.06 0.16 2 sym-version 232139 0:25 38.87 24.83 83.83 0.15 1 sym-drop 432753 8:00 38.85 25.31 0.02 11.97 1 sym-mod-1 380751 8:00 41.10 27.04 0.79 32.44 sym-mod-5 241116 7:00 40.11 26.11 8.35 33.02 sym-mod-10 4118 8:01 32.11 18.79 78.78 5.34 1

Symbolic Input

• Every packet dropped symbolically

Bugs detected

• Nullpointer-deref when certain packets are dropped

▶ Testcase drops 4th, 5th, 7th, error on 9th Points toward a hard-to-detect issue

16

Felix Rath

Sym-Drop

Confjguration Instrs/s Time[h] ICov[%] BCov[%] TSolver[%] MaxMem[GB] Unique errors sym-stream 1725742 0:01 38.96 24.81 0.06 0.16 2 sym-version 232139 0:25 38.87 24.83 83.83 0.15 1 sym-drop 432753 8:00 38.85 25.31 0.02 11.97 1 sym-mod-1 380751 8:00 41.10 27.04 0.79 32.44 sym-mod-5 241116 7:00 40.11 26.11 8.35 33.02 sym-mod-10 4118 8:01 32.11 18.79 78.78 5.34 1

Symbolic Input

• Every packet dropped symbolically

Bugs detected

• Nullpointer-deref when certain packets are dropped

▶ Testcase drops 4th, 5th, 7th, error on 9th ▶ Points toward a hard-to-detect issue

16

Felix Rath

Sym-Mod-X

Confjguration Instrs/s Time[h] ICov[%] BCov[%] TSolver[%] MaxMem[GB] Unique errors sym-stream 1725742 0:01 38.96 24.81 0.06 0.16 2 sym-version 232139 0:25 38.87 24.83 83.83 0.15 1 sym-drop 432753 8:00 38.85 25.31 0.02 11.97 1 sym-mod-1 380751 8:00 41.10 27.04 0.79 32.44 sym-mod-5 241116 7:00 40.11 26.11 8.35 33.02 sym-mod-10 4118 8:01 32.11 18.79 78.78 5.34 1

Symbolic Input

Made fjrst X bytes of each sent packet symbolic

Bugs detected

Error when fjrst 10 bytes of fjrst packet are changed

Testcase changes them to

[0xff, 0x01, 0x01, 0x01, 0x01, 0x67, 0xff, 0xff, 0xff]

Also points toward a hard-to-detect issue

17

Felix Rath

Sym-Mod-X

Confjguration Instrs/s Time[h] ICov[%] BCov[%] TSolver[%] MaxMem[GB] Unique errors sym-stream 1725742 0:01 38.96 24.81 0.06 0.16 2 sym-version 232139 0:25 38.87 24.83 83.83 0.15 1 sym-drop 432753 8:00 38.85 25.31 0.02 11.97 1 sym-mod-1 380751 8:00 41.10 27.04 0.79 32.44 sym-mod-5 241116 7:00 40.11 26.11 8.35 33.02 sym-mod-10 4118 8:01 32.11 18.79 78.78 5.34 1

Symbolic Input

• Made fjrst X bytes of each sent packet symbolic

Bugs detected

Error when fjrst 10 bytes of fjrst packet are changed

Testcase changes them to

[0xff, 0x01, 0x01, 0x01, 0x01, 0x67, 0xff, 0xff, 0xff]

Also points toward a hard-to-detect issue

17

Felix Rath

Sym-Mod-X

Confjguration Instrs/s Time[h] ICov[%] BCov[%] TSolver[%] MaxMem[GB] Unique errors sym-stream 1725742 0:01 38.96 24.81 0.06 0.16 2 sym-version 232139 0:25 38.87 24.83 83.83 0.15 1 sym-drop 432753 8:00 38.85 25.31 0.02 11.97 1 sym-mod-1 380751 8:00 41.10 27.04 0.79 32.44 sym-mod-5 241116 7:00 40.11 26.11 8.35 33.02 sym-mod-10 4118 8:01 32.11 18.79 78.78 5.34 1

Symbolic Input

• Made fjrst X bytes of each sent packet symbolic

Bugs detected

• Error when fjrst 10 bytes of fjrst packet are changed

Testcase changes them to

[0xff, 0x01, 0x01, 0x01, 0x01, 0x67, 0xff, 0xff, 0xff]

Also points toward a hard-to-detect issue

17

Felix Rath

Sym-Mod-X

Confjguration Instrs/s Time[h] ICov[%] BCov[%] TSolver[%] MaxMem[GB] Unique errors sym-stream 1725742 0:01 38.96 24.81 0.06 0.16 2 sym-version 232139 0:25 38.87 24.83 83.83 0.15 1 sym-drop 432753 8:00 38.85 25.31 0.02 11.97 1 sym-mod-1 380751 8:00 41.10 27.04 0.79 32.44 sym-mod-5 241116 7:00 40.11 26.11 8.35 33.02 sym-mod-10 4118 8:01 32.11 18.79 78.78 5.34 1

Symbolic Input

• Made fjrst X bytes of each sent packet symbolic

Bugs detected

• Error when fjrst 10 bytes of fjrst packet are changed

▶ Testcase changes them to

[0xff, 0x01, 0x01, 0x01, 0x01, 0x67, 0xff, 0xff, 0xff]

Also points toward a hard-to-detect issue

17

Felix Rath

Sym-Mod-X

Confjguration Instrs/s Time[h] ICov[%] BCov[%] TSolver[%] MaxMem[GB] Unique errors sym-stream 1725742 0:01 38.96 24.81 0.06 0.16 2 sym-version 232139 0:25 38.87 24.83 83.83 0.15 1 sym-drop 432753 8:00 38.85 25.31 0.02 11.97 1 sym-mod-1 380751 8:00 41.10 27.04 0.79 32.44 sym-mod-5 241116 7:00 40.11 26.11 8.35 33.02 sym-mod-10 4118 8:01 32.11 18.79 78.78 5.34 1

Symbolic Input

• Made fjrst X bytes of each sent packet symbolic

Bugs detected

• Error when fjrst 10 bytes of fjrst packet are changed

▶ Testcase changes them to

[0xff, 0x01, 0x01, 0x01, 0x01, 0x67, 0xff, 0xff, 0xff]

▶ Also points toward a hard-to-detect issue

17

Felix Rath

Future Work

Belief-State

• Defjnition of belief-state
• Standard interface
• Use this for divergence testing
• Would also enable thorough testing using other approaches

Scalability

• 10 symbolic bytes already diffjcult
• Some simple strategies available
• Take a closer look at structure/semantics of packets

18

Felix Rath

Conclusion

The goal: Interoperability testing Main fjndings:

• Software testing approaches (e.g., SymEx) are promising
• But: Common defjnition of an interoperability measure,

e.g., Belief State, desirable for testing Case study:

• Tested picoquic and QUANT
• Concrete test scenarios with symbolic input
• Mock libraries to enable SymEx
• Found fjve bugs

▶ Including two deep interaction issues

19

Felix Rath