Semi-valid Input Coverage for Fuzz Testjng Petar Tsankov , Mohammad - - PowerPoint PPT Presentation

▶
semi valid input coverage for fuzz testjng
SMART_READER_LITE
LIVE PREVIEW

Semi-valid Input Coverage for Fuzz Testjng Petar Tsankov , Mohammad - - PowerPoint PPT Presentation

Jun 15, 2023 240 likes •646 views

Semi-valid Input Coverage for Fuzz Testjng Petar Tsankov , Mohammad Torabi Dashtj, David Basin Instjtute of Informatjon Security ETH Zurich Fuzz Testjng Testjng a PDF Viewer Pass / Fail PDF Viewer Valid inputs Test Oracle Are the PDF fjles

slide-1
SLIDE 1

Semi-valid Input Coverage for Fuzz Testjng

Petar Tsankov, Mohammad Torabi Dashtj, David Basin Instjtute of Informatjon Security ETH Zurich

slide-2
SLIDE 2

2

Fuzz Testjng

Testjng a PDF Viewer

Valid inputs PDF Viewer

Are the PDF fjles displayed correctly?

Pass / Fail Test Oracle

slide-3
SLIDE 3

3

Fuzz Testjng

Fuzz-testjng a PDF viewer testjng

Invalid inputs PDF Viewer

Are there any security faults? (e.g. memory errors)

Pass / Fail Test Oracle

slide-4
SLIDE 4

4

Semi-valid Inputs

PDF Viewer Open

Inputs

slide-5
SLIDE 5

5

Semi-valid Inputs

PDF Viewer Valid Open View

Inputs

slide-6
SLIDE 6

6

Semi-valid Inputs

PDF Viewer Valid Open View

Inputs

slide-7
SLIDE 7

7

Semi-valid Inputs

PDF Viewer Valid Open View Block

Inputs

slide-8
SLIDE 8

8

Semi-valid Inputs

PDF Viewer Valid Open View Block

Inputs

slide-9
SLIDE 9

9

Semi-valid Inputs

PDF Viewer Valid Open View Block

Inputs

slide-10
SLIDE 10

10

Semi-valid Inputs

  • Entjrely-invalid inputs get blocked.
  • Semi-valid inputs are essentjal for fuzz testjng.

Entjrely-invalid PDF Viewer Semi-valid Valid Open View Block

Inputs

slide-11
SLIDE 11

11

Coverage Criteria

  • Low coverage hints at missing test cases.
  • No existjng coverage metric tailored to fuzz testjng.
  • existjng metrics do not tell us how thoroughly we have

tested with semi-valid inputs. Generate

Test set Coverage

Improve Measure

slide-12
SLIDE 12

12

Coverage for Fuzz Testjng

slide-13
SLIDE 13

13

Semi-valid Input Coverage (SVCov)

  • Constraints defjne whether an input is valid or not.

“The third byte is the XOR of the fjrst two bytes.” (C1)

slide-14
SLIDE 14

14

Semi-valid Input Coverage (SVCov)

  • Constraints defjne whether an input is valid or not.

“The third byte is the XOR of the fjrst two bytes.” (C1)

Input Domain

slide-15
SLIDE 15

15

Semi-valid Input Coverage (SVCov)

  • Constraints defjne whether an input is valid or not.

“The third byte is the XOR of the fjrst two bytes.” (C1)

Input Domain

Inputs that satjsfy C1

C1

slide-16
SLIDE 16

16

Semi-valid Input Coverage (SVCov)

  • Constraints defjne whether an input is valid or not.

“The third byte is the XOR of the fjrst two bytes.” (C1)

Input Domain

Inputs that satjsfy C1

C1 C2 C3

slide-17
SLIDE 17

17

Semi-valid Input Coverage (SVCov)

  • Constraints defjne whether an input is valid or not.

“The third byte is the XOR of the fjrst two bytes.” (C1)

Input Domain

Inputs that satjsfy C1

C1 C2 C3

Valid inputs

slide-18
SLIDE 18

18

Semi-valid Input Coverage (SVCov)

  • Constraints defjne whether an input is valid or not.

“The third byte is the XOR of the fjrst two bytes.” (C1)

Input Domain

Inputs that satjsfy C1

C1 C2 C3

Valid inputs Semi-valid input

slide-19
SLIDE 19

19

Semi-valid Input Coverage (SVCov)

  • Constraints defjne whether an input is valid or not.

“The third byte is the XOR of the fjrst two bytes.” (C1)

Input Domain

Inputs that satjsfy C1

C1 C2 C3

Valid inputs Semi-valid input Entjrely-invalid inputs

slide-20
SLIDE 20

20

Semi-valid Input Coverage (SVCov)

  • Constraints defjne whether an input is valid or not.

“The third byte is the XOR of the fjrst two bytes.” (C1)

Input Domain

Inputs that satjsfy C1

C1 C2 C3

Valid inputs Semi-valid input Entjrely-invalid inputs

SVCov = # covered semi-valid partjtjons

# total semi-valid partjtjons

slide-21
SLIDE 21

21

SVCov Propertjes

Independent to test generatjon method. Valid inputs do not contribute to SVCov. The usefulness of SVCov depends on the constraints. 100% SVCov does not guarantee that the tests reveal all faults.

C1 C2 C3

SVCov =

# covered semi-valid partjtjons # total semi-valid partjtjons

slide-22
SLIDE 22

22

Using SVCov

Test set SVCov Fuzzing tool

C1 C2 C3

slide-23
SLIDE 23

23

Using SVCov

Problems with the fuzzing tool

Test set SVCov Fuzzing tool

C1 C2 C3

slide-24
SLIDE 24

24

Using SVCov

Problems with the fuzzing tool

Valid inputs Test set SVCov Fuzzing tool

C1 C2 C3

slide-25
SLIDE 25

25

Using SVCov

Problems with the fuzzing tool

Valid inputs

Missing valid inputs

Test set SVCov Fuzzing tool

C1 C2 C3

slide-26
SLIDE 26

26

Using SVCov

Problems with the fuzzing tool Redundant constraints

Valid inputs

Missing valid inputs

Test set SVCov Fuzzing tool

C1 C2 C3

slide-27
SLIDE 27

27

Case Study

slide-28
SLIDE 28

28

Case Study

Research questjons:

  • RQ1: Feasibility

Can we precisely defjne the semi-valid inputs of the SUT and effjciently measure SVCov?

  • RQ2: Relevance to coverage

Does measuring SVCov provide meaningful informatjon on how to improve a test set's coverage?

  • RQ3: Relevance to discovering faults

Does increasing SVCov result in discovering additjonal faults?

slide-29
SLIDE 29

29

Case Study: Artjfacts

  • Test subject: OpenSwan
  • IKE implementatjon for Linux, 600K LOC.
  • Input specifjcatjon: RFC2407, RFC2408, RFC2409.
  • Fuzzing tool: SecFuzz
  • Mutatjon-based fuzzer for security protocols.
  • Test oracle: MemCheck
  • Detects memory errors.
  • SVCov checker
  • Currently supports only IKE.
slide-30
SLIDE 30

30

RQ1: Feasibility

  • We focused on “must (not) sentences” in the RFCs:

“If a message contains a proposal payload, then the proposal’s next-payload fjeld must be set to 2 or 0.”

  • The specifjcatjon of constraints for IKE is

straightgorward:

– Number of constraints: 217. – Time to extract the constraints: 8 person hours.

  • Negligible overhead for measuring SVCov:

– Time to check all constraints for each test case: 41 ms. – Time to execute a test case: 1000 ms.

slide-31
SLIDE 31

31

RQ2: Relevance to Coverage

0K 10K 20K 30K 0.2 0.4 0.6 0.8 1

Violated SVCov Number of test cases Coverage

  • Many constraints are violated, but not uniquely.
  • Some constraints are never violated.

SVCov (initjal)

Imprecise fuzz-operators Missing valid inputs

  • r fuzz-operators
slide-32
SLIDE 32

32

RQ2: Relevance to Coverage

SVCov analysis

  • Problems in the fuzzing tool
  • Imprecision in the “insert payload” fuzz operator.
  • Insert random numbers limited to [0, 100].
  • ...
  • Missing valid inputs
  • No valid inputs for IPv6 and ASN.1 X500 DN.
  • Redundant constraints

C1 C2 C3

slide-33
SLIDE 33

33

RQ2: Relevance to Coverage

0K 10K 20K 30K 0.2 0.4 0.6 0.8 1

Violated SVCov Number of test cases Coverage

SVCov (afuer improvements)

  • SVCov improved from 41% to 89%.
  • All constraints are violated.
  • 9% of the constraints are not uniquely violated.
slide-34
SLIDE 34

34

RQ3: Relevance to Discovering Faults

OpenSwan

MemCheck

SecFuzz Unallocated memory access

  • A previously unknown security fault revealed afuer

improving SVCov.

Valid input Test case

  • The valid input was missing in the fjrst experiment.
  • The test case belongs to a semi-valid partjtjon.
slide-35
SLIDE 35

35

SVCov Contributjons

C1 C2 C3

Easy-to-use coverage for fuzz testjng Independent of the fuzz-testjng technique Pinpoint subtle problems in fuzz testjng Promising initjal empirical results

slide-36
SLIDE 36

36

Backup Slides

slide-37
SLIDE 37

37

Redundant Constraints

Input Domain

  • Constraint C1 is redundant.
  • removing C1 does not change the set of valid inputs.
  • Constraint C1 cannot be uniquely violated.
  • Any input that violates C1 also violates C2.

C1 C2 C3

slide-38
SLIDE 38

38

Missing Valid Inputs

  • To violate a constraint we need an input that

satjsfjes the constraint non-vacuously. Cguard Ctarget

Violated Vacuously satjsfjed Non-vacuously satjsfjed

slide-39
SLIDE 39

39

Case Study: Setup

OpenSwan (initjator) OpenSwan (responder)

MemCheck

SUT SecFuzz Valid inputs Fuzzed inputs

  • We measure and report SVCov of the fuzzed inputs.
  • Measure SVCov of the valid inputs to check

for missing inputs.