Learning to Fuzz from Symbolic Execution with Application to Smart - - PowerPoint PPT Presentation

Learning to Fuzz from Symbolic Execution with Application to Smart Contracts

Mislav Balunovic Nodar Ambroladze Petar Tsankov Martin Vechev Jingxuan He

Random Fuzzing vs. Symbolic Execution

Random Fuzzing Symbolic Execution Speed Inputs Coverage

Fast Ineffective Effective Slow Low Low

1

2

Initial

. . . . . . . . . . . . . . . . . .

, deposit(), P ether , setOwner( ) , withdraw(P) steals P ether from

Smart Contract Testing: Challenge

3

Smart Contract Testing: Challenge

Wanted: Transaction sequences that thoroughly explore the state space

4

Random Fuzzing vs. Symbolic Execution

Imitation Learning based Fuzzer

ILF (this work)

Fast Effective High

Random Fuzzing Symbolic Execution Speed Inputs Coverage

Fast Ineffective Effective Slow Low Low

5

• Analyzing Ethereum’s Contract Topology. Kiffer et al.. IMC ’18

~120K contracts ~16K clusters

Imitation Learning

6

Demonstration

Symbolic execution Human expert

Demonstration

Robot Fuzzer

Learning to Fuzz from Symbolic Execution

Training

Transaction sequences

Symbolic execution expert

Smart contracts ≈ 15K contracts

Fuzzing

New contract Fuzzing policy (neural networks)

7

Coverage Vulnerability Report

Learning to Fuzz from Symbolic Execution

Training

Transaction sequences Smart contracts

Symbolic execution expert

Fuzzing

New contract Coverage Vulnerability Report Fuzzing policy (neural networks)

8

Smart Contract Fuzzing Policy

Tested Contract Transaction

Feedback

! = ($ ̅ & , ()*+),, -./0*!)

= + + +

$ ̅ & ()*+),

• ./0*!

Example: a Uniformly Random Policy

2*3$/,.(4)

:

2*3$/,.(536*-!0,)($))

:

72*3$/,. 0, 9: $ is payable C(0) = 1

• therwise

:

2*3$/,.(5JKLJM5)

:

may modify blockchain state

9

Fuzzing Policy

Neural Network Fuzzing Policy

GRUfuzz at step 3 FCNfunc GRUint + FCNint FCNsender FCNamount GRUfuzz at step 3 − 1 GRUfuzz at step 3 + 1

ℎQRS

ℎQ

10

Feature

• f $

QRS

ℎQ hidden state hidden state

$

Q

̅ &Q

()*+),

Q

• ./0*!Q

Features of 4

Neural Network Fuzzing Policy – Fuzzing State

GRUfuzz at step 3 − 1 GRUfuzz at step 3

Feature of $

QRS

Current hidden state

11

e.g., Coverage, opcodes, function name. (can be dynamic)

Last hidden state [3.5, 0.3, 4.0, …] [1, 6.2, 5, …] [1.2, 8.7, 2.5, …]

Neural Network Fuzzing Policy

GRUfuzz at step 3 FCNfunc GRUint + FCNint FCNsender FCNamount GRUfuzz at step 3 − 1 GRUfuzz at step 3 + 1

ℎQRS

ℎQ

12

Feature

• f $

QRS

ℎQ hidden state hidden state

$

Q

̅ &Q

()*+),

Q

• ./0*!Q

Features of 4

Neural Network Fuzzing Policy – Function

FCNfunc + Softmax

13

Current hidden state [1.2, 8.7, 2.5, …] [[1, 6.2, 5, …], [4, 3.7, 6, …], … [2, 9.2, 7, …]] Feature of 4

SetOwner Deposit Withdraw

Withdraw

Neural Network Fuzzing Policy

GRUfuzz at step 3 FCNfunc GRUint + FCNint FCNsender FCNamount GRUfuzz at step 3 − 1 GRUfuzz at step 3 + 1

ℎQRS

ℎQ

14

Feature

• f $

QRS

ℎQ hidden state hidden state

$

Q

̅ &Q

()*+),

Q

• ./0*!Q

Features of 4

One-hot

Neural Network Fuzzing Policy – Arguments

GRUint at step 0 GRUint at step 1

15

. . .

FCNint FCNint

[0, 0, 1, 0, …] Current hidden state [1.2, 8.7, 2.5, …] 1 0x10 0x800

• 0x200. . .

1

1 0x10 0x800

• 0x200. . .

0x200

Distribution over 50 seed integer values from expert

Neural Network Fuzzing Policy

GRUfuzz at step 3 FCNfunc GRUint + FCNint FCNsender FCNamount GRUfuzz at step 3 − 1 GRUfuzz at step 3 + 1

ℎQRS

ℎQ

16

Feature

• f $

QRS

ℎQ hidden state hidden state

$

Q

̅ &Q

()*+),

Q

• ./0*!Q

Features of 4

Learning to Fuzz from Symbolic Execution

Training

Transaction sequences Smart contracts

Symbolic execution expert

Fuzzing

New contract Coverage Vulnerability Report Fuzzing policy (neural networks)

17

Symbolic Execution Expert

. . .

Execute E x e c u t e

TQUQV TS TW

18

Symbolic:

VerX

S&P 2020

!S !W Revisit Revisit Revisit

Learning to Fuzz from Symbolic Execution

Training

Transaction sequences Smart contracts

Symbolic execution expert

Fuzzing

New contract Coverage Vulnerability Report Fuzzing policy (neural networks)

19

Training Neural Network Fuzzing Policy

20

$

Q

̅ &Q ()*+),

Q

• ./0*!Q

!Q by expert Cross-Entropy loss Back Prop.

NN Policy at step 3 − 1

Inference

NN Policy at step 3

Features !Q by expert Hidden State

Learning to Fuzz from Symbolic Execution

Training

Transaction sequences Smart contracts

Symbolic execution expert

Fuzzing

New contract Coverage Vulnerability Report Fuzzing policy (neural networks)

21

ILF System: Coverage & Vulnerability Detection

• Instruction coverage.
• Basic block coverage.

22

• Locking: The contract cannot send out but can receive ether.
• Leaking: An attacker can steal ether from the contract.
• Suicidal: An attacker can deconstruct the contract.
• Block Dependency: Ether transfer depends on block state variables.
• Unhandled Exception: Root call does not catch exceptions from child calls.
• Controlled Delegatecall: Transaction parameters explicitly flow into

arguments of a delegatecall instruction.

Evaluation

23

• 18,496 Contracts (5,013 Large & 13,483 Small)
• 5-fold Cross Validation
• Echidna
• UNIF
• EXPERT
• MAIAN
• ContractFuzzer
• Vulnerability Detection
• Coverage & Speed
• Fuzzing Components
• Case Study

Coverage: ILF vs. Fuzzers

0% 20% 40% 60% 80% 100% 200 400 600 800 1000

Number of Transactions

• Instr. Coverage

ILF UNIF Echidna

24

0% 20% 40% 60% 80% 100% 200 400 600 800 1000

Number of Transactions

• Instr. Coverage

ILF UNIF Echidna

Large contracts Small contracts

Coverage: ILF vs. Symbolic Expert

25

50% 60% 70% 80% 90% 100% Small Contracts Large Contracts

• Instr. Coverage

EXPERT ILF (#tx same as EXPERT) ILF (2k txs)

Small: 30 txs, 547s Large: 49 txs, 2,580s Small: 13s Large: 17s 148 txs/s

Vulnerability Detection

0% 20% 40% 60% 80% 100% Leaking Suicidal Locking

% of True Vulnerabilities

ILF UNIF MAIAN 0% 20% 40% 60% 80% 100% Block Dependency Unhandled Exception Controlled Delegatecall

% of True Vulnerabilities

ILF UNIF ContractFuzzer

26

∪ ∪

13 FPs 6 FPs ILF: 0 FPs

Importance of Policy Components

27

50% 60% 70% 80% 90% 100% Coverage Leaking

ILF ILF-func ILF-args ILF-sender ILF-amount

Most important

All components are necessary

Summary

28

?

Q & A