The Human Component in Automated Bug Finding Christian Holler - - PowerPoint PPT Presentation

▶
the human component in automated bug finding
SMART_READER_LITE
LIVE PREVIEW

The Human Component in Automated Bug Finding Christian Holler - - PowerPoint PPT Presentation

Aug 14, 2022 381 likes •1.13k views

The Human Component in Automated Bug Finding Christian Holler (:decoder) Staff Security Engineer ~ 16M lines source code ~ 16M lines source code Last month: 340 authors with 2,475 commits Interfaces Media Formats Markup Fonts Languages

slide-1
SLIDE 1

The Human Component in Automated Bug Finding

Christian Holler (:decoder) Staff Security Engineer

slide-2
SLIDE 2
slide-3
SLIDE 3

~ 16M lines source code

slide-4
SLIDE 4

Last month: 340 authors with 2,475 commits ~ 16M lines source code

slide-5
SLIDE 5

Media Formats Markup Languages Interfaces JavaScript Fonts Networking

slide-6
SLIDE 6

Domain Knowledge

Developers know...

slide-7
SLIDE 7

Domain Knowledge

Developers know...

… code architecture/contracts

slide-8
SLIDE 8

Domain Knowledge

Developers know...

… code architecture/contracts … expected behavior

slide-9
SLIDE 9

Domain Knowledge

Developers know...

… code architecture/contracts … expected behavior … weaknesses

slide-10
SLIDE 10

Domain Knowledge

Developers know...

… code architecture/contracts … expected behavior … weaknesses

“Lone warrior” approach not sustainable

slide-11
SLIDE 11

Mutual Trust Relationship

slide-12
SLIDE 12

The Do’s and Don’ts

slide-13
SLIDE 13

Ninja Style

slide-14
SLIDE 14

Ninja Style

Build fuzzer alone and in secret

slide-15
SLIDE 15

Ninja Style

Build fuzzer alone and in secret Rapid fire bugs at developers

slide-16
SLIDE 16

Ninja Style

Build fuzzer alone and in secret Rapid fire bugs at developers “They’ll never know what hit them. Tehehe!!11oneeleven”

slide-17
SLIDE 17

Defensive Behavior

slide-18
SLIDE 18

Overwhelmed

Defensive Behavior

slide-19
SLIDE 19

Overwhelmed

Defensive Behavior

Lack of Resources

slide-20
SLIDE 20

Overwhelmed

Defensive Behavior

Lack of Resources Code Ownership Bias

slide-21
SLIDE 21

21

slide-22
SLIDE 22

22

“please stop filing fuzz bugs for the next few weeks until they can be addressed.”

slide-23
SLIDE 23

DON’T

Surprise your developers

slide-24
SLIDE 24

DON’T

Surprise your developers Act superior or adversarial

slide-25
SLIDE 25

DON’T

Surprise your developers Act superior or adversarial Assume equal priorities

slide-26
SLIDE 26

DO: Kickoff Meeting

slide-27
SLIDE 27

DO: Kickoff Meeting

Developers, Fuzzing and Management

slide-28
SLIDE 28

DO: Kickoff Meeting

Developers, Fuzzing and Management Show previous success stories

slide-29
SLIDE 29

DO: Kickoff Meeting

Developers, Fuzzing and Management Show previous success stories Offer your help, ask about problems

slide-30
SLIDE 30

DO: Kickoff Meeting

Developers, Fuzzing and Management Show previous success stories Offer your help, ask about problems Define Goals - Allocate Resources

slide-31
SLIDE 31

DO: Kickoff Meeting

Developers, Fuzzing and Management Show previous success stories Offer your help, ask about problems Define Goals - Allocate Resources Educate on Requirements

slide-32
SLIDE 32

DO: Kickoff Meeting

Developers, Fuzzing and Management Show previous success stories Offer your help, ask about problems Define Goals - Allocate Resources Educate on Requirements

slide-33
SLIDE 33

DO: Kickoff Meeting

Developers, Fuzzing and Management Show previous success stories Offer your help, ask about problems Define Goals - Allocate Resources Educate on Requirements

slide-34
SLIDE 34

Requirements and Goals

Developer Fuzzing

slide-35
SLIDE 35

Requirements and Goals

Developer Fuzzing

slide-36
SLIDE 36

Requirements and Goals

Developer Fuzzing

slide-37
SLIDE 37

Requirements and Goals

Developer Fuzzing

slide-38
SLIDE 38

Bugs must be fixed Example Requirement

slide-39
SLIDE 39

Bugs must be fixed

“That bug isn’t interesting, please ignore it.”

Example Requirement

slide-40
SLIDE 40

Bugs must be fixed

“... but that’s not a bug.”

Example Requirement

slide-41
SLIDE 41

“Contract” about what constitutes a bug Bugs must be fixed

“... but that’s not a bug.”

Example Requirement

slide-42
SLIDE 42

$ js js>

slide-43
SLIDE 43

$ js js> print("Hello watman") Hello watman js>

slide-44
SLIDE 44

$ js js> print("Hello watman") Hello watman js> crash(); Hit MOZ_CRASH(forced crash) at shell/js.cpp:3700 Segmentation fault

slide-45
SLIDE 45

$ js --fuzzing-safe js>

slide-46
SLIDE 46

$ js --fuzzing-safe js> crash(); typein:1:1 ReferenceError: crash is not defined

slide-47
SLIDE 47

Requirements vs. Goals

Developer Fuzzing

slide-48
SLIDE 48

Fuzzblockers

slide-49
SLIDE 49

Fuzzblockers

Disruptive effect on fuzzing operations

slide-50
SLIDE 50

Fuzzblockers

Disruptive effect on fuzzing operations

(e.g. highly frequent, resource intensive) + Hard to avoid

slide-51
SLIDE 51

Fuzzblockers

Disruptive effect on fuzzing operations

(e.g. highly frequent, resource intensive) + Hard to avoid Highest Priority for Fuzzing (Usually) low priority for developers

slide-52
SLIDE 52

Fuzzblockers

Disruptive effect on fuzzing operations

(e.g. highly frequent, resource intensive) + Hard to avoid Highest Priority for Fuzzing (Usually) low priority for developers

Try writing a fix yourself!

slide-53
SLIDE 53

I want you to fix a bug

slide-54
SLIDE 54

I want you to fix a bug

You learn something about the code

slide-55
SLIDE 55

I want you to fix a bug

You learn something about the code You learn something about development

slide-56
SLIDE 56

I want you to fix a bug

You learn something about the code You learn something about development You can progress faster

slide-57
SLIDE 57

I want you to fix a bug

You learn something about the code You learn something about development You can progress faster Developers will be happy

slide-58
SLIDE 58

When?

slide-59
SLIDE 59

When to Fuzz?

slide-60
SLIDE 60

When to Fuzz?

slide-61
SLIDE 61

When to Fuzz?

slide-62
SLIDE 62

When to Fuzz?

slide-63
SLIDE 63

When to Fuzz?

Earliest version with well-defined behavior

slide-64
SLIDE 64

When to Fuzz?

Earliest version with well-defined behavior Could help developers

slide-65
SLIDE 65

When to Fuzz?

As early as possible (*)

Earliest version with well-defined behavior Could help developers

slide-66
SLIDE 66

DO: Simple Steps to reproduce

slide-67
SLIDE 67

DO: Simple Steps to reproduce

slide-68
SLIDE 68

DO: Measure Code Coverage

slide-69
SLIDE 69

DO: Measure Code Coverage SHARE!

slide-70
SLIDE 70

DO: Educate

slide-71
SLIDE 71

Fuzzing is Teamwork

slide-72
SLIDE 72

Thank You