[PPT] - When Abstraction Fails Andreas Zeller Saarland University PowerPoint Presentation

SLIDE 1

When Abstraction Fails

Andreas Zeller Saarland University http://www.st.cs.uni-sb.de/

SLIDE 2 2

Mining

SLIDE 3 3

The First Bug

(September 9, 1947)

SLIDE 4 4

SLIDE 5 5

How to Debug

(Sommerville 2004)

Locate error Design error repair Repair error Re-test program

SLIDE 6

Locate error

6

How to Debug

(not Sommerville 2004)

Design error repair Repair error Re-test program

SLIDE 7 7

The Traffjc Principle

T R A F F I C rack the problem eproduce utomate ind Origins

cus

solate

rrect

SLIDE 8 8

FAIL

A Guide to Systematic Debugging

ANDREAS ZELLER

WHY

PROGRAMS

“The definitive book on debugging” – WALTER F. TICHY TU Karlsruhe

On Display!

SLIDE 9 9

Pythagoras Heraclitus Parmenides Diogenes Euclid Ptolemy Socrates

Plato Aristotle

SLIDE 10 10

Idealism Empiricism Abstract Concrete Deduction Induction Models Facts

SLIDE 11 11

Code Runs Static Dynamic Future Past Prevention Cure

SLIDE 12 12

SLIDE 13 13

double bug(double z[], int n) { int i, j; i = 0; for (j = 0; j < n; j++) { i = i + j + 1; z[i] = z[i] * (z[0] + 1.0); } return z[n]; }

bug.c

SLIDE 14 14

What is the cause

f this failure?

SLIDE 15 15

Alternate world Effect does not occur

Causality

Actual world Effect does occur Causes

SLIDE 16 16

bug.c

double bug(double z[], int n) { int i, j; i = 0; for (j = 0; j < n; j++) { i = i + j + 1; z[i] = z[i] * (z[0] + 1.0); } return z[n]; } ✘

SLIDE 17 17

double bug(double z[], int n) { int i, j; i = 0; for (j = 0; j < n; j++) { i = i + j + 1; z[i] = z[i] * (z[0] + 1.0); } return z[n]; } ✔

empty.c

SLIDE 18 18

Alternate world empty.c: GCC works fine

Causes as Difgerences

Actual world bug.c: GCC crashes Cause: bug.c

SLIDE 19 19

Actual Causes

Actual cause “The” cause (actual cause) is a minimal difference

SLIDE 20 20

double bug(double z[], int n) { int i, j; i = 0; for (j = 0; j < n; j++) { i = i + j + 1; z[i] = z[i] * (z[0] + 1.0); } return z[n]; } ✔

Isolating Causes

SLIDE 21 21

double bug(double z[], int n) { int i, j; i = 0; for (j = 0; j < n; j++) { i = i + j + 1; z[i] = z[i] * (z[0] + 1.0); } return z[n]; } ✔

Isolating Causes

SLIDE 22 22

Isolating Causes

double bug(double z[], int n) { int i, j; i = 0; for (j = 0; j < n; j++) { i = i + j + 1; z[i] = z[i] * (z[0] + 1.0); } return z[n]; } ✘

SLIDE 23 23

Isolating Causes

double bug(double z[], int n) { int i, j; i = 0; for (j = 0; j < n; j++) { i = i + j + 1; z[i] = z[i] * (z[0] + 1.0); } return z[n]; }

Actual cause narrowed down

SLIDE 24 24

double bug(double z[], int n) { int i, j; i = 0; for (j = 0; j < n; j++) { i = i + j + 1; z[i] = z[i] * (z[0] + 1.0); } return z[n]; } ✔

Isolating Causes

SLIDE 25 25

Isolating Causes

double bug(double z[], int n) { int i, j; i = 0; for (j = 0; j < n; j++) { i = i + j + 1; z[i] = z[i] * (z[0] + 1.0); } return z[n]; } ✘

SLIDE 26 26

Isolating Causes

double bug(double z[], int n) { int i, j; i = 0; for (j = 0; j < n; j++) { i = i + j + 1; z[i] = z[i] * (z[0] + 1.0); } return z[n]; }

Actual cause of the GCC crash

SLIDE 27 27

Alternate world Actual world

Isolating Causes

Mixed world

✔ ✘

Test

?

SLIDE 28 28

Alternate world Actual world

Isolating Causes

Mixed world

✔ ✘

Test

?

“+ 1.0”

SLIDE 29 29

The Scientific Method

Hypothesis

Problem Report Code Run More Runs

Prediction Experiment

Observation + Conclusion

Hypothesis is supported: refine hypothesis Hypothesis is rejected: create new hypothesis

Diagnosis

SLIDE 30

The Scientific Method

30

SLIDE 31 31

Circumstances

Any aspect that may influence a problem is

a circumstance:

Aspects of the problem environment
Individual steps of the problem history

SLIDE 32 32

Experimentation

By experimentation, one finds out whether a

circumstance is relevant or not:

Omit the circumstance and try to

reproduce the problem.

The circumstance is relevant iff the

problem no longer occurs.

SLIDE 33 33

Mozilla Bug #24735

Ok the following operations cause mozilla to crash consistently on my machine

> Start mozilla
> Go to bugzilla.mozilla.org
> Select search for bug
> Print to file setting the bottom and right margins to .50

(I use the file /var/tmp/netscape.ps)

> Once it's done printing do the exact same thing again on

the same file (/var/tmp/netscape.ps)

> This causes the browser to crash with a segfault

SLIDE 34 34

<SELECT NAME="op_sys" MULTIPLE SIZE=7> <OPTION VALUE="All">All<OPTION VALUE="Windows 3.1">Windows 3.1<OPTION VALUE="Windows 95">Windows 95<OPTION VALUE="Windows 98">Windows 98<OPTION VALUE="Windows ME">Windows ME<OPTION VALUE="Windows 2000">Windows 2000<OPTION VALUE="Windows NT">Windows NT<OPTION VALUE="Mac System 7">Mac System 7<OPTION VALUE="Mac System 7.5">Mac System 7.5<OPTION VALUE="Mac System 7.6.1">Mac System 7.6.1<OPTION VALUE="Mac System 8.0">Mac System 8.0<OPTION VALUE="Mac System 8.5">Mac System 8.5<OPTION VALUE="Mac System 8.6">Mac System 8.6<OPTION VALUE="Mac System 9.x">Mac System 9.x<OPTION VALUE="MacOS X">MacOS X<OPTION VALUE="Linux">Linux<OPTION VALUE="BSDI">BSDI<OPTION VALUE="FreeBSD">FreeBSD<OPTION VALUE="NetBSD">NetBSD<OPTION VALUE="OpenBSD">OpenBSD<OPTION VALUE="AIX">AIX<OPTION VALUE="BeOS">BeOS<OPTION VALUE="HP-UX">HP-UX<OPTION VALUE="IRIX">IRIX<OPTION VALUE="Neutrino">Neutrino<OPTION VALUE="OpenVMS">OpenVMS<OPTION VALUE="OS/2">OS/2<OPTION VALUE="OSF/ 1">OSF/1<OPTION VALUE="Solaris">Solaris<OPTION VALUE="SunOS">SunOS<OPTION VALUE="other">other</SELECT> </td> <td align=left valign=top> <SELECT NAME="priority" MULTIPLE SIZE=7> <OPTION VALUE="--">--<OPTION VALUE="P1">P1<OPTION VALUE="P2">P2<OPTION VALUE="P3">P3<OPTION VALUE="P4">P4<OPTION VALUE="P5">P5</SELECT>

bugzilla.mozilla.org

What’s relevant in here?

SLIDE 35 35

Why simplify?

Ease of communication. A simplified test

case is easier to communicate.

Easier debugging. Smaller test cases result

in smaller states and shorter executions.

Identify duplicates. Simplified test cases

subsume several duplicates.

SLIDE 36 36

The Gecko BugAThon

Download the Web page to your machine.
Using a text editor, start removing HTML

from the page. Every few minutes, make sure it still reproduces the bug.

Code not required to reproduce the bug

can be safely removed.

When you’ve cut away as much as you can,

you’re done.

SLIDE 37 37

Rewards

5 bugs - invitation to the Gecko launch party 10 bugs - the invitation, plus an attractive Gecko stuffed animal 12 bugs - the invitation, plus an attractive Gecko stuffed animal autographed by Rick Gessner, the Father of Gecko 15 bugs - the invitation, plus a Gecko T-shirt 20 bugs - the invitation, plus a Gecko T-shirt signed by the whole raptor team

SLIDE 38 38

Proceed by binary search. Throw away half

the input and see if the output is still wrong.

If not, go back to the previous state and

discard the other half of the input.

Binary Search

HTML input

✘ ✔ ✘ ✔

SLIDE 39 39

Simplified Input

Simplified from 896 lines to one single line
Required 12 tests only

SLIDE 40 40

Benefits

Ease of communication. All one needs is

“Printing <SELECT> crashes”.

Easier debugging. We can directly focus on

the piece of code that prints <SELECT>.

Identify duplicates. Check other test cases

whether they’re <SELECT>-related, too.

SLIDE 41 41

Why automate?

Manual simplification is tedious.
Manual simplification is boring.
We have machines for tedious and boring

tasks.

SLIDE 42 42

Basic Idea

We set up an automated test that checks

whether the failure occurs or not (= Mozilla crashes when printing or not)

We implement a strategy that realizes the

binary search.

SLIDE 43 43

Automated Test

1. Launch Mozilla
2. Replay (previously recorded) steps from

problem report

3. Wait to see whether
Mozilla crashes (= the test fails)
Mozilla still runs (= the test passes)
4. If neither happens, the test is unresolved

SLIDE 44 44

Binary Search ✔ ✘

✔

What do we do if both halves pass?

<SELECT NAME="priority" MULTIPLE SIZE=7> ✔

✘ ✔ ✘

SLIDE 45 45

Configuration

All circumstances

C = {δ1, δ2, . . . }

Configuration

c = {δ1, δ2, . . . δn} c ⊆ C

Circumstance

δ

SLIDE 46 46

Tests

test(c) ∈ {✔, ✘, ?}

Testing function

test(c✘) = ✘

Failure-inducing configuration Relevant configuration

∀δi ∈ c

✘ · test

c

✘ \ {δi}

≠ ✘

c

✘ ⊆ c✘

SLIDE 47 47

Binary Strategy

If removing first half fails…

test(c✘ \ c1) = ✘ = ⇒ c✘ = c✘ \ c1

If removing second half fails…

test(c✘ \ c2) = ✘ = ⇒ c✘ = c✘ \ c2

Otherwise, increase granularity:

c✘ = c1 ∪ c2 ∪ c3 ∪ c4 c✘ = c1 ∪ c2 ∪ c3 ∪ c4 ∪ c5 ∪ c6 ∪ c7 ∪ c8 c✘ = c1 ∪ c2

Split input

SLIDE 48 48

General Strategy

Split input into n parts (initially 2)

c✘ = c1 ∪ c2 ∪ · · · ∪ cn

If some removal fails…

∃i ∈ {1, . . . , n} · test(c✘ \ ci) = ✘ = ⇒ c✘ = c✘ \ ci n = max(n − 1, 2)

Otherwise, increase granularity

c✘ = c✘ n = 2n

SLIDE 49 49

ddmin in a Nutshell

ddmin(c✘) = ddmin(c

✘, 2)

c

✘ = ddmin(c✘) is a relevant configuration

                 c

✘

if |c

✘| = 1

ddmin c

✘ \ ci, max(n − 1, 2)

else if ∃i ∈ {1..n} · test(c

✘ \ ci) = ✘

(“some removal fails”) ddmin c

✘, min(2n,

c

✘

)
else if n <
c

✘

(“increase granularity”)

c

✘

therwise

ddmin(c

✘, n) =

with

c

✘ = c1 ∪ c2 ∪ · · · ∪ cn

∀ci, cj · ci ∩ cj = ∅ ∧ |ci| ≈ |cj|

where

SLIDE 50 50

def _ddmin(circumstances, n): while len(circumstances) >= 2: subsets = split(circumstances, n) some_complement_is_failing = 0 for subset in subsets: complement = listminus(circumstances, subset) if test(complement) == FAIL: circumstances = complement n = max(n - 1, 2) some_complement_is_failing = 1 break if not some_complement_is_failing: if n == len(circumstances): break n = min(n * 2, len(circumstances)) return circumstances

SLIDE 51 51 Input: <SELECT NAME="priority" MULTIPLE SIZE=7> 40 characters ✘ <SELECT NAME="priority" MULTIPLE SIZE=7> 0 characters ✔ 1 <SELECT NAME="priority" MULTIPLE SIZE=7> 20 ✔ 2 <SELECT NAME="priority" MULTIPLE SIZE=7> 20 ✔ 3 <SELECT NAME="priority" MULTIPLE SIZE=7> 30 ✔ 4 <SELECT NAME="priority" MULTIPLE SIZE=7> 30 ✘ 5 <SELECT NAME="priority" MULTIPLE SIZE=7> 20 ✔ 6 <SELECT NAME="priority" MULTIPLE SIZE=7> 20 ✘ 7 <SELECT NAME="priority" MULTIPLE SIZE=7> 10 ✔ 8 <SELECT NAME="priority" MULTIPLE SIZE=7> 10 ✔ 9 <SELECT NAME="priority" MULTIPLE SIZE=7> 15 ✔ 10 <SELECT NAME="priority" MULTIPLE SIZE=7> 15 ✔ 11 <SELECT NAME="priority" MULTIPLE SIZE=7> 15 ✘ 12 <SELECT NAME="priority" MULTIPLE SIZE=7> 10 ✔ 13 <SELECT NAME="priority" MULTIPLE SIZE=7> 10 ✔ 14 <SELECT NAME="priority" MULTIPLE SIZE=7> 10 ✔ 15 <SELECT NAME="priority" MULTIPLE SIZE=7> 12 ✔ 16 <SELECT NAME="priority" MULTIPLE SIZE=7> 13 ✔ 17 <SELECT NAME="priority" MULTIPLE SIZE=7> 12 ✔ 18 <SELECT NAME="priority" MULTIPLE SIZE=7> 13 ✘ 19 <SELECT NAME="priority" MULTIPLE SIZE=7> 10 ✔ 20 <SELECT NAME="priority" MULTIPLE SIZE=7> 10 ✔ 21 <SELECT NAME="priority" MULTIPLE SIZE=7> 11 ✔ 22 <SELECT NAME="priority" MULTIPLE SIZE=7> 10 ✘ 23 <SELECT NAME="priority" MULTIPLE SIZE=7> 7 ✔ 24 <SELECT NAME="priority" MULTIPLE SIZE=7> 8 ✔ 25 <SELECT NAME="priority" MULTIPLE SIZE=7> 7 ✔ 26 <SELECT NAME="priority" MULTIPLE SIZE=7> 8 ✔ 27 <SELECT NAME="priority" MULTIPLE SIZE=7> 9 ✔ 28 <SELECT NAME="priority" MULTIPLE SIZE=7> 9 ✔ 29 <SELECT NAME="priority" MULTIPLE SIZE=7> 9 ✔ 30 <SELECT NAME="priority" MULTIPLE SIZE=7> 9 ✔ 31 <SELECT NAME="priority" MULTIPLE SIZE=7> 8 ✔ 32 <SELECT NAME="priority" MULTIPLE SIZE=7> 9 ✔ 33 <SELECT NAME="priority" MULTIPLE SIZE=7> 8 ✘ 34 <SELECT NAME="priority" MULTIPLE SIZE=7> 7 ✔ 35 <SELECT NAME="priority" MULTIPLE SIZE=7> 7 ✔ 36 <SELECT NAME="priority" MULTIPLE SIZE=7> 7 ✔ 37 <SELECT NAME="priority" MULTIPLE SIZE=7> 7 ✔ 38 <SELECT NAME="priority" MULTIPLE SIZE=7> 7 ✔ 39 <SELECT NAME="priority" MULTIPLE SIZE=7> 6 ✔ 40 <SELECT NAME="priority" MULTIPLE SIZE=7> 7 ✔ 41 <SELECT NAME="priority" MULTIPLE SIZE=7> 7 ✔ 42 <SELECT NAME="priority" MULTIPLE SIZE=7> 7 ✔ 43 <SELECT NAME="priority" MULTIPLE SIZE=7> 7 ✔ 44 <SELECT NAME="priority" MULTIPLE SIZE=7> 7 ✔ 45 <SELECT NAME="priority" MULTIPLE SIZE=7> 7 ✔ 46 <SELECT NAME="priority" MULTIPLE SIZE=7> 7 ✔ 47 <SELECT NAME="priority" MULTIPLE SIZE=7> 7 ✔ 48 <SELECT NAME="priority" MULTIPLE SIZE=7> 7 ✔ Result: <SELECT>

ddmin at Work

SLIDE 52 52

Binary Search

If

there is only one failure-inducing

circumstance, and

all configurations that include this

circumstance fail, the number of tests is t ≤ log2(|c✘|)

SLIDE 53 53

More Simplification

Simplified failure-inducing fuzz input:

FLEX crashes on 2,121 or more non-

newline characters

NROFF crashes on “\D^J%0F” or “\302\n”
CRTPLOT crashes on “t”

SLIDE 54 54

Minimal Interaction

Ok the following operations cause mozilla to crash consistently on my machine

> Start mozilla
> Go to bugzilla.mozilla.org
> Select search for bug
> Print to file setting the bottom and right margins to .50

(I use the file /var/tmp/netscape.ps)

> Once it's done printing do the exact same thing again on

the same file (/var/tmp/netscape.ps)

> This causes the browser to crash with a segfault

SLIDE 55 55

Minimal Interaction

Basic idea: Apply ddmin to recorded user interaction

To reproduce the Mozilla printing crash:
Press P while holding Alt
Press mouse button 1
Release mouse button 1

SLIDE 56 56

Delta Debugging

Delta Debugging isolates failure causes automatically: Inputs: 1 of 900 HTML lines in Mozilla Code changes: 1 of 8,721 code changes in GDB Threads: 1 of 3.8 bln thread switches in Scene.java Messages: n of m Java method calls (in progress) Fully automatic + purely test-based

SLIDE 57 57

Isolating Causes

double bug(double z[], int n) { int i, j; i = 0; for (j = 0; j < n; j++) { i = i + j + 1; z[i] = z[i] * (z[0] + 1.0); } return z[n]; }

Actual cause of the GCC crash

SLIDE 58 58

✘

1. The programmer creates a

defect – an error in the code.

2. When executed, the defect

creates an infection – an error in the state.

3. The infection propagates.
4. The infection causes a failure.

From Defect to Failure

✘

✘ ✘ ✘

Variables

This infection chain must be traced back – and broken. t

SLIDE 59 59

Tracing Infections

✘

For every infection, we must find the earlier

infection that causes it.

Program analysis tells us possible causes

SLIDE 60 60

Tracing Infections

✘

SLIDE 61 61

SLIDE 62 62

Sane state Infected state

Causes in State

The difference causes GCC to crash!

SLIDE 63 63

Sane state Infected state

Search in Space

Mixed state

✔ ✘

Test

?

SLIDE 64 64

Search in Space

100 200 300 400 500 600 700 800 900 5 10 15 20 25 30 35 40 45 Deltas Tests executed Delta Debugging Log cpass cfail

SLIDE 65 65

Search in Space

100 200 300 400 500 600 700 800 900 5 10 15 20 25 30 35 40 45 Deltas Tests executed Delta Debugging Log cpass cfail

first_loop_store_insn→fld[1].rtx→fld[1].rtx→ fld[3].rtx→fld[1].rtx→code == PLUS

SLIDE 66 66

Sane state Infected state

Search in Space

Mixed state

✔ ✘

Test

?

SLIDE 67 67

Sane state Infected state

Search in Space

Mixed state

✔ ✘

Test

?

SLIDE 68 68

Passing run Failing run

Search in Time

t

SLIDE 69 69

Passing run Failing run

Search in Time

t

<PLUS node> <PLUS node> link→fld[0].rtx→fld[0].rtx == link

SLIDE 70

Passing run Failing run

t

<PLUS node> <Tree cycle> Transition from PLUS to cycle <PLUS node>

Search in Time

70

SLIDE 71

All GCC Transitions

71 # Location Cause transition to variable Start argv[3] 1 toplev.c:4755 name 2 toplev.c:2909 dump base name 3 c-lex.c:187 finput→ IO buf base 4 c-lex.c:1213 nextchar 5 c-lex.c:1213 yyssa[41] 6 c-typeck.c:3615 yyssa[42] 7 c-lex.c:1213 last insn→fld[1].rtx →fld[1].rtx→fld[3].rtx →fld[1].rtx.code 8 c-decl.c:1213 sequence result[2] →fld[0].rtvec →elem[0].rtx→fld[1].rtx →fld[1].rtx→fld[1].rtx →fld[1].rtx→fld[1].rtx →fld[1].rtx→fld[1].rtx →fld[3].rtx→fld[1].rtx.code 9 combine.c:4271 x→fld[0].rtx→fld[0].rtx

SLIDE 72 72

combine.c

if (GET_CODE (XEXP (x, 0)) == PLUS { x = apply_distributive_law (gen_binary (PLUS, mode, gen_binary (MULT, mode, XEXP (XEXP (x, 0), 0), XEXP (x, 1)), gen_binary (MULT, mode, XEXP (XEXP (x, 0), 1), XEXP (x, 1)))); if (GET_CODE (x) != MULT) return x; }

Should be copy_rtx() 1 line out of 1.047.562

SLIDE 73 73

Implementations C Java Python

Web service + command line Eclipse plug-in Module 24 months 12 months 2 days

SLIDE 74 74

SLIDE 75 75

Code Runs Static Dynamic Future Past Prevention Cure Errors Causes

SLIDE 76 76

Code + Runs Static + Dynamic Future + Past Prevention + Cure Errors + Causes

SLIDE 77 77