[PPT] - Pavel Alex James Zach Panchekha Sanchez-Stern Wilcox Tatlock PowerPoint Presentation

SLIDE 1

Automatically Improving Accuracy for Floating Point Expressions

Pavel Panchekha Alex Sanchez-Stern James Wilcox Zach Tatlock

⇒

error

SLIDE 2

Floating Point’s Wild Success

SLIDE 3

Floating Point’s Wild Success

F ≈ R

But not always! Often floating point is close to real arithmetic

SLIDE 4

Floating Point’s Wild Success

But not always!

Numerous articles retracted [Altman ’99, ’03] Financial regulations [Euro ’98] Market distortions [McCullough ’99, Quinn ’83]

SLIDE 5

Rounding Error in Sculpture

Blake Courter @bcourter

SLIDE 6

Rounding Error in Sculpture

Rounding error

⇒

Blake Courter @bcourter

SLIDE 7

Rounding Error in Sculpture

Rounding error

⇒

Blake Courter @bcourter

SLIDE 8

Existing options

Big Float

Unreliable

+ Fast Code + More Reliable

Slow Code

+ Reliable + Fast Code

Expert Task

SLIDE 9

Heuristic search to find expert transformations

e’ e

SLIDE 10

Heuristic search to find expert transformations

Worked Example How Herbie Works Evaluation

e’ e

SLIDE 11

Heuristic search to find expert transformations

Worked Example How Herbie Works Evaluation

eF eR

e’ e

SLIDE 12

Rounding Error in Quadratic

−b + √ b2 − 4ac 2a

error

64

SLIDE 13

Rounding Error in Quadratic

−b + √ b2 − 4ac 2a

64

What is rounding error?

              

7 ULPs

error JeKR exact JeKF computed

SLIDE 14

Rounding Error in Quadratic

−b + √ b2 − 4ac 2a

64

What is rounding error?

              

7 ULPs log(ULPs) estimates # of incorrect bits

log(ULPs) JeKR exact JeKF computed

SLIDE 15

Rounding Error in Quadratic

b log(ULPs)

−b + √ b2 − 4ac 2a

64

SLIDE 16

                

c b − b a

if b ∈ A

−b+ √ b2−4ac 2a

if b ∈ B

2c −b− √ b2−4ac

if b ∈ C − c

b

if b ∈ D

Rounding Error in Quadratic

b log(ULPs) A B C D

⇒

−b + √ b2 − 4ac 2a

64

SLIDE 17

                

c b − b a

if b ∈ A

−b+ √ b2−4ac 2a

if b ∈ B

2c −b− √ b2−4ac

if b ∈ C − c

b

if b ∈ D

Rounding Error in Quadratic

b log(ULPs) A

⇒

−b + √ b2 − 4ac 2a

B C D

64

Overflow If is large, overflows and the the whole expression returns .

Jb2KF ∞

b A

SLIDE 18

                

c b − b a

if b ∈ A

−b+ √ b2−4ac 2a

if b ∈ B

2c −b− √ b2−4ac

if b ∈ C − c

b

if b ∈ D

Rounding Error in Quadratic

b log(ULPs)

⇒

−b + √ b2 − 4ac 2a

Pretty Accurate

64

A B C D A B

SLIDE 19

A B C D                 

c b − b a

if b ∈ A

−b+ √ b2−4ac 2a

if b ∈ B

2c −b− √ b2−4ac

if b ∈ C − c

b

if b ∈ D

Rounding Error in Quadratic

b log(ULPs)

⇒

−b + √ b2 − 4ac 2a

Catastrophic Cancellation If is large, but and are small, and the difference is rounded off.

b

a c

b ≈ p b2 − 4ac

64

A B C

SLIDE 20

                

c b − b a

if b ∈ A

−b+ √ b2−4ac 2a

if b ∈ B

2c −b− √ b2−4ac

if b ∈ C − c

b

if b ∈ D D

Rounding Error in Quadratic

b log(ULPs)

⇒

−b + √ b2 − 4ac 2a

Overflow again

64

A B C D A B C

SLIDE 21

Rounding Error in Quadratic

b

⇒

b

−b + √ b2 − 4ac 2a

log(ULPs)

⇒

A B C D                 

c b − b a

if b ∈ A

−b+ √ b2−4ac 2a

if b ∈ B

2c −b− √ b2−4ac

if b ∈ C − c

b

if b ∈ D

64 64

A B C D

SLIDE 22

Heuristic search to find expert transformations

Worked Example How Herbie Works Evaluation

eF eR

e’ e

SLIDE 23

Heuristic search to find expert transformations

Worked Example How Herbie Works Evaluation

eF eR

e’ e

SLIDE 24

Herbie Architecture

sample cands focus generate more candidates regimes

eF eR

e’ e

SLIDE 25

Herbie Architecture

sample cands focus regimes

Ground truth

eF eR

e’ e

generate more candidates

SLIDE 26

Herbie Architecture

sample cands focus regimes

Localize error

eF eR

e’ e

generate more candidates

SLIDE 27

Herbie Architecture

sample cands focus regimes

Heuristic search

eF eR

e’ e

generate more candidates

SLIDE 28

Herbie Architecture

sample cands focus regimes

Keep all good candidates

eF eR

e’ e

generate more candidates

SLIDE 29

Herbie Architecture

sample cands focus regimes

Combine candidates

eF eR

e’ e

generate more candidates

SLIDE 30

Herbie Architecture

sample cands focus regimes

Ground truth

eF eR

e’ e

generate more candidates

SLIDE 31

Determine ground truth

Compute in F

X = sample(domain(e))

e.g. X = {1.2 · 10−17, −3.8 · 10204, 173.5, . . . }

error e.g. {13.2b, 51.7b, 1b, . . . }

Get 64-bit prefix with MPFR. Subtle! See paper.

JeKF(X)

Round(JeKR(X))

64 random bits

SLIDE 32

Herbie Architecture

sample cands focus regimes

Ground truth

eF eR

e’ e

generate more candidates

SLIDE 33

Herbie Architecture

sample cands focus regimes

eF eR

e’ e

generate more candidates

Localize error

SLIDE 34

Cancellation Overflow

Focus: Estimate Error Source

+

p b2 − 4ac

−b

+F

+R

Round(x +R y)

(x +F y)

1. For each op in
2. Evaluate args in
3. Apply to them
4. Apply to them
5. Compare

R

fR

fF

−b + √ b2 − 4ac 2a

J KR

x =

J KR

y =

f

e

SLIDE 35

Herbie Architecture

sample cands focus regimes

eF eR

e’ e

generate more candidates

Localize error

SLIDE 36

Herbie Architecture

sample cands focus regimes

Heuristic search

eF eR

e’ e

generate more candidates

SLIDE 37

Herbie Architecture

sample cands focus rewrite regimes

Create candidates

series simplify

eF eR

e’ e

SLIDE 38

Apply rewrites to

−b + √ b2 − 4ac 2a

(−b)2 − ( √ b2 − 4ac)2 −b − √ b2 − 4ac ! /2a

Recursive rewrites:

Database of rules
Flexible
Chains of rewrites

(0 − b) + √ b2 − 4ac 2a

0 −

b −

√ b2 − 4ac

2a

… 120 more … Rule DB −x 0 − x

x + y x2 − y2 x − y

(x − y) + z x − (y − z)

No cancellation in denominator

SLIDE 39

Herbie Architecture

sample cands focus rewrite regimes

Create candidates

series simplify

eF eR

e’ e

SLIDE 40

Herbie Architecture

sample cands focus rewrite series simplify regimes

Approximate expr

eF eR

e’ e

SLIDE 41

Idea: near-identities

Series Expansions

−b + √ b2 − 4ac 2a

Bounded Laurent series:

Transcendental functions
Singularities
Number of terms to take

(for )

√ 1 − x ≈ 1 − x/2

x ≈ 0

−b + b(1 − 4ac/2b2) 2a

SLIDE 42

Herbie Architecture

sample cands focus rewrite series simplify regimes

Approximate expr

eF eR

e’ e

SLIDE 43

Herbie Architecture

sample cands focus rewrite series simplify regimes

Cancel & clean up

eF eR

e’ e

SLIDE 44

Simplify Expressions

(−b)2 − ( √ b2 − 4ac)2 −b − √ b2 − 4ac ! /2a

Difficult! [Caviness ’70]

many possible rewrites
huge search space
avoid undoing progress!

E-graphs [Nelson ’79]

Terminate early
Prune useless nodes
Restrict rewrites

= 2c −b − √ b2 − 4ac =

✓ 4ac −b − √ b2 − 4ac ◆ /2a

=

✓ b2 − (b2 − 4ac) −b − √ b2 − 4ac ◆ /2a

=

b2 − ( √ b2 − 4ac)2 −b − √ b2 − 4ac ! /2a

SLIDE 45

Herbie Architecture

sample cands focus rewrite series simplify regimes

Cancel & clean up

eF eR

e’ e

SLIDE 46

Herbie Architecture

sample cands focus regimes

Keep all good candidates

eF eR

e’ e

rewrite series simplify

SLIDE 47

Herbie Architecture

sample cands focus regimes

Combine candidates

eF eR

e’ e

rewrite series simplify

SLIDE 48

−b + √ b2 − 4ac 2a

−c b c b − b a

2c −b − √ b2 − 4ac

Regime Inference

SLIDE 49

Regime Inference

                

c b − b a

if b ∈ (∞, −1.15E122]

−b+ √ b2−4ac 2a

if b ∈ (−1.125E122, 1.06E−304]

2c −b− √ b2−4ac

if b ∈ (1.06E−304, 4.62E63] − c

b

if b ∈ (4.62E63, ∞)

Dynamic programming:

Bounds quickly
Tune: binary search
Pick best variable

A B C D

SLIDE 50

Herbie Architecture

sample cands focus regimes

Combine candidates

eF eR

e’ e

rewrite series simplify

SLIDE 51

Heuristic search to find expert transformations

Worked Example How Herbie Works Evaluation

eF eR

e’ e

SLIDE 52

Heuristic search to find expert transformations

Worked Example How Herbie Works Evaluation

eF eR

e’ e

SLIDE 53

Evaluating Herbie

A. Does accuracy improve?
B. Does it reproduce expert transformations?
C. Is the output code fast?
D. Does it work in the real world?

SLIDE 54

Examples from Hamming’s NMSE

Chapter 3: Function evaluation 28 worked examples & problems Quadratic formula (4) Algebraic rearrangement (12) Series expansion (12) Branches and regimes (2)

SLIDE 55

A. Improves accuracy in every test

Average bits correct (longer is better)

Accuracy of input Accuracy of output Dramatic improvement

SLIDE 56

B. Reproduces expert changes

Average bits correct (longer is better)

Handle overflow More accurate series expansion No trig factorization Only branch on var

Of 12 with answers: Same in 8 Different in 4

SLIDE 57

C. Output code is fast

Overhead CDF (left is better)

Median: 40%

SLIDE 58

D. Two MathJS Patches Accepted

SLIDE 59

D. Machine Learning Anecdote

I wasn't sure how to best rewrite [my]

equations. Herbie found numerically

stable versions of the formulas, and fixed all the divide-by-zero errors.

Harley Montgomery Clustering (bigger, darker blocks better)

⇒

SLIDE 60

Heuristic search to find expert transformations

Worked Example How Herbie Works Evaluation

eF eR

e’ e

SLIDE 61

Improve accuracy of floating point programs Sampling to estimate error Reduce global error to per-operation error Iterative rewriting highest-error operations Different expressions for different inputs

http: // herbie.uwplse.org /

eF eR

e’ e

SLIDE 62

SLIDE 63

Herbie and Maximum Error

Often improved by Herbie: Improvements large (28b) and small (.5b) 1+b improvement for 10/28 programs Fewer high-error pts, same max error.

Bits error (histogram)

SLIDE 64

Herbie as Part of a Pipeline

Find inaccurate expressions Improve accuracy Prove accuracy satisfactory Optimize code FPDebug Herbie Rosa FPTaylor STOKE-FP

SLIDE 65

Error graphs along and

a

c

a c

SLIDE 66

Finding the rewrite rules

Standard mathematical identities:

Commutativity, inverses, fractions, trig identities

No numerical methods knowledge Don’t need to be true identities

False rules do not improve accuracy Herbie will ignore them

SLIDE 67

Regimes often gains ~15 bits

Improvement from regimes (longer is better) Dot : input program average accurage Bar : Herbie result w/out regimes