Icing Supporting Fast-Math Style Optimizations in a Verified - - PowerPoint PPT Presentation

Icing Supporting Fast-Math Style Optimizations in a Verified Compiler

Heiko Becker, Eva Darulova, Magnus Myreen, Zachary Tatlock

readability over performance

2

How we develop programs

readability over performance

2

How we develop programs

compiler should make program fast

readability over performance

2

How we develop programs

compiler should make program fast Compiler optimizations are a vital part of the development process

3

The need for understandable optimizations

3

The need for understandable optimizations

3

The need for understandable optimizations

4

The long road of compiler verification

1967: McCarthy, Painter; Correctness of a Compiler for Arithmetic Expressions (Integer’s only) 2009: Leroy; Formal Verification of a Realistic Compiler 2019: Lööw et al; Verified Compilation on a Verified Processor

4

The long road of compiler verification

1967: McCarthy, Painter; Correctness of a Compiler for Arithmetic Expressions (Integer’s only) 2009: Leroy; Formal Verification of a Realistic Compiler 2019: Lööw et al; Verified Compilation on a Verified Processor CompCert C compiler

4

The long road of compiler verification

1967: McCarthy, Painter; Correctness of a Compiler for Arithmetic Expressions (Integer’s only) 2009: Leroy; Formal Verification of a Realistic Compiler 2019: Lööw et al; Verified Compilation on a Verified Processor CompCert C compiler CakeML & Silver

4

The long road of compiler verification

1967: McCarthy, Painter; Correctness of a Compiler for Arithmetic Expressions (Integer’s only) 2009: Leroy; Formal Verification of a Realistic Compiler 2019: Lööw et al; Verified Compilation on a Verified Processor Where are floating-points?

4

The long road of compiler verification

1967: McCarthy, Painter; Correctness of a Compiler for Arithmetic Expressions (Integer’s only) 2009: Leroy; Formal Verification of a Realistic Compiler 2019: Lööw et al; Verified Compilation on a Verified Processor 2015: Boldo et al.; Verified Compilation of Floating-Point Programs

Unverified Compilers (gcc, clang, ….) Verified Compilers (CakeML, ...)

5

The state-of-the-art for fast-math

Unverified Compilers (gcc, clang, ….)

• apply aggressive optimizations

Verified Compilers (CakeML, ...)

5

The state-of-the-art for fast-math

Unverified Compilers (gcc, clang, ….)

• apply aggressive optimizations
• do not preserve IEEE754 semantics

Verified Compilers (CakeML, ...)

5

The state-of-the-art for fast-math

Unverified Compilers (gcc, clang, ….)

• apply aggressive optimizations
• do not preserve IEEE754 semantics
• give no guarantees on the result

Verified Compilers (CakeML, ...)

5

The state-of-the-art for fast-math

Unverified Compilers (gcc, clang, ….)

• apply aggressive optimizations
• do not preserve IEEE754 semantics
• give no guarantees on the result

Verified Compilers (CakeML, ...)

• apply no floating-point optimizations

5

The state-of-the-art for fast-math

Unverified Compilers (gcc, clang, ….)

• apply aggressive optimizations
• do not preserve IEEE754 semantics
• give no guarantees on the result

Verified Compilers (CakeML, ...)

• apply no floating-point optimizations
• fully preserve IEEE754 semantics

5

The state-of-the-art for fast-math

Unverified Compilers (gcc, clang, ….)

• apply aggressive optimizations
• do not preserve IEEE754 semantics
• give no guarantees on the result

Verified Compilers (CakeML, ...)

• apply no floating-point optimizations
• fully preserve IEEE754 semantics
• guarantee preserving literal meaning

5

The state-of-the-art for fast-math

Unverified Compilers (gcc, clang, ….)

• apply aggressive optimizations
• do not preserve IEEE754 semantics
• give no guarantees on the result

Verified Compilers (CakeML, ...)

• apply no floating-point optimizations
• fully preserve IEEE754 semantics
• guarantee preserving literal meaning

5

The state-of-the-art for fast-math

We need a semantics to handle fast-math optimizations in verified compilers

6

Contributions

Icing, a nondeterministic semantics for floating-points:

• Support for subset of gcc’s fast-math optimizations
• Optimization with fine-grained control
• Implementation of three optimizers
• Verification in HOL4
• Connection to CakeML

Example Optimizations:

7

Optimizations in Icing

Source Target

s t

𝑏 + 𝑐 𝑐 + 𝑏 𝑏 + 𝑐 + 𝑑 𝑏 + (𝑐 + 𝑑) 𝑏 × 𝑐 × 𝑑 𝑏 × (𝑐 × 𝑑) 𝐺𝑁𝐵(𝑏, 𝑐, 𝑑) 𝑏 × 𝑐 + 𝑑 𝑏 × 𝑐 𝑐 × 𝑏

Example Optimizations:

7

Optimizations in Icing

Source Target

s t

𝑏 + 𝑐 𝑐 + 𝑏 𝑏 + 𝑐 + 𝑑 𝑏 + (𝑐 + 𝑑) 𝑏 × 𝑐 × 𝑑 𝑏 × (𝑐 × 𝑑) 𝐺𝑁𝐵(𝑏, 𝑐, 𝑑) 𝑏 × 𝑐 + 𝑑 𝑏 × 𝑐 𝑐 × 𝑏

Commutativity (preserves IEEE754)

Example Optimizations:

7

Optimizations in Icing

Source Target

s t

𝑏 + 𝑐 𝑐 + 𝑏 𝑏 + 𝑐 + 𝑑 𝑏 + (𝑐 + 𝑑) 𝑏 × 𝑐 × 𝑑 𝑏 × (𝑐 × 𝑑) 𝐺𝑁𝐵(𝑏, 𝑐, 𝑑) 𝑏 × 𝑐 + 𝑑 𝑏 × 𝑐 𝑐 × 𝑏

Associativity (no IEEE754)

Example Optimizations:

7

Optimizations in Icing

Source Target

s t

𝑏 + 𝑐 𝑐 + 𝑏 𝑏 + 𝑐 + 𝑑 𝑏 + (𝑐 + 𝑑) 𝑏 × 𝑐 × 𝑑 𝑏 × (𝑐 × 𝑑) 𝐺𝑁𝐵(𝑏, 𝑐, 𝑑) 𝑏 × 𝑐 + 𝑑 𝑏 × 𝑐 𝑐 × 𝑏

FMA introduction (locally more accurate)

Example Optimizations:

7

Optimizations in Icing

Source Target

s t

𝑏 + 𝑐 𝑐 + 𝑏 𝑏 + 𝑐 + 𝑑 𝑏 + (𝑐 + 𝑑) 𝑏 × 𝑐 × 𝑑 𝑏 × (𝑐 × 𝑑) 𝐺𝑁𝐵(𝑏, 𝑐, 𝑑) 𝑏 × 𝑐 + 𝑑 𝑏 × 𝑐 𝑐 × 𝑏

IEEE754: 3.5 + 2.0 = 5.5 Icing:

8

Floating-Point Values in Icing

IEEE754: 3.5 + 2.0 = 5.5 Icing:

8

Floating-Point Values in Icing

floating-point word

IEEE754: 3.5 + 2.0 = 5.5 Icing: 3.5 + 2.0 =

8

Floating-Point Values in Icing

floating-point word

IEEE754: 3.5 + 2.0 = 5.5 Icing: 3.5 + 2.0 =

8

Floating-Point Values in Icing

+ 2.0 3.5

floating-point word

IEEE754: 3.5 + 2.0 = 5.5 Icing: 3.5 + 2.0 =

8

Floating-Point Values in Icing

+ 2.0 3.5

floating-point word value tree for addition

IEEE754: 3.5 + 2.0 = 5.5 Icing: 3.5 + 2.0 =

8

Floating-Point Values in Icing

+ 2.0 3.5

floating-point word value tree for addition

3.5 + (2.0 + 1.5) = 12.25

IEEE754: 3.5 + 2.0 = 5.5 Icing: 3.5 + 2.0 =

8

Floating-Point Values in Icing

+ 2.0 3.5

floating-point word value tree for addition

3.5 + (2.0 + 1.5) = 12.25 3.5 + (2.0 + 1.5) =

IEEE754: 3.5 + 2.0 = 5.5 Icing: 3.5 + 2.0 =

8

Floating-Point Values in Icing

+ 2.0 3.5

floating-point word value tree for addition

3.5 + (2.0 + 1.5) = 12.25 3.5 + (2.0 + 1.5) = + 1.5 2.0 + 3.5

IEEE754: 3.5 + 2.0 = 5.5 Icing: 3.5 + 2.0 =

8

Floating-Point Values in Icing

+ 2.0 3.5

floating-point word value tree for addition

3.5 + (2.0 + 1.5) = 12.25 3.5 + (2.0 + 1.5) = + 1.5 2.0 + 3.5

9

Icing’s semantics

• pt:(x * 2.4 + y)

Allowed Optimization: 𝐺𝑁𝐵(𝑏, 𝑐, 𝑑) 𝑏 × 𝑐 + 𝑑 𝑏 × 𝑐 𝑐 × 𝑏

9

Icing’s semantics

• pt:(x * 2.4 + y)

Allowed Optimization: 𝐺𝑁𝐵(𝑏, 𝑐, 𝑑) 𝑏 × 𝑐 + 𝑑 𝑏 × 𝑐 𝑐 × 𝑏 Included in the semantics

9

Icing’s semantics

fine-grained control

• pt:(x * 2.4 + y)

Allowed Optimization: 𝐺𝑁𝐵(𝑏, 𝑐, 𝑑) 𝑏 × 𝑐 + 𝑑 𝑏 × 𝑐 𝑐 × 𝑏 Included in the semantics

9

Icing’s semantics

fine-grained control

• pt:(x * 2.4 + y)

Allowed Optimization: 𝐺𝑁𝐵(𝑏, 𝑐, 𝑑) 𝑏 × 𝑐 + 𝑑 𝑏 × 𝑐 𝑐 × 𝑏 Included in the semantics × 𝑦 + 𝑧 2.4 × 2.4 + 𝑧 𝑦 𝑦 2.4 F𝑁𝐵 𝑧

9

Icing’s semantics

fine-grained control

• pt:(x * 2.4 + y)

Allowed Optimization: 𝐺𝑁𝐵(𝑏, 𝑐, 𝑑) 𝑏 × 𝑐 + 𝑑 𝑏 × 𝑐 𝑐 × 𝑏 Included in the semantics × 𝑦 + 𝑧 2.4 × 2.4 + 𝑧 𝑦 𝑦 2.4 F𝑁𝐵 𝑧 × 𝑦 + 𝑧 2.4

9

Icing’s semantics

fine-grained control

• pt:(x * 2.4 + y)

Allowed Optimization: 𝐺𝑁𝐵(𝑏, 𝑐, 𝑑) 𝑏 × 𝑐 + 𝑑 𝑏 × 𝑐 𝑐 × 𝑏 Included in the semantics × 𝑦 + 𝑧 2.4 × 2.4 + 𝑧 𝑦 𝑦 2.4 F𝑁𝐵 𝑧 × 𝑦 + 𝑧 2.4 Icing: a direct fit for fast-math with fine-grained control and support for different optimizations

Icing provides: Unverified Compilers (gcc, clang, ….)

• aggressive optimizations
• no IEEE754 semantics
• no guarantees on the result

Verified Compilers (CakeML, ...)

• no floating-point optimizations
• IEEE754 semantics
• introduces no new behaviour

10

Modelling the state-of-the-art

IEEE754 Translator

greedy optimizer

Icing provides: Unverified Compilers (gcc, clang, ….)

• aggressive optimizations
• no IEEE754 semantics
• no guarantees on the result

Verified Compilers (CakeML, ...)

• no floating-point optimizations
• IEEE754 semantics
• introduces no new behaviour

10

Modelling the state-of-the-art

IEEE754 Translator

greedy optimizer



11

What can we prove about the optimizers

Greedy optimizer: IEEE754 translator: ⊢ if evaluating the greedily optimized program 𝑞 returns 𝑤 then 𝑤 is a possible result of evaluating 𝑞 with the optimizations of the greedy

• ptimizer

⊢ after running the IEEE754 translator

• n program 𝑞 no optimizations can be

applied by Icing semantics ⊢ after running the IEEE754 translator

• n program 𝑞 Icing semantics are

deterministic no matter which

• ptimizations are allowed

The IEEE754 translator preserves literal meaning (like CompCert/CakeML) The greedy optimizer applies optimizations with respect to Icing semantics

12

Distributivity in Icing

𝑏 × 𝑐 + 𝑏 × 𝑑 𝑏 × (𝑐 + 𝑑)

12

Distributivity in Icing

x * (y + z)

𝑏 × 𝑐 + 𝑏 × 𝑑 𝑏 × (𝑐 + 𝑑)

12

Distributivity in Icing

x * (y + z) x * y + x * z Compiler

𝑏 × 𝑐 + 𝑏 × 𝑑 𝑏 × (𝑐 + 𝑑)

12

Distributivity in Icing

x * (y + z) x1 * y + x2 * z x * y + x * z Compiler Semantics

𝑏 × 𝑐 + 𝑏 × 𝑑 𝑏 × (𝑐 + 𝑑)

12

Distributivity in Icing

x * (y + z) x1 * y + x2 * z x * y + x * z Compiler Semantics

𝑏 × 𝑐 + 𝑏 × 𝑑 𝑏 × (𝑐 + 𝑑)

What if these do not match?

12

Distributivity in Icing

x * (y + z) x‘ * (y + z) x1 * y + x2 * z x * y + x * z Compiler Semantics Semantics

𝑏 × 𝑐 + 𝑏 × 𝑑 𝑏 × (𝑐 + 𝑑)

What if these do not match?

12

Distributivity in Icing

x * (y + z) x‘ * (y + z) x1 * y + x2 * z x * y + x * z Compiler Semantics Semantics x’ rewrites into x1 and x2

𝑏 × 𝑐 + 𝑏 × 𝑑 𝑏 × (𝑐 + 𝑑)

What if these do not match?

12

Distributivity in Icing

x * (y + z) x‘ * (y + z) x1 * y + x2 * z x * y + x * z x1 * y + x2 * z Compiler Semantics Semantics Semantics x’ rewrites into x1 and x2

𝑏 × 𝑐 + 𝑏 × 𝑑 𝑏 × (𝑐 + 𝑑)

What if these do not match?

12

Distributivity in Icing

x * (y + z) x‘ * (y + z) x1 * y + x2 * z x * y + x * z x1 * y + x2 * z Compiler Semantics Semantics Semantics x’ rewrites into x1 and x2

𝑏 × 𝑐 + 𝑏 × 𝑑 𝑏 × (𝑐 + 𝑑)

What if these do not match? Conditionals: Tricky! (see paper)

13

Handling more of gcc’s rewrites

Official clang documentation

13

Handling more of gcc’s rewrites

Official clang documentation

13

Handling more of gcc’s rewrites

s t

Source Target Official clang documentation

gcc:

13

Handling more of gcc’s rewrites

s t

Source Target

isNaN (c) F

Official clang documentation

gcc:

13

Handling more of gcc’s rewrites

s t

Source Target

P

Precondition

isNaN (c) F

Precondition allows to check condition before applying a rewrite Official clang documentation

Icing: gcc:

13

Handling more of gcc’s rewrites

s t

Source Target

P

Precondition

isNaN (c) F c = c

Precondition allows to check condition before applying a rewrite Official clang documentation

Icing: gcc:

13

Handling more of gcc’s rewrites

s t

Source Target

P

Precondition

isNaN (c) F c = c

Precondition allows to check condition before applying a rewrite Official clang documentation

Only 𝑂𝑏𝑂 unequal to itself

14

How can the preconditions be checked

Roundoff Errors Global Range Bounds Exceptions Gappa [SAC ‘06] Daisy [TACAS ’18] Verasco [POPL ‘15] FPTaylor [TOPLAS March 19] SMT-solvers (Z3 [TACAS ‘08], …)

15

Icings interface to external tools

Record assumed proposition Discharge checks in-place simple local check ⟹ checked before applying optimization complex global property ⟹ checked offline after compiling 𝑏 × ( 𝑐 + 𝑑) 𝑏 × 𝑐 + 𝑏 × 𝑑 𝑏, 𝑐, 𝑑 variables 𝑑 = 𝑑 𝑗𝑡𝑂𝑏𝑂(𝑑) 𝐺𝑏𝑚𝑡𝑓

16

Outlook:

• integrate with external tools
• verify larger optimizations
• integrate into CakeML semantics

Nondeterministic Icing (with optimizations) deterministic Icing (without optimizations) CakeML source

What does gcc’s fast-math actually do?