against Platform Uncertainties Thomas Wahl Northeastern University - - PowerPoint PPT Presentation

Stabilizing Numeric Programs against Platform Uncertainties

Thomas Wahl Northeastern University August 28, 2017

Example: Ray Tracing

For the input on the right: AMD GPU A8-3850: 𝐸 = −0.000122 NVIDIA Quadro 600 GPU: 𝐸 = +0.000244

int raySphere(float *r, float *s, float radiusSq) { float A = dot3(r,r); float B = -2.0 * dot3(s,r); float C = dot3(s,s) - radiusSq; float D = B*B - 4*A*C; if (D > 0.00001) { float sign = ( C > 0.00001 ) ? 1 : -1; ⋮

9.0 + 0.9 + 0.09 + 0.009 + 0.0009 + 0.00009 + 0.000009 + 0.0000009

0.099 9.999

9.9999999

0.0009999

CPU: GPU:

9.9 9.9 0.00099

9.0 + 0.9 + 0.09 + 0.009 + 0.0009 + 0.00009 + 0.000009 + 0.0000009

0.0000099

Platform Variations: Contracted Operations

Fused Multiply-Add (FMA) MULT ADD ROUND ROUND ROUND MULT ADD

𝑏 𝑏 𝑐 𝑐 𝑑 𝑑

𝑏 ⊗ 𝑐 ⊕ 𝑑 𝑏 × 𝑐 ⊕ 𝑑

VOLATILITY IN NUMERIC PROGRAMS

Volatile Expressions

= expression whose semantics depends on the (expression) evaluation model, which determines: Sums, products: Dot products:

• evaluation order
• availability and use of hardware features

such as fused multiply-add (FMA) 𝑔

1 𝐽 + 𝑔 2 𝐽 + … + 𝑔 𝑜(𝐽)

𝑔

1 𝐽 × 𝑕1 𝐽 + … + 𝑔 𝑜 𝐽 × 𝑕𝑜(𝐽)

Quantifying Volatility

Questions:

• How do we compute this, for input 𝐽 ?
• Once computed, what is it good for?

Let 𝑌 be a floating-point expression in the program. The volatile bound of 𝑌 for input 𝐽 is min

𝑁 𝑌 𝐽, 𝑁

, max

𝑁 𝑌 𝐽, 𝑁

𝑁: expression evaluation models

Volatility in Matrix Calculations

well-designed programs high degree of reproducibility

But: Ray Tracing

For some inputs: AMD GPU A8-3850: 𝐸 = −0.000122 NVIDIA Quadro 600 GPU: 𝐸 = +0.000244

int raySphere(float *r, float *s, float radiusSq) { float A = dot3(r,r); float B = -2.0 * dot3(s,r); float C = dot3(s,s) - radiusSq; float D = B*B - 4*A*C; if (D > 0.00001) { float sign = ( C > 0.00001 ) ? 1 : -1; ⋮

Stabilizing Numeric Programs: Overview

Goal:

• Improve reproducibility of program results

Naive solution: “determinize” the whole program cl /fp:strict source.cpp

• implements strict evaluation:

FMA disabled, expressions from left to right

Local Stabilization

Identifying Major Destabilizers

int raySphere(float *r, float *s, float radiusSq) { float A = dot3(r,r); float B = -2.0 * dot3(s,r); float C = dot3(s,s) - radiusSq; float D = B*B - 4*A*C; if (D > 0.00001) { float sign = ( C > 0.00001 ) ? 1 : -1; ⋮

Stabilizing 𝐸’s expression

int raySphere(float *r, float *s, float radiusSq) { float A = dot3(r,r); float B = -2.0 * dot3(s,r); float C = dot3(s,s) - radiusSq; /* don’t optimize ... */ float D = B*B - 4*A*C; if (D > 0.00001) { float sign = ( C > 0.00001 ) ? 1 : -1; ⋮

Stabilizing D = B*B – 4*A*C : new volatile bound for 𝐸 of [−0.250000000, 0.125000000]

Stabilizing 𝐸’s expression not enough

Two causes of large volatile bounds for 𝑬:

• 1. volatility caused by 𝐸’s defining expression
• 2. volatility inherited from earlier expressions,

which causes bloated inputs to 𝐸 !

Provenance of 𝐸’s Volatility

= for each preceding expression (here: 𝐵, 𝐶, 𝐷), the contribution to (impact on) 𝐸’s volatility

[VMCAI 2017]

int raySphere(float *r, float *s, float radiusSq) { float A = dot3(r,r); /* FMA forbidden! */ float B = -2.0 * dot3(s,r); /* don’t reorder! */ float C = dot3(s,s) - radiusSq; float D = B*B - 4*A*C; if (D > 0.00001) { float sign = ( C > 0.00001 ) ? 1 : -1; ⋮

• Stabilizing B = 2.0 * dot3(s,r):

new bound for 𝐸 : [-0.002806663, 0.156753913]

• Stabilizing C = dot3(s,s) - radiusSq

new bound for 𝐸 : [0.125000000, 0.156753913]

 FPA expressions are volatile: semantics fragile against platform variations  expose volatility, or prove robustness, on a per-input basis  repair: make program robust against platform uncertainties

Take-Home Messages

Future plans:

• platform uncertainties in machine learning (or other big-data)
• prove robustness for a range of inputs
• trade-offs: num. stability vs. accuracy (ℝ!) vs. efficiency

Acknowledgements

Joint with (@ NEU):

• Yijia Gu (stud.), Computer and Information Science
• Miriam Leeser (fac.) and Mahsa Bayati (stud.), Engineering

http://www.ccs.neu.edu/home/wahl/Research/ /FPA-Heterogeneous/Non-Portability Financial Support: