against Platform Uncertainties Thomas Wahl Northeastern University - PowerPoint PPT Presentation

Stabilizing Numeric Programs against Platform Uncertainties Thomas Wahl Northeastern University August 28, 2017

Example: Ray Tracing int raySphere( float *r, float *s, float radiusSq) { float A = dot3(r,r); float B = -2.0 * dot3(s,r); float C = dot3(s,s) - radiusSq; float D = B*B - 4*A*C; if (D > 0.00001) { float sign = ( C > 0.00001 ) ? 1 : -1; ⋮ For the input on the right: AMD GPU A8-3850: 𝐸 = −0.000122 NVIDIA Quadro 600 GPU: 𝐸 = +0.000244

CPU: 9.0 + 0.9 + 0.09 + 0.009 + 0.0009 + 0.00009 + 0.000009 + 0.0000009 9.9 GPU: 9.0 + 0.9 + 0.09 + 0.009 + 0.0009 + 0.00009 + 0.000009 + 0.0000009 0.099 0.0000099 9.9 0.00099 9.999 0.0009999 9.9999 999

Platform Variations: Contracted Operations 𝑏 𝑐 𝑏 𝑐 Fused Multiply-Add (FMA) MULT MULT 𝑑 ROUND 𝑑 ADD ADD ROUND ROUND 𝑏 × 𝑐 ⊕ 𝑑 𝑏 ⊗ 𝑐 ⊕ 𝑑

VOLATILITY IN NUMERIC PROGRAMS

Volatile Expressions = expression whose semantics depends on the (expression) evaluation model, which determines: • evaluation order • availability and use of hardware features such as fused multiply-add (FMA) 𝑔 1 𝐽 + 𝑔 2 𝐽 + … + 𝑔 𝑜 (𝐽) Sums, products: Dot products: 𝑔 1 𝐽 × 𝑕 1 𝐽 + … + 𝑔 𝑜 𝐽 × 𝑕 𝑜 (𝐽)

Quantifying Volatility Let 𝑌 be a floating-point expression in the program. The volatile bound of 𝑌 for input 𝐽 is min 𝑁 𝑌 𝐽, 𝑁 , max 𝑁 𝑌 𝐽, 𝑁 𝑁 : expression evaluation models Questions:  How do we compute this, for input 𝐽 ?  Once computed, what is it good for?

Volatility in Matrix Calculations  well-designed programs  high degree of reproducibility

But: Ray Tracing int raySphere( float *r, float *s, float radiusSq) { float A = dot3(r,r); float B = -2.0 * dot3(s,r); float C = dot3(s,s) - radiusSq; float D = B*B - 4*A*C; if (D > 0.00001) { float sign = ( C > 0.00001 ) ? 1 : -1; ⋮ For some inputs: AMD GPU A8-3850: 𝐸 = −0.000122 NVIDIA Quadro 600 GPU: 𝐸 = +0.000244

Stabilizing Numeric Programs: Overview Goal: • Improve reproducibility of program results Naive solution: “ determinize ” the whole program cl /fp:strict source.cpp • implements strict evaluation: FMA disabled, expressions from left to right

Local Stabilization

Identifying Major Destabilizers int raySphere( float *r, float *s, float radiusSq) { float A = dot3(r,r); float B = -2.0 * dot3(s,r); float C = dot3(s,s) - radiusSq; float D = B*B - 4*A*C; if (D > 0.00001) { float sign = ( C > 0.00001 ) ? 1 : -1; ⋮

Stabilizing 𝐸 ’s expression int raySphere( float *r, float *s, float radiusSq) { float A = dot3(r,r); float B = -2.0 * dot3(s,r); float C = dot3(s,s) - radiusSq; /* don’t optimize ... */ float D = B*B - 4*A*C; if (D > 0.00001) { float sign = ( C > 0.00001 ) ? 1 : -1; ⋮ Stabilizing D = B*B – 4*A*C : new volatile bound for 𝐸 of [ −0.250000000, 0.125000000 ]

Stabilizing 𝐸 ’s expression not enough Two causes of large volatile bounds for 𝑬 : 1. volatility caused by 𝐸 ’s defining expression 2. volatility inherited from earlier expressions, which causes bloated inputs to 𝐸 !

Provenance of 𝐸 ’s Volatility = for each preceding expression (here: 𝐵, 𝐶, 𝐷 ), the contribution to (impact on) 𝐸 ’s volatility [VMCAI 2017]

int raySphere( float *r, float *s, float radiusSq) { float A = dot3(r,r); /* FMA forbidden! */ float B = -2.0 * dot3(s,r); /* don’t reorder! */ float C = dot3(s,s) - radiusSq; float D = B*B - 4*A*C; if (D > 0.00001) { float sign = ( C > 0.00001 ) ? 1 : -1; ⋮  Stabilizing B = 2.0 * dot3(s,r) : new bound for 𝐸 : [-0.002806663, 0.156753913]  Stabilizing C = dot3(s,s) - radiusSq new bound for 𝐸 : [0.125000000, 0.156753913]

Take-Home Messages  FPA expressions are volatile: semantics fragile against platform variations  expose volatility , or prove robustness , on a per-input basis  repair: make program robust against platform uncertainties Future plans:  platform uncertainties in machine learning (or other big-data)  prove robustness for a range of inputs  trade-offs: num. stability vs. accuracy ( ℝ !) vs. efficiency

Acknowledgements Joint with (@ NEU): • Yijia Gu (stud.), Computer and Information Science • Miriam Leeser (fac.) and Mahsa Bayati (stud.), Engineering http://www.ccs.neu.edu/home/wahl/Research/ /FPA-Heterogeneous/Non-Portability Financial Support: