Towards Numerical Assistants Trust, Measurement, Community, and - - PowerPoint PPT Presentation

▶
towards numerical assistants
SMART_READER_LITE
LIVE PREVIEW

Towards Numerical Assistants Trust, Measurement, Community, and - - PowerPoint PPT Presentation

Feb 02, 2024 257 likes •889 views

Towards Numerical Assistants Trust, Measurement, Community, and Generality for the Numerical Workbench Pavel Panchekha Zachary Tatlock pavpan@cs.utah.edu ztatlock@cs.washington.edu University of Utah University of Washington Writing

slide-1
SLIDE 1

Towards Numerical Assistants

Trust, Measurement, Community, and Generality for the Numerical Workbench

Pavel Panchekha

pavpan@cs.utah.edu University of Utah

Zachary Tatlock

ztatlock@cs.washington.edu University of Washington

slide-2
SLIDE 2

Writing Numerical Code

Diabolical errors

  • Silent, non-compositional, non-local

Subtle tradeoffs

  • Performance, accuracy, determinism

Evolving landscape

  • bfloat, posits, Flexpoint, AdaptivFloat, ...
slide-3
SLIDE 3

Writing Numerical Code

Diabolical errors

  • Silent, non-compositional, non-local

Subtle tradeoffs

  • Performance, accuracy, determinism

Evolving landscape

  • bfloat, posits, Flexpoint, AdaptivFloat, ...
slide-4
SLIDE 4

Ideal Numerical Coding

  • 1. Look up the formula you need from Wikipedia.
  • 2. Type formula into your program.
  • 3. Get accurate results.
slide-5
SLIDE 5

Ideal Numerical Coding

  • 1. Look up the formula you need from Wikipedia.
  • 2. Type formula into your program.
  • 3. Get accurate results.

https://xkcd.com/217/

slide-6
SLIDE 6

Ideal Numerical Coding

  • 1. Look up the formula you need from Wikipedia.
  • 2. Type formula into your program.
  • 3. Get accurate results.

Reality.

slide-7
SLIDE 7

Numerical Code in Practice

Blake Courter CAD Researcher

slide-8
SLIDE 8

Numerical Code in Practice

Blake Courter CAD Researcher

slide-9
SLIDE 9

Vision: Numerical Assistants

Blake Courter CAD Researcher

slide-10
SLIDE 10

Vision: Numerical Assistants

Blake Courter CAD Researcher 2015

slide-11
SLIDE 11

Vision: Numerical Assistants

slide-12
SLIDE 12

Vision: Numerical Assistants

FPBench Titanic

slide-13
SLIDE 13

Vision: Numerical Assistants

Developing since 2015:

  • 1. Trust
  • 2. Measurement
  • 3. Community
  • 4. Generality

FPBench Titanic

slide-14
SLIDE 14

Vision: Numerical Assistants

FPBench Titanic

Developing since 2015:

  • 1. Trust
  • 2. Measurement
  • 3. Community
  • 4. Generality

First, quick background

slide-15
SLIDE 15

Herbie

slide-16
SLIDE 16

Herbie

Given input expression

  • ver the reals
slide-17
SLIDE 17

Herbie

Given input expression

  • ver the reals

Synthesize a floating point implementation

slide-18
SLIDE 18

Herbie

Given input expression

  • ver the reals

Synthesize a floating point implementation Less error than naive translation

slide-19
SLIDE 19

Herbie

“The drunk numerical analyst”

slide-20
SLIDE 20

Herbie

“The drunk numerical analyst” Live demo: herbie.uwplse.org

slide-21
SLIDE 21

FPBench

Benchmarks

  • Mine papers, enable apples-to-apples

Compilers

  • Generate C, JavaScript, Scala, TeX, ...

Standards

  • Support tool interchange, composition
slide-22
SLIDE 22

FPBench

Benchmarks

  • Mine papers, enable apples-to-apples

Compilers

  • Generate C, JavaScript, Scala, TeX, ...

Standards

  • Support tool interchange, composition

Live tour: fpbench.org

slide-23
SLIDE 23
  • 1. Trust

Why should users trust Herbie?

slide-24
SLIDE 24
  • 1. Trust

Compilers forbidden from touching floating point at all … and they even get that wrong!

Why should users trust Herbie?

slide-25
SLIDE 25
  • 1. Trust

Compilers forbidden from touching floating point at all … and they even get that wrong! No really, why should users trust Herbie?

Why should users trust Herbie?

slide-26
SLIDE 26
  • 1. Trust

Why should users trust Herbie?

slide-27
SLIDE 27
  • 1. Trust

Why should users trust Herbie?

Sample inputs, estimate error

slide-28
SLIDE 28
  • 1. Trust

Why should users trust Herbie?

Sample inputs, estimate error Heuristically search for more accurate expressions

slide-29
SLIDE 29
  • 1. Trust

Why should users trust Herbie?

Sample inputs, estimate error Heuristically search for more accurate expressions Select best exprs, infer branches

slide-30
SLIDE 30
  • 1. Trust

Why should users trust Herbie?

Sample inputs, estimate error Heuristically search for more accurate expressions Select best exprs, infer branches

Unpredictable, mysterious!

slide-31
SLIDE 31
  • 1. Trust

Why should users trust Herbie?

Test set, not training set

slide-32
SLIDE 32
  • 1. Trust

Why should users trust Herbie?

slide-33
SLIDE 33
  • 1. Trust

Why should users trust Herbie?

Identify where error is arising

slide-34
SLIDE 34
  • 1. Trust

Why should users trust Herbie?

Identify where error is arising Show stepwise derivation

slide-35
SLIDE 35
  • 1. Trust

Why should users trust Herbie?

Identify where error is arising Show stepwise derivation Justify by accuracy impact

slide-36
SLIDE 36
  • 2. Measurement

That which is measured, improves.

slide-37
SLIDE 37
  • 2. Measurement

Herbie is a complex pipeline. Each component individually robust.

  • Actually works against debugging!!

Need to have per-module specs and metrics.

That which is measured, improves.

slide-38
SLIDE 38
  • 2. Measurement

That which is measured, improves.

No “just enough” level of detail is right for logs...

slide-39
SLIDE 39
  • 2. Measurement

That which is measured, improves.

slide-40
SLIDE 40
  • 2. Measurement

That which is measured, improves.

Build generic telemetry! Iteratively develop viz! 100s bugs fixed, 1000x speedups!

slide-41
SLIDE 41
  • 2. Measurement

That which is measured, improves.

Build generic telemetry! Iteratively develop viz! 100s bugs fixed, 1000x speedups!

Improves Trust!

slide-42
SLIDE 42
  • 2. Measurement

That which is measured, improves.

Global overviews

slide-43
SLIDE 43
  • 2. Measurement

That which is measured, improves.

Global overviews Per component logs

slide-44
SLIDE 44
  • 2. Measurement

That which is measured, improves.

Global overviews Per component logs Compare to oracles where possible!

slide-45
SLIDE 45
  • 3. Community

We are all in this together.

slide-46
SLIDE 46
  • 3. Community

Herbie is just one piece of the puzzle. The workbench demands cooperating tools:

  • Rigorous error bounds, precision

tuning, performance optimization, … How should they interact? Need standards!

We are all in this together.

Our thinking in 2016

slide-47
SLIDE 47
  • 3. Community

Herbie is just one piece of the puzzle. The workbench demands cooperating tools:

  • Rigorous error bounds, precision

tuning, performance optimization, … How should they interact? Need standards!

We are all in this together.

Our thinking in 2016 Numerics research is a community effort. We need more open communications! We need more collaboration opportunities! We need more shared infrastructure! View today

slide-48
SLIDE 48
  • 3. Community

We are all in this together.

slide-49
SLIDE 49
  • 3. Community

We are all in this together.

slide-50
SLIDE 50
  • 3. Community

We are all in this together.

slide-51
SLIDE 51
  • 3. Community

We are all in this together.

slide-52
SLIDE 52
  • 3. Community

We are all in this together.

Link to join on fpbench.org

slide-53
SLIDE 53
  • 4. Generality

Supporting a rapidly evolving landscape

slide-54
SLIDE 54
  • 4. Generality

Explosion of numerical representations. How do we adapt code? Optimize? Need tools!

Supporting a rapidly evolving landscape

slide-55
SLIDE 55
  • 4. Generality

Explosion of numerical representations. How do we adapt code? Optimize? Need tools!

Supporting a rapidly evolving landscape

“Precision scopes” specify MPMF!

slide-56
SLIDE 56
  • 4. Generality

Supporting a rapidly evolving landscape

Enables parameter sweeps: iters, cost, accuracy

slide-57
SLIDE 57
  • 4. Generality

Supporting a rapidly evolving landscape

Increasing interest in numerics of tensor code.

slide-58
SLIDE 58
  • 4. Generality

Supporting a rapidly evolving landscape

Increasing interest in numerics of tensor code.

(FPCore matmul ((A am an) (B bm bn)) :pre (== an bm) (tensor ([m am] [n bn]) (for ([i bm]) ([prod 0 (+ prod (* (ref A m i) (ref B i n)))]) prod)) ) FPBench 2.0 adds tensor support, keeps minimal core

slide-59
SLIDE 59

Where do we go from here?

Trust : ground truth for MPMF Measurement : metrics for tensor benchmarks Community : outreach from research to practice Generality : extensibility for new formats

slide-60
SLIDE 60

A brief announcement... Herbie 1.4 just released! Please take it for a spin :)

herbie.uwplse.org

slide-61
SLIDE 61

Many Acknowledgements!

Alex Sanchez-Stern Bill Zorn David Thien Oliver Flatt Brett Saiki Jason Qiu Ian Briggs Heiko Becker Eva Darulova Max Willsey James Wilcox Jack Firth Mike Lam … and more!

slide-62
SLIDE 62

Thank You!

Trust : ground truth for MPMF Measurement : metrics for tensor benchmarks Community : outreach from research to practice Generality : extensibility for new formats

herbie.uwplse.org fpbench.org