Deep Learning of Optimization Heuristics - - PowerPoint PPT Presentation

▶

Jun 19, 2023 104 likes •452 views

End-to-end Deep Learning of Optimization Heuristics http://chriscummins.cc/pact17 Chris Cummins University of Edinburgh Pavlos Petoumenos University of Edinburgh Zheng Wang Lancaster University Hugh Leather University of Edinburgh

SLIDE 1

Optimization Heuristics

Deep Learning

http://chriscummins.cc/pact17

End-to-end

SLIDE 2

Chris Cummins

Lancaster University University of Edinburgh University of Edinburgh University of Edinburgh

Pavlos Petoumenos Zheng Wang Hugh Leather

SLIDE 3

compilers are very complex

h a n d

e d h e u r i s t i c s

f choices

hundreds,

thousands,

millions}

{

int main( int argc, char** arg) {... _main: .cfi_start proc ## BB#0: pushq %rbp ... (

d a t e b y t i m e

r e l e a s e )

SLIDE 4

Machine learning in compilers

y = f(x)

ptimization

decision model features

( d e r i v e d f r

I R )

SLIDE 5

Machine learning in compilers

Training Programs Driver Feature Extractor Feature Vectors Best Decisions Training Data Training Data Training Data Optimization Heuristic

SLIDE 6

Machine learning in compilers

Training Programs Driver Feature Extractor Feature Vectors Best Decisions Training Data Training Data Training Data Optimization Heuristic

the human bit!

1 . h a r d t

e t r i g h t

2. time consuming
3. repetitious

SLIDE 7 Use a GPU Use a CPU Learned Heuristic

Feature space

Feature “Y”

Feature “X”

SLIDE 8 Use a GPU Use a CPU Learned Heuristic

Feature space

Feature “Y”

Feature “X”

need good features!

SLIDE 9

irrelevant

e.g. not capturing the right information e.g. missing critical information

incomplete

Ways to fail

unsuitable

e.g. wrong combination of features / model

SLIDE 10

What we have

Training Programs Driver Feature Extractor Feature Vectors Best Decisions Training Data Training Data Training Data Predictive Model

SLIDE 11

Training Programs Driver Best Decisions Training Data Training Data Training Data Predictive Model

What we need

SLIDE 12

Heuristics without features Beats expert approach Learning across heuristics

Contributions

SLIDE 13

int main(int argc, char **argv) { ...

Our approach

Deep Learning Optimization Decision Program Code

SLIDE 14

int main(int argc, char **argv) { ...

Our approach

Deep Learning Optimization Decision Program Code

{

p r e p r

e s s i n g

Rewriter Encoder Code in

encode as sequence of vocabulary indices

Vocabulary table for characters + lang keywords

normalize identifiers & code style

1.var/fun names: ‘foo’ , ‘bar’ , … to ‘a’ , ‘b’ , … 2.sanitize whitespace 3.consistent use of optional braces

SLIDE 15

Our approach

Deep Learning Optimization Decision Program Code

Rewriter Encoder Embedding Heuristic Model Language Model Rewriter Encoder Code in

m a p v

a b i n d i c e s i n t

e a l s p a c e summarize sequence as vector

(2 layer LSTM network)

predict optimization

n vector

(2 layer DNN)

SLIDE 16

Our approach

Deep Learning Optimization Decision Program Code

Embedding Heuristic Model Language Model Rewriter Encoder Code in

SLIDE 17

How does it work?

SLIDE 18

SLIDE 19

How does it work?

w e l l

SLIDE 20

Heterogeneous Mapping Thread Coarsening

Prior Art

CGO’13 Grewe et. al PACT’14 Magni et. al

SLIDE 21

Heterogeneous Mapping Thread Coarsening

Prior Art

Binary classification

{CPU, GPU}

One-of-six classification

{1, 2, 4, 8, 16, 32} CGO’13 PACT’14

Decision Space Model

Decision Tree

Cascading

Neural Networks

SLIDE 22

Heterogeneous Mapping Thread Coarsening

Prior Art

4 features

Combined from 7 raw values. Instruction counts / ratios.

7 features

Principle Components of 34 raw values. Instruction counts / ratios / relative deltas.

CGO’13 PACT’14

Features

2 p a p e r s !

SLIDE 23

Heterogeneous Mapping Thread Coarsening

int main(int argc ... int main(int argc ...

Our Approach

1. Use the same model design for both
2. No tweaking of parameters
3. Minimum change - 3 line diff

SLIDE 24

Heterogeneous Mapping Thread Coarsening

Prior Art

2x CPU-GPU architectures 4x GPU architectures

CGO’13 PACT’14

Hardware Training Programs

7 Benchmark Suites 3 Benchmark Suites

SLIDE 25

results

SLIDE 26

14% and 5% improvements over state-of-the-art

Speedup Heterogeneous Mapping

2.38x 2.09x

Speedup Thread Coarsening

1.06x 1.01x

State-of-the-art DeepTune

w. Transfer Learning

SLIDE 27

14% and 5% improvements over state-of-the-art

Speedup Heterogeneous Mapping

2.38x 2.09x

Speedup Thread Coarsening

1.06x 1.01x

State-of-the-art DeepTune

w. Transfer Learning

2 5 6 b e n c h m a r k s 1 7 b e n c h m a r k s

SLIDE 28

Heterogeneous Mapping Thread Coarsening

Transfer Learning

Embed- ding Heuristic Model Language Model Embed- ding Heuristic Model Language Model general specialized

SLIDE 29

Heterogeneous Mapping Thread Coarsening

Transfer Learning

Embed- ding Heuristic Model Language Model Embed- ding Heuristic Model Language Model general specialized

initialize with values

SLIDE 30

14% and 5% improvements over state-of-the-art

Speedup Heterogeneous Mapping

2.38x 2.09x

Speedup Thread Coarsening

1.06x 1.01x

State-of-the-art DeepTune

w. Transfer Learning

SLIDE 31

Speedup Heterogeneous Mapping

2.38x 2.09x

Speedup Thread Coarsening

1.12x 1.06x 1.01x

State-of-the-art DeepTune

w. Transfer Learning

14% and 11% improvements over state-of-the-art

SLIDE 32

Try it for yourself!

http://chriscummins.cc/pact17

code and data on GitHub r u n s i n t h e b r

s e r

s i s t e n t * C

p l e t e * W e l l D

u m e n t e d * E a s y t

e u s e * * E v a l u a t e d * A C T * A r t i f a c t * A E C P

SLIDE 33

Problem: feature design is hard Featureless heuristics First cross-domain learning 11-14% speedups

Deep Learning Optimisation Heuristics

End-to-end

http://chriscummins.cc/pact17