Proving Equivalence Between Imperative and MapReduce Implementations - - PowerPoint PPT Presentation

▶
proving equivalence between imperative and mapreduce
SMART_READER_LITE
LIVE PREVIEW

Proving Equivalence Between Imperative and MapReduce Implementations - - PowerPoint PPT Presentation

Mar 06, 2024 350 likes •968 views

Proving Equivalence Between Imperative and MapReduce Implementations Using Program Transformations VPT 2018 Thessaloniki 20 April 2018 Bernhard Beckert, Timo Bingman, Moritz Kiefer, Peter Sanders, Mattias Ulbrich , Alexander Weigl www.kit.edu

slide-1
SLIDE 1

Proving Equivalence Between Imperative and MapReduce Implementations Using Program Transformations

VPT 2018 Thessaloniki 20 April 2018

Bernhard Beckert, Timo Bingman, Moritz Kiefer, Peter Sanders, Mattias Ulbrich, Alexander Weigl

KIT – Die Forschungsuniversit¨ at in der Helmholtz-Gemeinschaft

www.kit.edu

slide-2
SLIDE 2

Motivation

Simple algorithm implementations are understandable, . . .

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 2/27

slide-3
SLIDE 3

Motivation

Simple algorithm implementations are understandable, . . . . . . but not efficient.

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 2/27

slide-4
SLIDE 4

Motivation

Simple algorithm implementations are understandable, . . . . . . but not efficient. Efficient implementations are often complex, error-prone

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 2/27

slide-5
SLIDE 5

Motivation

Simple algorithm implementations are understandable, . . . . . . but not efficient. Efficient implementations are often complex, error-prone = ⇒ Prove equivalence between reference and efficient implementation

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 2/27

slide-6
SLIDE 6

Motivation

Simple algorithm implementations are understandable, . . . . . . but not efficient. Efficient implementations are often complex, error-prone = ⇒ Prove equivalence between reference and efficient implementation Verify MapReduce against imperative reference implementation.

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 2/27

slide-7
SLIDE 7

Motivation

Simple algorithm implementations are understandable, . . . . . . but not efficient. Efficient implementations are often complex, error-prone = ⇒ Prove equivalence between reference and efficient implementation Verify MapReduce against imperative reference implementation.

Challenge for relational reasoning

Programs not (necessarily) structurally close

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 2/27

slide-8
SLIDE 8

Idea Combine Rewriting and Relational Reasoning

... and be open to automation

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 3/27

slide-9
SLIDE 9

MapReduce

distributed programming framework / paradigm first used large scale by google using concepts from functional programming to allow implicit parallisation. algorithms are quite different to their IMP counterparts

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 4/27

slide-10
SLIDE 10

Recap: MapReduce

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 5/27

slide-11
SLIDE 11

Recap: MapReduce

. . . 3 . . . 1 . . . 4 . . . 1 . . . 5 . . . 9 . . . 2 . . . 6 . . . 5 . . . 3 . . . 5 . . . 9 . . .

MAP

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 5/27

slide-12
SLIDE 12

Recap: MapReduce

. . . 3 . . . 1 . . . 4 . . . 1 . . . 5 . . . 9 . . . 2 . . . 6 . . . 5 . . . 3 . . . 5 . . . 9 . . .

MAP

3 4 1 2 5 9 6 3 1 5 5 9

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 5/27

slide-13
SLIDE 13

Recap: MapReduce

. . . 3 . . . 1 . . . 4 . . . 1 . . . 5 . . . 9 . . . 2 . . . 6 . . . 5 . . . 3 . . . 5 . . . 9 . . .

MAP

3 4 1 2 5 9 6 3 1 5 5 9

REDUCE

15 18 20 Σ Σ Σ

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 5/27

slide-14
SLIDE 14

Recap: MapReduce

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 6/27

slide-15
SLIDE 15

Recap: MapReduce

MAP

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 6/27

slide-16
SLIDE 16

Recap: MapReduce

MAP SHUFFLE REDUCE

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 6/27

slide-17
SLIDE 17

Assumption

The results produced by used reducers do not depend on the order in the array. Then we can consider the deterministic non-distributed setting.

[Commutativity of Reducers, Chen et al. 2016]

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 7/27

slide-18
SLIDE 18

Our Approach

Imperative algorithm MapReduce algorithm

Goal: Imperative ∼ = MapReduce Algorithm

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 8/27

slide-19
SLIDE 19

Our Approach

Imperative algorithm MapReduce algorithm intermed. 1

B

intermed. 2

B

intermed. n

. . .

Goal: Imperative ∼ = MapReduce Algorithm User-provided intermediate steps that guide the proof.

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 8/27

slide-20
SLIDE 20

Our Approach

Imperative algorithm MapReduce algorithm intermed. 1

B

intermed. 2

B

intermed. n

. . .

translates to

λ1 λ2 λ5 λ4 λ3

Goal: Imperative ∼ = MapReduce Algorithm User-provided intermediate steps that guide the proof. Translate into equivalent functional expressions

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 8/27

slide-21
SLIDE 21

Our Approach

Imperative algorithm MapReduce algorithm intermed. 1

B

intermed. 2

B

intermed. n

. . .

translates to

λ1 λ2 λ5 λ4 λ3 ∼ = ∼ = ∼ = ∼ =

Goal: Imperative ∼ = MapReduce Algorithm User-provided intermediate steps that guide the proof. Translate into equivalent functional expressions Prove equivalences there

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 8/27

slide-22
SLIDE 22

Our Approach

Imperative algorithm MapReduce algorithm intermed. 1

B

intermed. 2

B

intermed. n

. . .

translates to

λ1 λ2 λ5 λ4 λ3 ∼ = ∼ = ∼ = ∼ = ∼ = ∼ = ∼ = ∼ =

implies

Goal: Imperative ∼ = MapReduce Algorithm User-provided intermediate steps that guide the proof. Translate into equivalent functional expressions Prove equivalences there Hence, equivalence on the original programs

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 8/27

slide-23
SLIDE 23

Our Approach

Imperative Interaction Language Formalized Functional Language

Imperative algorithm MapReduce algorithm intermed. 1

B

intermed. 2

B

intermed. n

. . .

translates to

λ1 λ2 λ5 λ4 λ3 ∼ = ∼ = ∼ = ∼ = ∼ = ∼ = ∼ = ∼ =

implies

Goal: Imperative ∼ = MapReduce Algorithm User-provided intermediate steps that guide the proof. Translate into equivalent functional expressions Prove equivalences there Hence, equivalence on the original programs

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 8/27

slide-24
SLIDE 24

Languages

Interaction (IL) Formalised Functional (FFL)

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 9/27

slide-25
SLIDE 25

Languages

Interaction (IL) Formalised Functional (FFL) while typed λ calculus (+ µ)

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 9/27

slide-26
SLIDE 26

Languages

Interaction (IL) Formalised Functional (FFL) while typed λ calculus (+ µ) integers, rationals, sequences (arrays)

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 9/27

slide-27
SLIDE 27

Languages

Interaction (IL) Formalised Functional (FFL) while typed λ calculus (+ µ) integers, rationals, sequences (arrays)

P(xs) { sum := 0 for i := 0..n sum := sum + xs[i] return sum; } λxs. fold(λsum.λi.sum + xs[i]) 0 range(0, n)

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 9/27

slide-28
SLIDE 28

Languages

Interaction (IL) Formalised Functional (FFL) while typed λ calculus (+ µ) integers, rationals, sequences (arrays)

P(xs) { sum := 0 for i := 0..n sum := sum + xs[i] return sum; } λxs. fold(λsum.λi.sum + xs[i]) 0 range(0, n)

semantics by translation to FFL Big steps semantics ⇒bs

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 9/27

slide-29
SLIDE 29

Languages

Interaction (IL) Formalised Functional (FFL) while typed λ calculus (+ µ) integers, rationals, sequences (arrays)

P(xs) { sum := 0 for i := 0..n sum := sum + xs[i] return sum; } λxs. fold(λsum.λi.sum + xs[i]) 0 range(0, n)

semantics by translation to FFL Big steps semantics ⇒bs Implemented in Coq

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 9/27

slide-30
SLIDE 30

Languages

Interaction (IL) Formalised Functional (FFL) while typed λ calculus (+ µ) integers, rationals, sequences (arrays)

P(xs) { sum := 0 for i := 0..n sum := sum + xs[i] return sum; } λxs. fold(λsum.λi.sum + xs[i]) 0 range(0, n)

semantics by translation to FFL Big steps semantics ⇒bs Implemented in Coq ∀xs. P(xs) == P’(xs) ∀xs v.(P xs)⇒bsv ↔ (P′ xs)⇒bsv

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 9/27

slide-31
SLIDE 31

Two Types of Rules

Context-Independent Rules

local and uniform rewriting rules on subexpressions for paradigm shifting: (e.g., from loop to map)

Context-Dependent Rules

(more) global and flexible relational reasoning using coupling predicates maintaining control structure, adapt data

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 10/27

slide-32
SLIDE 32

Two Types of Rules

Context-Independent Rules

local and uniform rewriting rules on subexpressions for paradigm shifting: (e.g., from loop to map)

Context-Dependent Rules

(more) global and flexible relational reasoning using coupling predicates maintaining control structure, adapt data

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 10/27

slide-33
SLIDE 33

Context-Independent Rules – Example

Transform parallisable loop to map

fold((λ(xs, i). write(xs, i, f (xs[i]))), ys, range(0, length(xs)))

  • map(f , ys)

xs ∈ FV (f ), i ∈ FV (f ), xs ∈ FV (ys), i ∈ FV (ys), f is not stuck

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 11/27

slide-34
SLIDE 34

Context-Independent Rules – Example

Transform parallisable loop to map

fold((λ(xs, i). write(xs, i, f (xs[i]))), ys, range(0, length(xs)))

  • map(f , ys)

xs ∈ FV (f ), i ∈ FV (f ), xs ∈ FV (ys), i ∈ FV (ys), f is not stuck

IL-level:

for i : range(0, length(xs)) { xs[i] := f(xs[i]) }

  • xs := map(f, xs)

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 11/27

slide-35
SLIDE 35

Context-Independent Rules – Example

Transform parallisable loop to map

fold((λ(xs, i). write(xs, i, f (xs[i]))), ys, range(0, length(xs)))

  • map(f , ys)

xs ∈ FV (f ), i ∈ FV (f ), xs ∈ FV (ys), i ∈ FV (ys), f is not stuck

IL-level:

for i : range(0, length(xs)) { xs[i] := f(xs[i]) }

  • xs := map(f, xs)

Collection of rules

inspected the examples delivered with Thrill. identified 13 typical patterns for steps.

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 11/27

slide-36
SLIDE 36

Context-Independent Rules

1 Extract independent part of loop body to map 2 Group accesses to the same index of an array 3 Group accesses to the same key 4 Fuse consecutive calls to map into a single call of map 5 Separate arrays that are read from and written to 6 Flatten fold over array of arrays 7 Transform iter to fold 8 Transform fold to map 9 fold over the values in an array instead of over index range 10 map over the values in an array instead of over the index range 11 Commute writing back updates to an array and applying map

to the result

12 Commute read and zip 13 Commute read and map Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 12/27

slide-37
SLIDE 37

Context-Independent Rules

1 Extract independent part of loop body to map 2 Group accesses to the same index of an array 3 Group accesses to the same key 4 Fuse consecutive calls to map into a single call of map 5 Separate arrays that are read from and written to 6 Flatten fold over array of arrays 7 Transform iter to fold 8 Transform fold to map 9 fold over the values in an array instead of over index range 10 map over the values in an array instead of over the index range 11 Commute writing back updates to an array and applying map

to the result

12 Commute read and zip 13 Commute read and map Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 12/27

Formalised in Coq

mostly proved

(work in progress)

slide-38
SLIDE 38

Two Types of Rules

Context-Independent Rules

local and uniform rewriting rules on subexpressions for paradigm shifting: (e.g., from loop to map)

Context-Dependent Rules

(more) global and flexible relational reasoning using coupling predicates maintaining control structure, adapt data

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 13/27

slide-39
SLIDE 39

Relational While Rule

while(c1) { B1 } while(c2) { B2 }

1 a loop (plus surrounding statements) can be rewritten ... Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 14/27

slide-40
SLIDE 40

Relational While Rule

[x1 = x2] while(c1){B1} while(c2){B2} [x1 = x2] while(c1) { B1 } while(c2) { B2 }

1 a loop (plus surrounding statements) can be rewritten ... 2 if the two programs can be proved equivalent ... Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 14/27

slide-41
SLIDE 41

Relational While Rule

[x1 = x2] while(c1 or c2) { if(c1) B1; if(c2) B2; } [x1 = x2] [x1 = x2] while(c1){B1} while(c2){B2} [x1 = x2] while(c1) { B1 } while(c2) { B2 }

1 a loop (plus surrounding statements) can be rewritten ... 2 if the two programs can be proved equivalent ... 3 which can be shown using product programs Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 14/27

slide-42
SLIDE 42

Example: PageRank

ranks0(p) = 1 #links

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 15/27

slide-43
SLIDE 43

Example: PageRank

ranks0(p) = 1 #links ∆k(p) =

  • (o,p)∈links

ranksk−1(o) #links[o]

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 15/27

slide-44
SLIDE 44

Example: PageRank

ranks0(p) = 1 #links ∆k(p) =

  • (o,p)∈links

ranksk−1(o) #links[o] ranksk(p) = α · ∆k(p) + 1 − α #links

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 15/27

slide-45
SLIDE 45

Example: PageRank

ranks0(p) = 1 #links ∆k(p) =

  • (o,p)∈links

ranksk−1(o) #links[o] ranksk(p) = α · ∆k(p) + 1 − α #links

Sidenote

a special case of sparse matrix-vector multiplication; broader applications in scientific computing.

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 15/27

slide-46
SLIDE 46

Example: PageRank

fn pageRank(links : [[Int]], α : Rat, iterations : Int) -> [Rat] { var iter : Int := 0; var ranks : [Rat] := replicate(#links, 1. / #links); while (iter < iterations) { var ∆ : [Rat] := replicate(#links, 0); for (pageId : range(0, #links)) { var contribution : Rat := ranks[pageId] / #links[pageId]; for (outgoingId : links[pageId]) { ∆[outgoingId] := ∆[outgoingId] + contribution; } } for (pageId : range(0, #links)) { ranks[pageId] := α * ∆[pageId] + (1-α)/#links; } iter := iter + 1; } return ranks; } 1

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 16/27

slide-47
SLIDE 47

Example: PageRank

fn pageRank(links : [[Int]], α : Rat, iterations : Int) -> [Rat] { var iter : Int := 0; var ranks : [Rat] := replicate(#links, 1. / #links); while (iter < iterations) { var ∆ : [Rat] := replicate(#links, 0); for (pageId : range(0, #links)) { var contribution : Rat := ranks[pageId] / #links[pageId]; for (outgoingId : links[pageId]) { ∆[outgoingId] := ∆[outgoingId] + contribution; } } ranks := map((rank : Rat) => α * rank + (1 - α) / #links, ∆); iter := iter + 1; } return ranks; } 2

Rewrite rule: transform loop to map

for (i : range(0, #as)) { b[i] := g(as[i]); }

  • b := map(g, as)

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 17/27

slide-48
SLIDE 48

Example: PageRank

fn pageRank(links : [[Int]], α : Rat, iterations : Int) -> [Rat] { var iter : Int := 0; var ranks : [Rat] := replicate(#links, 1. / #links); while (iter < iterations) { var ∆ : [Rat] := replicate(#links, 0); for (pageId : range(0, #links)) { var contribution : Rat := ranks[pageId] / #links[pageId]; for (outgoingId : links[pageId]) { ∆[outgoingId] := ∆[outgoingId] + contribution; } } ranks := map((rank : Rat) => α * rank + (1 - α) / #links, ∆); iter := iter + 1; } return ranks; } 2

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 18/27

slide-49
SLIDE 49

Example: PageRank

fn pageRank(links : [[Int]], α : Rat, iterations : Int) -> [Rat] { var iter : Int := 0; var ranks : [Rat] := replicate(#links, 1 / #links); while (iter < iterations) { var ∆ : [Rat] := replicate(#links, 0); var outRanks : [[Int] * Rat] := zip(links, ranks); for (pageId : range(0, #links)) { var contribution : Rat := snd outRanks[pageId] / #(fst outRanks[pageId]); for (outgoingId : fst outRanks[pageId]) { ∆[outgoingId] := ∆[outgoingId] + contribution; } } ranks := map((rank : Rat) => α * rank + (1 - α) / #links, ∆); iter := iter + 1; } return ranks; } 3

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 19/27

slide-50
SLIDE 50

Example: PageRank

fn pageRank(links : [[Int]], α : Rat, iterations : Int) -> [Rat] { var iter : Int := 0; var ranks : [Rat] := replicate(#links, 1 / #links); while (iter < iterations) { var ∆ : [Rat] := replicate(#links, 0); var outRanks : [[Int] * Rat] := zip(links, ranks); for (pageId : range(0, #links)) { var contribution : Rat := snd outRanks[pageId] / #(fst outRanks[pageId]); for (outgoingId : fst outRanks[pageId]) { ∆[outgoingId] := ∆[outgoingId] + contribution; } } ranks := map((rank : Rat) => α * rank + (1 - α) / #links, ∆); iter := iter + 1; } return ranks; } 3

Relational loop invariant

∆1 = ∆2 ∧

  • utRanks2 = zip(links1, ranks1)

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 19/27

slide-51
SLIDE 51

Example: PageRank

fn pageRank(links : [[Int]], α : Rat, iterations : Int) -> [Rat] { var iter : Int := 0; var ranks : [Rat] := replicate(#links, 1 / #links); while (iter < iterations) { var ∆ : [Rat] := replicate(#links, 0); var outRanks : [[Int] * Rat] := zip(links, ranks); for (links_rank : outRanks) { var contribution : Rat := snd links_rank / #fst links_rank; for (outgoingId : fst links_rank) { ∆[outgoingId] := ∆[outgoingId] + contribution; } } ranks := map((rank : Rat) => α * rank + (1 - α) / #links, ∆); iter := iter + 1; } return ranks; } 4

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 20/27

slide-52
SLIDE 52

Example: PageRank

fn pageRank(links : [[Int]], α : Rat, iterations : Int) -> [Rat] { var iter : Int := 0; var ranks : [Rat] := replicate(#links, 1 / #links); while (iter < iterations) { var ∆ : [Rat] := replicate(#links, 0); var outRanks : [[Int] * Rat] := zip(links, ranks); var linksAndContrib : [[Int * Rat]] := map((links_rank : [Int] * Rat) => map((link : Int) => (link, snd links_rank / #fst links_rank), fst links_rank),

  • utRanks);

for (link_contribs : linksAndContrib) { for (link_contrib : link_contribs) { ∆[fst link_contrib] := ∆[fst link_contrib] + snd link_contrib; } } ranks := map((rank : Rat) => α * rank + (1 - α) / #links, ∆); iter := iter + 1; } return ranks; } 5

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 21/27

slide-53
SLIDE 53

Example: PageRank

fn pageRank(links : [[Int]], α : Rat, iterations : Int) -> [Rat] { var iter : Int := 0; var ranks : [Rat] := replicate(#links, 1 / #links); while (iter < iterations) { var ∆ : [Rat] := replicate(#links, 0); var outRanks : [[Int] * Rat] := zip(links, ranks); var linksAndContrib : [[Int * Rat]] := map((links_rank : [Int] * Rat) => map((link : Int) => (link, snd links_rank / #fst links_rank), fst links_rank),

  • utRanks);

for (link_contribs : linksAndContrib) { for (link_contrib : link_contribs) { ∆[fst link_contrib] := ∆[fst link_contrib] + snd link_contrib; } } ranks := map((rank : Rat) => α * rank + (1 - α) / #links, ∆); iter := iter + 1; } return ranks; } 5

Relational loop invariant

∆1 = ∆2 ∧ ∀ij. fst linksAndContrib2[i][j] = (fst outRanks1[i])[j] ∧ snd linksAndContrib2[i][j] = snd outRanks1[i]/#(fst outRanks1[i])

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 21/27

slide-54
SLIDE 54

Example: PageRank

fn pageRank(links : [[Int]], α : Rat, iterations : Int) -> [Rat] { var iter : Int := 0; var ranks : [Rat] := replicate(#links, 1 / #links); while (iter < iterations) { var ∆ : [Rat] := replicate(#links, 0); var outRanks : [[Int] * Rat] := zip(links, ranks); var linksAndContrib : [[Int * Rat]] := map((links_rank : [Int] * Rat) => map((link : Int) => (link, snd links_rank / #fst links_rank), fst links_rank),

  • utRanks);

for (link_contrib : concat(linksAndContrib)) { ∆[fst link_contrib] := ∆[fst link_contrib] + snd link_contrib; } ranks := map((rank : Rat) => α * rank + (1 - α) / #links, ∆); iter := iter + 1; } return ranks; } 6

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 22/27

slide-55
SLIDE 55

Example: PageRank

fn pageRank(links : [[Int]], α : Rat, iterations : Int) -> [Rat] { var iter : Int := 0; var ranks : [Rat] := replicate(#links, 1 / #links); while (iter < iterations) { var ∆ : [Rat] := replicate(#links, 0); var outRanks : [[Int] * Rat] := zip(links, ranks); var linksAndContrib : [Int * Rat] := flatMap((links_rank : [Int] * Rat) => map((link : Int) => (link, snd links_rank / #fst links_rank), fst links_rank),

  • utRanks);

for (link_contrib : linksAndContrib) { ∆[fst link_contrib] := ∆[fst link_contrib] + snd link_contrib; } ranks := map((rank : Rat) => α * rank + (1 - α) / #links, ∆); iter := iter + 1; } return ranks; } 7

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 23/27

slide-56
SLIDE 56

Example: PageRank

fn pageRank(links : [[Int]], α : Rat, iterations : Int) -> [Rat] { var iter : Int := 0; var ranks : [Rat] := replicate(#links, 1 / #links); while (iter < iterations) { var outRanks : [[Int] * Rat] := zip(links, ranks); var contribs : [Int * Rat] := flatMap((links_rank : [Int] * Rat) => map((link : Int) => (link, snd links_rank / #fst links_rank), fst links_rank),

  • utRanks);

var rankUpdates : [Int * Rat] := map((link : Int) (contribs : [Rat]) => (link, fold((x: Rat) (y : Rat) => x + y, 0, contribs)), group(contribs)); var ∆ : [Rat] := replicate(#links, 0); for (link_rank : rankUpdates) { ∆[fst link_rank] := snd link_rank; } ranks := map((rank : Rat) => α * rank + (1 - α) / ∆); iter := iter + 1; } return ranks; } 8

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 24/27

slide-57
SLIDE 57

Example: PageRank

fn pageRank(links : [[Int]], α : Rat, iterations : Int) -> [Rat] { var iter : Int := 0; var ranks : [Rat] := replicate(#links, 1 / #links); while (iter < iterations) { var outRanks : [[Int] * Rat] := zip(links, ranks); var contribs : [Int * Rat] := flatMap((links_rank : [Int] * Rat) => map((link : Int) => (link, snd links_rank / #fst links_rank), fst links_rank),

  • utRanks);

var rankUpdates : [Int * Rat] := map((link : Int) (contribs : [Rat]) => (link, fold((x: Rat) (y : Rat) => x + y, 0, contribs)), group(contribs)); var ∆ : [Rat] := replicate(#links, 0); for (link_rank : rankUpdates) { ∆[fst link_rank] := snd link_rank; } ranks := map((rank : Rat) => α * rank + (1 - α) / ∆); iter := iter + 1; } return ranks; } 8

Rule group-intro

for ((i,v) : xs) { acc[i] := f(acc[i], v); }

  • var xss := map((i,vs) => fold(f, acc[i], vs),

group(acc)); for (x : concat(xss)) { acc := f(acc, x); }

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 24/27

slide-58
SLIDE 58

Example: PageRank

fn pageRank(links : [[Int]], α : Rat, iterations : Int) -> [Rat] { var iter : Int := 0; var ranks : [Rat] := replicate(#links, 1 / #links); while (iter < iterations) { var outRanks : [[Int] * Rat] := zip(links, ranks); var contribs : [Int * Rat] := flatMap((links_rank : [Int] * Rat) => map((link : Int) => (link, snd links_rank / #fst links_rank), fst links_rank),

  • utRanks);

var rankUpdates : [Int * Rat] := reduceByKey(‘+’, 0, contribs); var ∆ : [Rat] := replicate(#links, 0); for (link_rank : rankUpdates) { ∆[fst link_rank] := snd link_rank; } ranks := map((rank : Rat) => α * rank + (1 - α) / #links, ∆); iter := iter + 1; } return ranks; } 9

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 25/27

slide-59
SLIDE 59

Automation

Manual proof:

  • c. 3700 LOC in Coq, mostly handwritten, partially generated

Designed with automation in mind:

user interaction on programming language level only. side conditions of rewrite rules are easily syntactically checked

Future work:

An automated tool as combination of relational reasoning and rewriting

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 26/27

slide-60
SLIDE 60

Conclusion

Equivalence Between Imperative and MapReduce Algorithms

Deductive equivalence verification using a combination of rewriting and relational reasoning equivalence proofs of structurally not so similar programs rewriting rules: change control structure relational reasoning: change data structure next step: automation

Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 27/27