Faucet: a user-level, modular technique for flow control in dataflow - - PowerPoint PPT Presentation

Faucet: a user-level, modular technique for flow control in dataflow engines

Andrea Lattuada Systems Group, ETH Zürich Frank McSherry Unaffiliated Zaheer Chothia Systems Group, ETH Zürich 1

2

RAM exhaustion due to buffered intermediate results

10-100x memory savings for 15-25% runtime overhead

Problem

• no system-level general strategy
• application-driven scheduling

Our Solution

Source of the problem Rate imbalance

Nout(t, tʹ) Nin(t, tʹ) 5 1 2 3 4 5 flat_map(|x| [1, …, x])

3

• perators

channels Storm Flink Naiad

Dataflow model

source

(,,,) (,,,) (,,,) (,,,) (,,,) A B

RAM exhaustion large

• utput

rate

• perator
• utput

buffered in RAM

Existing approach #1 - Source backpressure

4

Storm Heron Spark streaming source

• verloaded
• perators

backpressure signal

stopped by F

F G H

stopped by G source deadlock similar to TCP flow control

5

Existing approach #2 - Edge-by-edge backpressure

Akka Streams Flink

• verloaded
• perator

source backpressure signal

control scheduling to limit intermediate results

6

Our approach - Faucet based on Timely Dataflow’s concepts

• no fine-grained signal
• track completion of a batch
• f tuples

Scopes Timestamps nested operator structure tuple metadata (t1) enter leave

(,,)

Foundation - Timely Dataflow’s Progress Tracking

7

Scopes Timestamps nested operator structure tuple metadata

Foundation - Timely Dataflow’s Progress Tracking

7

(t1) enter leave

(,,)

(t1,t2)

(,,) (,,)

Foundation - Timely Dataflow’s Progress Tracking

7

Scopes Timestamps nested operator structure tuple metadata (t1) enter leave

(,,)

(t1,t2)

(,,) (,,)

(t1)

(,,)

Progress Tracking tracks pending timestamps (3,4) (3,4) in flight

(,,)

Foundation - Timely Dataflow’s Progress Tracking

7

Scopes Timestamps nested operator structure tuple metadata (t1) enter leave

(,,)

(t1,t2)

(,,) (,,)

(t1)

(,,)

probe batcher

scope controlled subgraph

8

Faucet - Track batches of intermediate results

(te)

(,,) (,,) (,,)

8

probe batcher scope controlled subgraph

Faucet - Track batches of intermediate results

(te,tb)

(,,) (,,) (,,)

(te)

(,,) (,,) (,,)

8

probe batcher scope controlled subgraph

Faucet - Track batches of intermediate results

(,,) (,,) (,,)

8

(te,tb)

(,,) (,,) (,,)

(te)

(,,) (,,) (,,)

8

probe batcher scope controlled subgraph

Faucet - Track batches of intermediate results

(,,) (,,)

(te,2)

(,,) (,,) (,,)

8

(te,tb)

(,,) (,,) (,,)

(te)

(,,) (,,) (,,)

8

probe batcher scope controlled subgraph (te,2) pending

Faucet - Track batches of intermediate results

• H. Q. Ngo, C. Ré, and A. Rudra - Generic Join

input graph

build result tuples by extending prefixes

(a11) (a11,a22) (a11,a22,a32)

9

Example - Enumerate triangles in a directed graph

a11 a21 a22 a31 a32 a33

a11 a21 a22 a31 a32 a33

9

Example - Enumerate triangles in a directed graph

(a11) propose Pa

9

a11 a21 a22 a31 a32 a33

Example - Enumerate triangles in a directed graph

(a11) (a11,a21) (a11,a22) (a11,a32) propose Pa

9

a11 a21 a22 a31 a32 a33

Example - Enumerate triangles in a directed graph

(a11) (a11,a21) (a11,a22) (a11,a32) (a11,a21,a31) (a11,a22,a32) (a11,a22,a33) propose propose count proposals Pa Pb1 Pb2 C

(a11) (a11,a21) (a11,a22) (a11,a32) (a11,a21,a31) (a11,a22,a32) (a11,a22,a32) (a11,a22,a33)

9

a11 a21 a22 a31 a32 a33

Example - Enumerate triangles in a directed graph

propose propose count proposals intersect Pa Pb1 Pb2 I1

T

C I2

10

propose propose count proposals intersect Pa Pb1 Pb2 I1

T

C I2

10

A naïve schedule can generate large intermediate state

propose propose count proposals intersect Pa Pb1 Pb2 I1

T

C I2

10

A naïve schedule can generate large intermediate state

(a11) (a12)

propose propose count proposals intersect Pa Pb1 Pb2 I1

T

C I2

10

A naïve schedule can generate large intermediate state

(a11) (a12) (a11,a21) (a11,a22) (a12,a23)

propose propose count proposals intersect Pa Pb1 Pb2 I1

T

C I2

10

A naïve schedule can generate large intermediate state

(a11) (a12) (a11,a21) (a11,a22) (a12,a23) (a11,a21,a31) (a11,a22,a32) (a11,a22,a33) (a12,a23,a34) (a12,a23,a35)

Faucet limits buffered intermediate results

(a11) (a12) (a11,a21) (a11,a22) (a12,a23) (a11,a21,a31) (a11,a22,a32) (a11,a22,a33) (a12,a23,a34) (a12,a23,a35)

11

large

• utput

rate

11

(a11) (a12) (a11,a21) (a11,a22) (a12,a23) (a11,a21,a31) (a11,a22,a32) (a11,a22,a33) (a12,a23,a34) (a12,a23,a35)

Faucet limits buffered intermediate results

11

(a11) (a12) (a11,a21) (a11,a22) (a12,a23) (a11,a21,a31) (a11,a22,a32) (a11,a22,a33) (a12,a23,a34) (a12,a23,a35)

Faucet limits buffered intermediate results

11

(a11) (a12) (a11,a21) (a11,a22) (a12,a23) (a11,a21,a31) (a11,a22,a32) (a11,a22,a33) (a12,a23,a34) (a12,a23,a35)

Faucet limits buffered intermediate results

12

Enumerate triangles in the Livejournal Dataset Hardware 4’847’571 nodes 68’993’773 edges 285’730’264 triangles Intel Xeon E5-2650 @ 2.00GHz 16 physical cores 10Gbps link

Evaluation - Dataset

20 40 60 80 100 100 1000 10000 100000 runtime (sec) batch size (# tuples)

Nbatches number of batches in-flight in parallel batch size Nbatches ≥ 2 mitigates stragglers B

13

Evaluation - Sensitivity to parameter choice

2 nodes x 4 threads

100 1000 10000 100000 100K 1000K total ram (MB) input size (tuples)

uncontrolled with Faucet

Evaluation

14

Memory savings

2 nodes x 4 threads

10x 100x

Runtime overhead

15-25%

probe scope batcher controlled subgraph

RAM is increasingly the main cost of a system Memory savings 10-100x

• r more

Overhead 15-25% limits intermediate state Faucet

15