[PPT] - UNDERSTANDING DISTRIBUTED DATAFLOW John Liagouris PowerPoint Presentation

SLIDE 1

Google

UNDERSTANDING DISTRIBUTED DATAFLOW SYSTEMS

OUTPUT EXPLANATION AND PERFORMANCE ANALYSIS

John Liagouris liagos@inf.ethz.ch

3 May 2017

SLIDE 2

PART I: Why is this record in the output of my distributed dataflow? PART II: Why is my distributed dataflow slow?

▸ Concise explanations of individual outputs ▸ On-demand output reproduction ▸ Bottleneck detection ▸ Critical path analysis

2

SLIDE 3

Desislava Dimitrova Vasiliki Kalavri Zaheer Chothia Moritz Hoffmann Andrea Lattuada Timothy Roscoe Sebastian Wicki Frank McSherry

COLLABORATORS

Ralf Sager 3

SLIDE 4

THE BIG PICTURE: UNDERSTANDING THE DATACENTER

The volume of datacenter logs is huge
Keeping archives is not a viable solution
We can process logs online

4

event logs Enterprise Datacenter Strymon

SLIDE 5

THE BIG PICTURE: UNDERSTANDING THE DATACENTER

Strymon is a novel system able to:

Perform deep analytics on thousands of distributed

streams of event logs in parallel

Explain its outputs interactively

5

event logs Enterprise Datacenter Strymon

SLIDE 6

for dataflow systems and different execution models

input stream

utput stream

iterative analytics worker 1 worker 2

synchronous vs asynchronous shared-nothing vs shared-memory

streaming analytics

IDEAS IN STRYMON CAN BE GENERALIZED

6

SLIDE 7

TIMELY DATAFLOW DIFFERENTIAL DATAFLOW

▸ A steaming framework for data-parallel computations ▸ Cyclic dataflows ▸ Logical timestamps (epochs) ▸ Asynchronous execution ▸ Low latency

D. Murray, F. McSherry, M. Isard, R. Isaacs, P. Barham, M. Abadi.

Naiad: A Timely Dataflow System. In SOSP, 2013.

7

F. McSherry, D. Murray, R. Isaacs, M. Isard.

Differential Dataflow. In CIDR, 2013.

▸ A high-level API on top of Timely Dataflow ▸ Incremental computation

SLIDE 8

8

Why is this record in the output of my distributed dataflow?

PART I

SLIDE 9

9

1 2 3

COMPUTATION PROVENANCE

EXPLANATIONS IN DATABASES

SLIDE 10

10

INPUT OUTPUT

THE PROBLEM: OUTPUT EXPLANATION

SLIDE 11

11

INPUT OUTPUT

THE PROBLEM: OUTPUT EXPLANATION

THIS RECORD LOOKS WRONG!

{App 115 344} {VM 233

22}

{App 100 55} {VM 333

124}

… … …

{A 115 344} {F 233 122} {W 100

95}

{V 30 23} … … …

SLIDE 12

12

INPUT OUTPUT

THE PROBLEM: OUTPUT EXPLANATION

THIS RECORD LOOKS WRONG!

{A 115 344} {F 233 122} {W 100

95}

{V 30 23} … … …

{App 115 344} {VM 233

22}

{App 100 55} {VM 333

124}

… … …

SLIDE 13

13

INPUT OUTPUT

THE PROBLEM: OUTPUT EXPLANATION

THIS RECORD LOOKS WRONG!

{A 115 344} {F 233 122} {W 100

95}

{V 30 23} … … …

Output explanation: A subset of the input that is sufficient to reproduce the selected subset of the output

{App 115 344} {VM 233

22}

{App 100 55} {VM 333

124}

… … …

SLIDE 14

14

ANNOTATION-BASED TECHNIQUES

▸ Fast ▸ Explode in size

1 2 3 metadata propagation

SLIDE 15

15

INVERSION-BASED TECHNIQUES

1’ 2’ 3’

▸ Small memory footprint ▸ Not generally applicable

SLIDE 16

16

1 2 3 dependencies

BACKWARD TRACING

▸ Small memory footprint ▸ Generally applicable ▸ Fast

SLIDE 17

PROBLEM 1: TOO MUCH INFORMATION

Use Case: Graph Rechability

17

1 5 2 3 4

SLIDE 18

18

1 5 2 3 4

Use Case: Graph Reachability

▸ Record (1,3) appears in the

result

WHY IS (1,3) IN THE OUTPUT?

PROBLEM 1: TOO MUCH INFORMATION

SLIDE 19

▸ Record (1,3) appears in the

result

▸ Naive backward tracing

returns as an explanation all edges of the graph

19

1 5 2 3 4

WHY IS (1,3) IN THE OUTPUT?

Use Case: Graph Reachability

PROBLEM 1: TOO MUCH INFORMATION

SLIDE 20

▸ Record (1,3) appears in the

result

▸ Naive backward tracing

returns as an explanation all edges of the graph

▸ A shortest path suffices

20

1 5 2 3 4

Use Case: Graph Reachability

WHY IS (1,3) IN THE OUTPUT?

PROBLEM 1: TOO MUCH INFORMATION

SLIDE 21

PROBLEM 2: NOT ENOUGH INFORMATION

21

THE QUICK BROWN FOX … THE LAZY DOG …

A B Use Case: Word Set Difference

SLIDE 22

Use Case: Word Set Difference

22

THE QUICK BROWN FOX … THE LAZY DOG …

▸ Record (doc A, 3 unique words)

appears in the result

WHY ONLY 3 WORDS ARE UNIQUE TO DOCUMENT A?

A B

PROBLEM 2: NOT ENOUGH INFORMATION

(doc A, 3 unique words) (doc B, 2 unique words)

SLIDE 23

23

THE QUICK BROWN FOX … THE LAZY DOG …

▸ Record (doc A, 3 unique words)

appears in the result

▸ Naive backward tracing returns

as an explanation only the words

f doc A

A B Use Case: Word Set Difference

WHY ONLY 3 WORDS ARE UNIQUE TO DOCUMENT A?

PROBLEM 2: NOT ENOUGH INFORMATION

(doc A, 3 unique words) (doc B, 2 unique words)

SLIDE 24

24

THE QUICK BROWN FOX … THE LAZY DOG …

▸ Record (doc A, 3 unique words)

appears in the result

▸ Naive backward tracing returns

as an explanation only the words

f doc A

▸ We also need the words of doc

B to reproduce the record (doc A, 3 unique words) A B Use Case: Word Set Difference

WHY ONLY 3 WORDS ARE UNIQUE TO DOCUMENT A?

PROBLEM 2: NOT ENOUGH INFORMATION

(doc A, 3 unique words) (doc B, 2 unique words)

SLIDE 25

CAN WE SOLVE BOTH PROBLEMS?

25

Yes! Given that the system is able to:

▸ Keep track of the exact point in the computation

a data record was produced

▸ Detect divergent records when replaying the

computation on a subset of the input

We exploit the main features of Differential Dataflow

SLIDE 26

EXPLANATIONS WITH DIFFERENTIAL DATAFLOW

26

Original dataflow:

Op A Op B Op C

INPUT OUTPUT

SLIDE 27

EXPLANATIONS WITH DIFFERENTIAL DATAFLOW

27

Augment the original dataflow with a shadow dataflow Original dataflow:

Op A Op B Op C

INPUT OUTPUT

Explanation dataflow:

Join Join Join

INPUT OUTPUT

SLIDE 28

28

ITERATIVE BACKWARD TRACING

Join Join Join

Explanation dataflow:

QUERY EXPL

Original dataflow:

Op A Op B Op C

INPUT OUTPUT

SLIDE 29

29

ITERATIVE BACKWARD TRACING

Join Join Join

Explanation dataflow:

QUERY EXPL

Original dataflow:

Op A Op B Op C

INPUT OUTPUT

Trace Backwards

SLIDE 30

30

ITERATIVE BACKWARD TRACING

Join Join Join

Explanation dataflow:

QUERY EXPL

Original dataflow:

Op A Op B Op C

INPUT OUTPUT

Replay Compare

SLIDE 31

31

ITERATIVE BACKWARD TRACING

Join Join Join

Explanation dataflow:

QUERY EXPL

Original dataflow:

Op A Op B Op C

INPUT OUTPUT

Trace divergent records backwards

k1 v k2 v’ … … k1 v k2 v’’ … …

SLIDE 32

32

ITERATIVE BACKWARD TRACING

Join Join Join

Explanation dataflow:

QUERY EXPL

Original dataflow:

Op A Op B Op C

INPUT OUTPUT

Replay again (for the new records) Compare Repeat until a fix-point

SLIDE 33

EXAMPLE: EXPLAINING OUTPUTS OF WORD SET DIFFERENCE

THE QUICK BROWN FOX … THE LAZY DOG …

A B

33 33

SLIDE 34

THE QUICK BROWN FOX … THE LAZY DOG …

A B

34

MAP MAP

(THE, A) (BROWN, A) (FOX, A) (THE, B) (LAZY, B) (DOG, B)

EXAMPLE: EXPLAINING OUTPUTS OF WORD SET DIFFERENCE

SLIDE 35

THE QUICK BROWN FOX … THE LAZY DOG …

A B

MAP MAP

(THE, A) (BROWN, A) (FOX, A) (THE, B) (LAZY, B) (DOG, B)

GROUP

(THE, [A,B]) (BROWN, A) (FOX, A) (LAZY, B) (DOG,B)

35

EXAMPLE: EXPLAINING OUTPUTS OF WORD SET DIFFERENCE

SLIDE 36

THE QUICK BROWN FOX … THE LAZY DOG …

A B

MAP MAP

(THE, A) (BROWN, A) (FOX, A) (THE, B) (LAZY, B) (DOG, B)

GROUP

(THE, [A,B]) (BROWN, A) (FOX, A) (LAZY, B) (DOG,B)

FILTER

(BROWN,A) (FOX,A) (LAZY,B) (DOG,B)

36

EXAMPLE: EXPLAINING OUTPUTS OF WORD SET DIFFERENCE

SLIDE 37

THE QUICK BROWN FOX … THE LAZY DOG …

A B

MAP MAP

(THE, A) (BROWN, A) (FOX, A) (THE, B) (LAZY, B) (DOG, B)

GROUP

(THE, [A,B]) (BROWN, A) (FOX, A) (LAZY, B) (DOG,B)

FILTER

(BROWN,A) (FOX,A) (LAZY,B) (DOG,B)

37

GROUP

(A, 3) (B, 2)

EXAMPLE: EXPLAINING OUTPUTS OF WORD SET DIFFERENCE

SLIDE 38

THE QUICK BROWN FOX … THE LAZY DOG …

A B

MAP MAP

(THE, A) (BROWN, A) (FOX, A) (THE, B) (LAZY, B) (DOG, B)

GROUP

(THE, [A,B]) (BROWN, A) (FOX, A) (LAZY, B) (DOG,B)

FILTER

(BROWN,A) (FOX,A) (LAZY,B) (DOG,B)

38

GROUP

(A, 3) (B, 2)

EXAMPLE: EXPLAINING OUTPUTS OF WORD SET DIFFERENCE

SLIDE 39

THE QUICK BROWN FOX … THE LAZY DOG …

A B

MAP MAP

(THE, A) (BROWN, A) (FOX, A) (THE, B) (LAZY, B) (DOG, B)

GROUP

(THE, [A,B]) (BROWN, A) (FOX, A) (LAZY, B) (DOG,B)

FILTER

(BROWN,A) (FOX,A) (LAZY,B) (DOG,B)

39

GROUP

(A, 3) (B, 2)

EXAMPLE: EXPLAINING OUTPUTS OF WORD SET DIFFERENCE

(THE, A)

SLIDE 40

THE QUICK BROWN FOX … THE LAZY DOG …

A B

MAP MAP

(THE, A) (BROWN, A) (FOX, A) (THE, B) (LAZY, B) (DOG, B)

GROUP

(THE, [A,B]) (BROWN, A) (FOX, A) (LAZY, B) (DOG,B)

FILTER

(BROWN,A) (FOX,A) (LAZY,B) (DOG,B)

40

GROUP

(A, 3) (B, 2) (THE, A)

EXAMPLE: EXPLAINING OUTPUTS OF WORD SET DIFFERENCE

SLIDE 41

RESULTS: EXPLAINING CONNECTED COMPONENTS

41

▸ Dataset: A subset of the Twitter graph with 1B edges ▸ Algorithm: Label propagation ▸ Output: Records of the form (A,B) denoting that nodes A and B belong

to the same connected component

▸ System used: Differential Dataflow ▸ Machine used: Intel Xeon E5-4640 at 2.4GHz with 32 cores and 500G

RAM

More results:

Z. Chothia, J. Liagouris, F. McSherry, T. Roscoe Explaining Outputs

in Modern Data Analytics PVDLB 9(12):1137-1148, 2016.

SLIDE 42

EXPLAINING CONNECTED COMPONENTS

42

Twitter

SLIDE 43

43

Why is my distributed dataflow slow?

PART II

SLIDE 44

client

W1 W1

scheduler

DISTRIBUTED DATAFLOWS

44 Apache Flink Naiad

SLIDE 45

client

W1 W1

scheduler

▸ many processes and

activities

▸ the cause is usually

not isolated but spans multiple processes

CHALLENGE: TROUBLESHOOTING IS HARD

45

SLIDE 46

PROFILING: THE PROGRAM ACTIVITY GRAPH (PAG)

▸ Models Happened-Before relationships

46

SLIDE 47

PROFILING: THE PROGRAM ACTIVITY GRAPH (PAG)

▸ Vertices: events with timestamps

47

SLIDE 48

PROFILING: THE PROGRAM ACTIVITY GRAPH (PAG)

▸ Edges: duration of activities

48

SLIDE 49

PROFILING: THE PROGRAM ACTIVITY GRAPH (PAG)

▸ Wait edges: time spent in waiting for a message

49

SLIDE 50

CRITICAL PATH ANALYSIS

The critical path is the path of non-waiting activities in the execution history of the program with the longest duration

50

SLIDE 51

CRITICAL PATH ANALYSIS

51

The program activity graph is a DAG so the critical path computation is tractable

SLIDE 52

CRITICAL PATH ANALYSIS

52

The critical path is constructed by starting from the last event and backtracking:

Following the edges with the longest duration
Avoiding waiting edges

SLIDE 53

How can we compute the critical path in long-running, dynamic distributed applications, with possibly unbounded input?

▸ There may be no “job end” ▸ The PAG is evolving while the job is running ▸ Stale profiling information is not useful

53

SLIDE 54

TRANSIENT CRITICAL PATHS (TCPS)

54

An adaptation of the standard critical path on trace snapshots

▸ tumbling, sliding or custom windows

ts te w1 w2 w1 w2

a b c d e f g h i b’ c’ d’ e’ f’ g’ h’ i’ 1 2 1 1 1 3 2 4 2 1 1 1 1 1 3 1 1

Input Trace Snapshot in [ts,te]

SLIDE 55

ts te w1 w2 w1 w2

a b c d e f g h i b’ c’ d’ e’ f’ g’ h’ i’ 1 2 1 1 1 3 2 4 2 1 1 1 1 1 3 1 1

55

b’ c’ d’ e’ g’ h’ b’ c’ d’ g’ h’ i’ c’ d’ e’ g’ h’ f’ c’ d’ g’ h’ f’ i’ b’ g’ h’ i’ g’ h’ f’ i’

TRANSIENT CRITICAL PATHS (TCPS)

Snapshot in [ts,te] Transient Critical Paths

Multiple transient critical paths per snapshot

▸ All TCPs are possible parts of the unknown global critical path

in the snapshot [ts,te]{

SLIDE 56

56

b’ c’ d’ e’ g’ h’ b’ c’ d’ g’ h’ i’ c’ d’ e’ g’ h’ f’ c’ d’ g’ h’ f’ i’ b’ g’ h’ i’ g’ h’ f’ i’

TRANSIENT PATH CENTRALITY (TPC)

The number of transient critical paths an edge belongs to

TPC(d’,i’) = 2 TPC(g’,h’) = 6

SLIDE 57

57

b’ c’ d’ e’ g’ h’ b’ c’ d’ g’ h’ i’ c’ d’ e’ g’ h’ f’ c’ d’ g’ h’ f’ i’ b’ g’ h’ i’ g’ h’ f’ i’

AVERAGE CRITICAL PARTICIPATION (CP)

An estimation of the activity’s participation in the critical path

CP(d’,i’) = 2*1/6*5 = 0.066 CP(g’,h’) = 6*1/6*5 = 0.2

1 1 1 1 1 1 1 1

CPa = TPC(a) · aw N(te ts)

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 3

number of transient critical paths edge weight

SLIDE 58

TRANSIENT CRITICAL PATHS ARE WIDELY APPLICABLE

58 “RDDs” “DataStreams” “Spouts and Bolts”

▸ data transformation ▸ data exchange ▸ control messages ▸ I/O ▸ data (de)-serialization ▸ buffer management ▸ scheduling

}

common set of low-level primitives!

“Tensors” Naiad

SLIDE 59

RESULTS: COMPARISON WITH CONVENTIONAL PROFILING

59

▸ Benchmark: TPC-DS [1] ▸ System under study: Spark (1.2.1) ▸ Setting: 20 machines with 8 workers each ▸ We actually used Spark logs from [2] ▸ Snapshot interval: 10 sec

[2] Ousterhout, K. Spark performance analysis (accessed: April 2017) https://kayousterhout.github.io/trace- analysis/ [1] TPC-DS. http://www.tpc.org/tpcds/

SLIDE 60

COMPARISON WITH CONVENTIONAL PROFILING

60

200 400 Snapshot 0.0 0.2 0.4 0.6 0.8 1.0 CP 200 400 Snapshot % weight Shuffling Processing Serialization Deserialization ControlMessage Unknown

SLIDE 61

COMPARISON WITH CONVENTIONAL PROFILING

61

200 400 Snapshot 0.0 0.2 0.4 0.6 0.8 1.0 CP 200 400 Snapshot % weight Shuffling Processing Serialization Deserialization ControlMessage Unknown

“Optimizing disk usage can improve performance by a median of at most 19%”

Ousterhout, K., Rasti, R., Ratnasamy, S., Shenker, S., and Chun, B.-G. Making sense of performance in data analytics frameworks. In NSDI (2015).

SLIDE 62

ONGOING AND FUTURE WORK

62 reference application Strymon

Timely

event logs

nline critical

path analysis

real-time performance summaries

adaptive scheduling dynamic scaling straggler mitigation

feedback dynamic job management

SLIDE 63

INTERESTING QUESTIONS

63

▸ Can we use the Program Activity Graph to verify

instrumentation?

▸ What is the appropriate snapshot size for analyzing the

performance of a dataflow execution?

▸ Can we use sampling to reduce the number of snapshots

we examine without affecting the quality of the results?

SLIDE 64

SUMMARY

64

reference application Strymon

Timely

event logs

nline critical

path analysis

real-time performance summaries adaptive scheduling dynamic scaling straggler mitigation feedback real-time job performance management

PART I: Iterative Backward Tracing

Part II: Transient Critical Path Analysis

concise explanations

utput reproduction

guarantees

real-time performance summaries

transient critical paths

IN OUT OUT IN

interactive times continuous computations

SLIDE 65

3 May 2017 Google

John Liagouris liagos@inf.ethz.ch

UNDERSTANDING DISTRIBUTED DATAFLOW SYSTEMS

OUTPUT EXPLANATION AND PERFORMANCE ANALYSIS

SLIDE 66

RESULTS: PROVENANCE OVERHEAD IN DATALOG

66

SNOMED CT GALEN8

SLIDE 67

RESULTS: EXPLANATIONS IN DATALOG

67

SNOMED CT GALEN8

SLIDE 68

RESULTS: UPDATING PROVENANCE IN DATALOG

68

SNOMED CT GALEN8

SLIDE 69

RESULTS: PROVENANCE OVERHEAD IN CONNECTED COMPONENTS

69

SLIDE 70

RESULTS: UPDATING EXPLANATIONS IN CONNECTED COMPONENTS

70

SLIDE 71

RESULTS: EXPLAINING STABLE MATCHING IN GRAPHS

71

▸ Dataset: A subset of the Twitter graph with 300M edges ▸ Algorithm: Stable Matching ▸ Output: Records of the form (A,B) denoting that nodes A and B

matched

▸ System used: Differential Dataflow ▸ Machine used: Intel Xeon E5-4640 at 2.4GHz with 32 cores and 500G

RAM

SLIDE 72

EXPLAINING STABLE MATCHING IN GRAPHS

72

SLIDE 73

RESULTS: PROVENANCE OVERHEAD IN STABLE MATCHING

73

SLIDE 74

RESULTS: UPDATING EXPLANATIONS IN STABLE MATCHING

74

SLIDE 75

RESULTS: COMPARISON WITH SINGLE-PATH APPROACH

75

▸ Benchmark: Yahoo Streaming Benchmark (YSB) [1] ▸ System under study: Flink (1.2.0) ▸ Setting: 1 machine with 8 workers ▸ Snapshot interval: 1 sec

[1] Yahoo Streaming Benchmark. https://github.com/yahoo/streaming-benchmarks

SLIDE 76

COMPARISON WITH SINGLE-PATH APPROACH

76

100 200 300 Snapshot 0.0 0.2 0.4 0.6 0.8 1.0 CP 100 200 300 Snapshot Single path CP Input Buffer Scheduling Processing BarrierProcessing Serialization Deserialization ControlMessage DataMessage Unknown

SLIDE 77

RESULTS: PROFILING DIFFERENT PHASES OF A ML JOB

77

▸ Benchmark: AlexNet program [1] on ImageNet [2] ▸ System under study: TensorFlow (1.0.1) ▸ Setting: 1 machine 16 workers (CPU threads) ▸ Snapshot interval: 1 sec

Krizhevsky, A., Sutskever, I., and Hinton, G. E. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2012,

pp. 1097–1105.

[1] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet Large Scale Visual Recognition

Challenge. International Journal of Computer Vision (IJCV) 115, 3 (2015), 211–252.

[2]

SLIDE 78

Activities: Unknown ControlMessage DataMessage Operator Categories: Math Primitives State and Initialization Transformations Machine Learning

PROFILING DIFFERENT PHASES OF A MACHINE LEARNING JOB

78

initialization training accuracy

1 2 3 0.0 0.2 0.4 0.6 0.8 1.0 CP 50 100 20 40 Snapshot

SLIDE 79

STABILITY ACROSS DIFFERENT SNAPSHOT INTERVALS

79

200 400 Snapshot 0.0 0.2 0.4 0.6 0.8 1.0 CP 200 400 Snapshot % weight Shuffling Processing Serialization Deserialization ControlMessage Unknown 20 40 Snapshot 0.0 0.2 0.4 0.6 0.8 1.0 CP

SLIDE 80

COMMUNICATION SKEW IN TIMELY DATAFLOW

80

SLIDE 81

COMPUTATION SKEW IN FLINK

81

50 100 Snapshot 0.0 0.1 0.2 0.3 CP 50 100 Snapshot % weight