Pre-production and Debugging Tools for Timely dataflow CS 848: - - PowerPoint PPT Presentation

▶
pre production and debugging tools for timely dataflow
SMART_READER_LITE
LIVE PREVIEW

Pre-production and Debugging Tools for Timely dataflow CS 848: - - PowerPoint PPT Presentation

Aug 04, 2023 358 likes •770 views

Pre-production and Debugging Tools for Timely dataflow CS 848: Models and Applications of Distributed Data Systems Mon, Dec 5th 2016 Amine Mhedhbi & Saifuddin Hitawala Distributed Data Processing Systems in 2006 Distributed Data Processing

slide-1
SLIDE 1

Pre-production and Debugging Tools for Timely dataflow

CS 848: Models and Applications of Distributed Data Systems Mon, Dec 5th 2016

Amine Mhedhbi & Saifuddin Hitawala

slide-2
SLIDE 2

Distributed Data Processing Systems in 2006

slide-3
SLIDE 3

Distributed Data Processing Systems in 2016

slide-4
SLIDE 4

Many topics of Interest Within These Systems

slide-5
SLIDE 5

We Picked ....

slide-6
SLIDE 6

Project Statement

  • “Timely Dataflow” is a rewrite of Naiad System in Rust

under the MIT License. * Prototype *

  • Goal:
slide-7
SLIDE 7

Flash Back of the Past

slide-8
SLIDE 8

Background

slide-9
SLIDE 9

Background

"OperatesEvent": // Type of the logged obj { "id": int, // unique id. "addr": [int, int, int], // address in terms of scope & id. "name": String, // operators name in timely dataflow }

slide-10
SLIDE 10

Background

"OperatesEvent": { ... "name": “OP1” } "OperatesEvent": { ... "name": “OP2” }

slide-11
SLIDE 11

Background

"ChannelsEvent": { "id": int, // unique id "scope_addr": [int, int], // scope & worker id "source": [int, int], // [op_id, scope_id] "target": [int, int], // [op_id, scope_id] }

slide-12
SLIDE 12

Background

"MessageEvent": { "is_send": bool, // push or pull "channel": int, // unique id "source": int, // worker id "target": int, // worker id "length": int, // number of typed records }

slide-13
SLIDE 13

Related Work

slide-14
SLIDE 14

Related Work : Tensorflow Dashboard & Apache Stats

slide-15
SLIDE 15

Features

slide-16
SLIDE 16

Features

  • Visualize The Computation Topology
slide-17
SLIDE 17

Features

  • Visualize The Computation Topology
  • Report skew between workers
slide-18
SLIDE 18

Features

  • Visualize The Computation Topology
  • Report skew between workers
  • Replay computation step-by-step

visually

slide-19
SLIDE 19

Features

  • Visualize The Computation Topology
  • Report skew between workers
  • Real-Time Machine Monitoring
  • Replay computation step-by-step

visually

slide-20
SLIDE 20

DEMO TIME(ly)!

slide-21
SLIDE 21

Experiments & Evaluation

slide-22
SLIDE 22

Pingpong: Topology

slide-23
SLIDE 23

Pingpong: Experimental Runs, num of iterations = 10000

Used Himrod Cluster with machines having 256GB memory

slide-24
SLIDE 24

Pingpong: Experimental Runs, num of iterations = [10, 100, 1000, 10000]

slide-25
SLIDE 25

BFS: Topology

slide-26
SLIDE 26

BFS: Experimental Runs

slide-27
SLIDE 27

Web App Back-end Profiling In Progress:

  • Profile server-client response time for the 4 features.
slide-28
SLIDE 28

Conclusion

slide-29
SLIDE 29

Conclusions

  • JSON -> Binary for logging.
slide-30
SLIDE 30

Conclusions

  • JSON -> Binary for logging.
  • Large scale testing is a must.
slide-31
SLIDE 31

Conclusions

  • Project is a prototype. A lot of needed improvements:
slide-32
SLIDE 32

Conclusions

  • Project is a prototype. A lot of needed improvements:
slide-33
SLIDE 33

Conclusions

  • Project is a prototype. A lot of needed improvements:
slide-34
SLIDE 34

Conclusions

  • Project is a prototype. A lot of needed improvements:
slide-35
SLIDE 35

Future Work

slide-36
SLIDE 36

Future Work

  • Real-Time Computation Monitoring
slide-37
SLIDE 37

Future Work

  • Real-Time Computation Monitoring
  • UI code generation (drag & drop) for

small computation

slide-38
SLIDE 38

Future Work

  • Real-Time Computation Monitoring
  • UI code generation (drag & drop) for

small computation

  • Step-by-step debugging of multiple

workers computations?!

slide-39
SLIDE 39

Resources

  • Timely Dataflow (Rust Implementation)
  • Frank blog posts:

○ Timely dataflow ○ Differential dataflow

  • Naiad Paper
  • For slides [2-5]: Class slides by Prof. Semih Salihoglu
slide-40
SLIDE 40

Fin.

Thank you! Q&A?!