CIEL: A UNIVERSAL EXECUTION ENGINE FOR DISTRIBUTED DATA-FLOW - - PowerPoint PPT Presentation

▶
ciel a universal execution engine for distributed data
SMART_READER_LITE
LIVE PREVIEW

CIEL: A UNIVERSAL EXECUTION ENGINE FOR DISTRIBUTED DATA-FLOW - - PowerPoint PPT Presentation

Dec 21, 2022 216 likes •414 views

CIEL: A UNIVERSAL EXECUTION ENGINE FOR DISTRIBUTED DATA-FLOW COMPUTING Derek G. Murray, Malte Schwarzkopf, Christopher Smowton, Steven Smith, Anil Madhavapeddy, Steven Hand University of Cambridge Computer Laboratory INTRODUCTION

slide-1
SLIDE 1

CIEL: A UNIVERSAL EXECUTION ENGINE FOR DISTRIBUTED DATA-FLOW COMPUTING

Derek G. Murray, Malte Schwarzkopf, Christopher Smowton, Steven Smith, Anil Madhavapeddy, Steven Hand University of Cambridge Computer Laboratory

slide-2
SLIDE 2

INTRODUCTION

  • Background Influences
  • What is CIEL?
  • Features
  • Skywriting
  • Evaluation
  • Conclusions
slide-3
SLIDE 3

BACKGROUND INFLUENCES

  • Map-Reduce/Hadoop
  • Dryad
  • Pregel
  • Piccolo
slide-4
SLIDE 4

WHAT IS CIEL?

  • Universal data-centric distributed execution engine
  • Designed for large dataset, coarse-grained parallelism
  • Based on data-dependent dynamic control flow
  • Uses 3 primitives - objects, references and tasks
  • Primary Goal is to produce object output
slide-5
SLIDE 5

FEATURES

  • Dynamic task graphs
  • System architecture
  • Deterministic naming & Memoisation
  • Fault tolerance
  • Streaming
slide-6
SLIDE 6

DYNAMIC TASK GRAPHS

Objects

  • Unstructured finite-length sequence of bytes
  • Unique name
  • Immutable when written
slide-7
SLIDE 7

DYNAMIC TASK GRAPHS

References

  • Comprises name and set of locations where object is stored
  • Can be a future reference to object yet produced
slide-8
SLIDE 8

DYNAMIC TASK GRAPHS

Tasks

  • Non-blocking atomic computation
  • Has one or more dependencies - represented as references
  • Includes special object that specifies the behaviour of the task
  • Two externally-observable behaviours - publish objects and spawn new tasks
slide-9
SLIDE 9

DYNAMIC TASK GRAPHS

Object Evaluation

  • Role = evaluate one or more objects corresponding to job outputs
  • Job can be specified as single root task with only concrete dependencies
  • Two natural strategies - Eager and Lazy evaluation
slide-10
SLIDE 10

FEATURES

  • Dynamic task graphs
  • System architecture
  • Deterministic naming & Memoisation
  • Fault tolerance
  • Streaming
slide-11
SLIDE 11

SYSTEM ARCHITECTURE

  • Single master coordinating end-to-end execution of jobs
  • Several workers are used for execution of individual tasks
  • DTG maintained by master in object and task table
  • Master Scheduler (multiple queue based) responsible for making progress in CIEL

computation

  • Executor = generic component that prepares input data for consumption
slide-12
SLIDE 12

FEATURES

  • Dynamic task graphs
  • System architecture
  • Deterministic naming & Memoisation
  • Fault tolerance
  • Streaming
slide-13
SLIDE 13

FEATURES

  • Dynamic task graphs
  • System architecture
  • Deterministic naming & Memoisation
  • Fault tolerance
  • Streaming
slide-14
SLIDE 14

FEATURES

  • Dynamic task graphs
  • System architecture
  • Deterministic naming & Memoisation
  • Fault tolerance
  • Streaming
slide-15
SLIDE 15

SKYWRITING

  • Key Features - ref, spawn, exec., spawn.exec, the dereference operator
  • Tasks - key feature = ability to spawn new tasks in the middle of jobs
  • Data-dependent control flow
slide-16
SLIDE 16

EVALUATION

  • Grep
  • k-means
  • Smith-Waterman
  • Binomial options pricing
  • Fault-tolerance
slide-17
SLIDE 17

CONCLUSIONS

  • Superset of features of existing distributed engines
  • Skywriting
  • Flexibility - Supports MapReduce job or Dryad graph
  • System-wide fault tolerance
  • Streaming
  • Memoisation
slide-18
SLIDE 18

THANKS

  • Any Questions?