CS 744: DATAFLOW Shivaram Venkataraman Fall 2020 - - PowerPoint PPT Presentation

CS 744: DATAFLOW

Shivaram Venkataraman Fall 2020

welcome

!

ADMINISTRIVIA

• Assignment 2 grades are up!
• Midterm grading in progress
• Course project proposal comments
• AEFIS feedback (next slide)

→

Canvas

↳

Peer

feedback

Thursday

tis

week

↳

Instructor

feedback

AEFIS FEEDBACK

Improve writing on the slides, speak slower Get a better internet connection? Better microphone? More office hour slots Discussion groups: same group each time? Also add prof. input More time for Midterm exam, more guidance on deliverables More homework/hands-on experience vs. too many evaluation components?

Better

• rganization

T

• ↳

Let

me

know how

it

this

sounds ?

ring

Scalable Storage Systems Datacenter Architecture Resource Management Computational Engines Machine Learning SQL Streaming Graph Applications

J f

f-

stream Processing

→ MapReduce

→

GFS

→

Meson

DRF

DATAFLOW MODEL (?)

• perators
• r

DAA

• f
• perators

spat

tape

"

Pytorch

MOTIVATION

Streaming Video Provider

• How much to bill each advertiser ?
• Need per-user, per-video viewing sessions
• Handle out of order data

Goals

• Easy to program
• Balance correctness, latency and cost

ESPN

• videos

each

video

has

some

ads

Foard

↳

main

phone

heard

each )

→ which city etc .

Offline

• 1
• unbounded

data ,

• ut
• f
• rder
• how

much delay till

results

are

↳

how accurate

are

your

results

available

APPROACH

API Design Separate user-facing model from execution Decompose queries into

• What is being computed
• Where in time is it computed
• When is it materialized
• How does it relate to earlier results

Developers writing

→ Dataflow Model

applications

I

L

L

TENET

d) Framework

processes

→ Output

data

bounded

arrives

data

similar to very

streaming

a

① MapReduce

②

small

batch

• iii. i

FEE

:# ftp..sk

viewing

→

e→

events

process

events

• '

and

arrive

TERMINOLOGY

Unbounded/bounded data Streaming/Batch execution Timestamps Event time: Processing time:

⇒

Dashboard

Processing - time

Syst€)

②

→

Data

is

constantly

arriving#

ESPN .com

↳

See

previous

slide

• ad

µµmtmt¥

'D )

• Time

when

event

• ccurs

wrt

user ( input

e.g ;

time

at

ad

was

viewed

in video

Time

at

which

an

event

is

processed

time

at

which

ad

• view

event

is

processed

to

update

the

dashboard

WINDOWING

logical

constructs

winadroewsae.ge/:;:::n?I:soam/

^

across

keys

window

• Id

• I -
• -

• remake

• FF
• ¥

,

I

'

HIGH

#

# rpm-

• Do

not

• vereat

←

Tuning

• imapbeueen

)

↳ noamtmatauidne

with each

windows

consecutive keys

• ther

windows

WATERMARK or SKEW

System has processed all events up to 12:02:30

Watermark

is

not

easy

to know

• Heuristics

you

• processing

time

• •

• After

10 mins

lags

event

time

most devices

serial

events

• Event

time skew

catch up

:

• .

• No

gap

between event - t &

processing time

API

ParDo: GroupByKey: Windowing AssignWindow MergeWindow

Map

in

MapReduce

• r

flatmate

Spark

Reduce

in

MapReduce

→

Buckets

tuple

into

a

window

→

Merge

buckets

based

• n

strategy

( sessions)

EXAMPLE

GroupByKey

Assign tuples

to

sessions

hwan

,

meant

• timestamp
• I

÷i¥

• verlap

and

• aedrdenfo.fi/ftamp
• merges

them

• .

I

TRIGGERS AND INCREMENTAL PROCESSING

Windowing: where in event time data are grouped Triggering: when in processing time groups are emitted Strategies Discarding Accumulating Accumulating & Retracting

.

÷

: FEI

;÷::?;÷iwsr

.

v1

6

Output

I

• 5

11

retracting

ataumulahng

RUNNING EXAMPLE

PCollection<KV<String, Integer>> input = IO.read(...); PCollection<KV<String, Integer>> output = input.apply(Sum.integersPerKey());

• ←

Single

summit for

f

key

each

key

'

GLOBAL WINDOWS, ACCUMULATE

PCollection<KV<String, Integer>> output = input .apply(Window.trigger(Repeat(AtPeriod(1, MINUTE))) .accumulating()) .apply(Sum.integersPerKey());

÷

33 t 18

I

22HI

• '

12+10

Crigger ed→

• .

O

every A

min in

ProutFane

GLOBAL WINDOWS, COUNT, DISCARDING

PCollection<KV<String, Integer>> output = input .apply(Window.trigger(Repeat(AtCount(2))) .discarding()) .apply(Sum.integersPerKey());

• r::fgE

. !

card.

. →

,

a

FiXED WINDOWS, MICRO BATCH

PCollection<KV<String, Integer>> output = input .apply(Window.into(FixedWindows.of(2, MINUTES)) .trigger(Repeat(AtWatermark()))) .accumulating())

• 12:00
• 12:02

5

• 12:02
• 12:04

# 14

• 12=04-12--006

3

• f

;D:

"-00

" o.

monk

iii:

in

:÷

a:*

A

M t

a

SUMMARY/LESSONS

Design for unbounded data: Don’t rely on completeness Be flexible, diverse use cases

• Billing
• Recommendation
• Anomaly detection

Windowing, Trigger API to simplify programming on unbounded data

DISCUSSION

https://forms.gle/jwHjTBbR49vyQASq6

Fixed

windows

streaming

a)

window fires

every time watermark pass

Assume

watermark

is

given

⇒ worse

latency

⇒ fewer

• utputs

Eat

EA

X

T

⇒

Micro

batch

nie

• D

partial

• 1

. .

rum

entry

streaming ⇒ *.

IEEE

'm

event - ts to

system Apache Kafka

Pub

• Sub

Ingest

proc - t

time

↳

update

query

persist

disk

Consider you are implementing a micro-batch streaming API on top of Apache

• Spark. What are some of the bottlenecks/challenges you might have in building

such a system?

NEXT STEPS

Next class: Naiad Course project proposal peer feedback