Big-Data Processing III (Stream Processing) Prof. Lus Veiga IST / - - PowerPoint PPT Presentation
CNV/CC&V MEIC-A/MEIC-T/METI Computao em Nuvem e Virtualizao Big-Data Processing III (Stream Processing) Prof. Lus Veiga IST / INESC-ID Lisboa https://fenix.tecnico.ulisboa.pt/disciplinas/AVExe7/2019-2020/2-semestre/ LV, JG
IST / INESC-ID Lisboa
https://fenix.tecnico.ulisboa.pt/disciplinas/AVExe7/2019-2020/2-semestre/
LV, JG 2015-20, sources Spark, Flink
2
Spark
overview, RDDs programming Model, Examples RDD operations fault-tolerance, Performance
Spark Streaming
overview, discretized stream processing windows, sliding windows, micro-batching
Flink
overview, windowing tumbling windows, sliding windows, custom windows time-based windows, watermarks state management, versioning, fault tolerance, distributed snapshots, execution semantics
Map Map Map Reduce Reduce
Input Output
4
Map Map Map Reduce Reduce
Input Output
Current popular programming models for
E.g., MapReduce:
5
Acyclic data flow is a powerful abstraction, but
Iterative algorithms (many in machine
Interactive data mining tools (R, Excel,
Spark makes working sets a first-class concept
6
Provide distributed memory abstractions for
Retain the attractive properties of
Fault tolerance (for crashes & stragglers) Data locality Scalability
7
Spark’s combination of data flow with RDDs
General data flow models: MapReduce, Dryad, SQL Specialized models for stateful apps: Pregel (Bulk
Synchronous Processing), HaLoop (iterative MR), Continuous Bulk Processing
Instead of specialized APIs for 1 type of app,
give the users first-class control of the
8
Resilient distributed datasets (RDDs)
Immutable collections partitioned across clusters
that can be rebuilt if a partition is lost.
Created by transforming data in stable storage
using data flow operators
(map, filter, group-by, …)
Can be cached across parallel operations
Parallel operations on RDDs
Reduce, collect, count, save, … 9
Load error messages from a log into
lines = spark.textFile(“hdfs://...”) errors = lines.filter(_.startsWith(“ERROR”)) messages = errors.map(_.split(‘\t’)(2)) cachedMsgs = messages.cache()
Block 1 Block 2 Block 3
Worker Worker Worker Driver
cachedMsgs.filter(_.contains(“foo”)).count cachedMsgs.filter(_.contains(“bar”)).count . . . tasks results
Cache 1 Cache 2 Cache 3
Base RDD Transformed RDD Cached RDD Parallel operation
Result: full-text search of Wikipedia in <1 sec (vs 20 sec for on-disk data)
10
An RDD is an immutable, partitioned, logical
needs not be materialized, but rather contains enough information to allow
rebuilding a dataset from stable storage
Partitioning can be based on a key in each
Built using bulk transformations on other RDDs Can be cached for future reuse
11
Transformations (define a new RDD) map filter sample union groupByKey reduceByKey join cache … Parallel actions/operations (return a result to driver) reduce collect count countByKey save …
12
RDDs maintain lineage information that can
i.e., track data dependencies in the data flow
Ex: cachedMsgs = textFile(...).filter(_.contains(“error”)) .map(_.split(‘\t’)(2)) .cache()
HdfsRDD
path: hdfs://…
FilteredRDD
func: contains(...)
MappedRDD
func: split(…)
CachedRDD
13
Consistency is easy due to immutability Inexpensive fault tolerance
(log lineage dependency information rather than replicating/checkpointing data)
Locality-aware scheduling of tasks on partitions Despite being restricted
(not as expressive as queries) model seems applicable to a broad variety of
applications
14
Goal: find best line separating two sets of
target random initial line
15
val data = spark.textFile(...).map(readPoint).cache() var w = Vector.random(D) for (i <- 1 to ITERATIONS) { val gradient = data.map(p => (1 / (1 + exp(-p.y*(w dot p.x))) - 1) * p.y * p.x ).reduce(_ + _) w -= gradient } println("Final w: " + w)
16
127 s / iteration first iteration 174 s further iterations 6 s
17
MapReduce data flow can be expressed using
res = data.flatMap(rec => myMapFunc(rec)) .groupByKey() .map((key, vals) => myReduceFunc(key, vals))
res = data.flatMap(rec => myMapFunc(rec)) .reduceByKey(myCombiner) .map((key, val) => myReduceFunc(key, val))
18
val lines = spark.textFile(“hdfs://...”) val counts = lines.flatMap(_.split(“\\s”)) .reduceByKey(_ + _) counts.save(“hdfs://...”)
19
21 Web server Logs Web server Logs Web server Logs HDFS / S3 Periodic (custom) or continuous ingestion into storage Batch job(s) for log analysis Periodic log analysis job Serving layer
E.g., log analysis example using a batch
Latency from log event to serving layer
every 2 hrs Job scheduler (Oozie)
22
Web server Web server Web server High throughput publish/subscribe bus Serving layer Stream processors allow to analyze events
Forward events immediately to pub/sub bus Stream Processor Process events in real time & update serving layer
Run a streaming computation as a series of very small, deterministic batch jobs Spark Spark Streami ng
batches of X seconds live data stream processed results
processes them using RDD operations
23
Run a streaming computation as a series of very small, deterministic batch jobs
Spark Spark Streamin g
batches of X seconds live data stream processed results
latency ~ 1 second (micro- batching)
combination of batch processing and stream processing in the same system
24
val tweets = ssc.twitterStream(<Twitter username>, <Twitter password>) val hashTags = tweets.flatMap (status => getTags(status)) val tagCounts = hashTags.countByValue()
flatMap map reduceByKey flatMap map reduceByKey
…flatMap map reduceByKey
batch @ t+1 batch @ t batch @ t+2
hashTags tweets tagCounts
[(#cat, 10), (#dog, 25), ... ] 25
val tweets = ssc.twitterStream(<Twitter username>, <Twitter password>) val hashTags = tweets.flatMap (status => getTags(status)) val tagCounts = hashTags.window(Minutes(10), Seconds(1)).countByValue()
sliding window
window length sliding interval
26
tagCounts
val tagCounts = hashTags.window(Minutes(10), Seconds(1)).countByValue()
hashTags t-1 t t+1 t+2 t+3 sliding window countByValue count over all the data in the window
27
RDDs remember the sequence (dataflow) of operations that created it from the original fault-tolerant input data
Batches of input data are replicated in memory of multiple worker nodes, therefore fault-tolerant
Data lost due to worker failure, can be recomputed from input data
input data replicated in memory flatMap lost partitions recomputed on
tweets RDD hashTags RDD
28
Apache Flink is an open source stream
Low latency High throughput Stateful Distributed
Developed at the Apache Software
Used in production
30
31
Apache Kafka: reliable message queue/feed broker
Web server
Kafka topic
Stream processor
32
33
Web server Web server Web server High throughput publish/subscrib e bus Servin g layer Options:
Pub/Sub Forward events immediately to pub/sub bus Stream Processor Options:
Dataflow Process events in real time & update serving layer
Low latency High throughput
snapshots
State handling
state
Windowing / Out
Fault tolerance and correctness
semantics
snapshots
34
“Number of visitors in the last 5 minutes per
Web server
Stream processor
// create stream from Kafka source DataStream<LogEvent> stream = env.addSource(new KafkaConsumer()); // group by country DataStream<LogEvent> keyedStream = stream.keyBy(“country“); // window of size 5 minutes keyedStream.timeWindow(Time.minutes(5)) // do operations per window .apply(new CountPerWindowFunction());
35
Kafka Source
Window Operator
S S S W W W
group by country
// window of size 5 minutes keyedStream.timeWindow(Time.minutes(5)); Job plan Parallel execution on the cluster Time
Tumbling windows Sliding windows Count windows: based on number of events Time-based: based on time (event, ingestion) Custom windows with window assigners,
36
37
Stream
Time Event data { “accessTime”: “1457002134”, “userId”: “1337”, “userLocation”: “UK” } Three notions of Time considered in Flink:
Processing time: wall clock time when window is
being built and processed
Ingestion time: time when event entered Flink, at
beginning of dataflow (possible network delays)
Event time: out-of-order handling of time when
event was truly created, e.g., sensor emitting data
Periodically, low-watermarks are sent through system
to indicate true progression of event time bound timestamps of upstream out-of-order events allow avoiding waiting forever for delayed events
38
33 11 28 21 15 9 5 8 System/application can guarantee that no event with time <= 5 will arrive afterwards Window is evaluated when watermarks arrive
39
Operator 3 5 Operators with multiple inputs always forward the lowest watermark. Conservative approach: no events are expected from any input older than lowest watermark
Earlier work on: “MillWheel: Fault-Tolerant Stream Processing at Internet Scale” by T. Akidau et. al.
Stream processor: Flink
Windows can have internal keyed processing state
e.g. counters, max, min of all tuples for each key
Flink automatically backups and restores state State can be larger than the available memory State backends:
(embedded) RocksDB, heap memory, custom
40
Operator with windows (large state) State backend (local) Distributed File System Periodic backup / recovery Web server Kafka
How can we operate such a pipeline 24x7? Losing state (by stopping the system) would
We need a way to store the state
41
Web server
Kafka topic
Stream processor
Savepoint:
Create an addressable copy of a job’s current state.
Restart a job from any savepoint.
42
> flink savepoint <JobID>
HDFS
> hdfs:///flink-savepoints/2 > flink run –s hdfs:///flink-savepoints/2 <jar>
HDFS
How do we ensure results are always correct?
e.g., number of visitors e.g., match of advert clicks and page visits wrong results may mean lost revenue or penalties
Failures should not lead to data loss or
43
Web server Stream processor
Kafka topic
Execution Semantics: at least once: ensure all operators see all
Storm (another stream processor):
Replay stream in failure case (acking of individual
records)
Exactly once: ensure that operators do not
Flink: Distributed Snapshots Spark: Micro-batches of RDDs on batch runtime 44
Lightweight approach of storing the state of
Implemented using barriers flowing through
45
Data Stream
barrier
Before barrier = part of the snapshot After barrier = Not in snapshot
Reference work on: “Lightweight Asynchronous Snapshots for Distributed Dataflows” by Carbone et. al.
46
Data Stream
barrier
Before barrier = part of the snapshot After barrier = Not in snapshot
Reference work on: “Lightweight Asynchronous Snapshots for Distributed Dataflows” by Carbone et. al.
47
Spark
overview, RDDs programming Model, Examples RDD operations fault-tolerance, Performance
Spark Streaming
overview, discretized stream processing windows, sliding windows, micro-batching
Flink
overview, windowing tumbling windows, sliding windows, custom windows time-based windows, watermarks state management, versioning, fault tolerance, distributed snapshots, execution semantics