Data At Rest Data In Motion! A Lambda Architecture Overview When - PowerPoint PPT Presentation

Data At Rest … Data In Motion! A Lambda Architecture Overview

When Things Go Wrong http://xkcd.com/327/ Fault T olerance !!!!

Fault T olerance Developer Software Hardware

Data Collection Three T ypes Of Data Streams Structured (Databases ...) Semi Structured (JSON, XML, XAML ...) UnStructured (Blogs, E-Mails, Log Files ...)

Lambda Architecture T o The Rescue !!

Lambda Architecture — Requirements Fault-tolerant against both hardware failures and human errors Support variety of use cases that include low latency querying as well as updates Linear scale-out capabilities Extensible, so that the system is manageable and can accommodate newer features easily

Lambda Architecture IMMUTABLE PRECOMPUTE MASTER DATA VIEWS BATCH RECOMPUTE View 1 View 2 View N NEW DATA QUERY MERGE STREAM View 1 View 2 View N PROCESS INCREMENT STREAM VIEWS REAL-TIME INCREMENT

Lambda Architecture IMMUTABLE PRECOMPUTE BATCH LAYER MASTER DATA VIEWS BATCH RECOMPUTE SERVING LAYER View 1 View 2 View N NEW DATA QUERY MERGE STREAM View 1 View 2 View N PROCESS INCREMENT SPEED LAYER STREAM VIEWS REAL-TIME INCREMENT

Lambda Architecture - Layers Batch Layer Managing the master data set, an immutable, append only set of raw data. Pre computing arbitrary query functions, called batch views.

Lambda Architecture - Layers Serving Layer Indexes batch views so that they can be queried in ad hoc with low latency. Merges and reconciles batch and real time views.

Lambda Architecture - Layers Speed Layer Accommodates all requests that are subject to low latency requirements. Using fast and incremental algorithms, deals with recent data only.

Lambda Architecture - Reconciliation Data absorbed into Batch Views Not yet absorbed Time

Lambda Architecture - Reconciliation Now Just a few hours of data Data absorbed into Batch Views Not yet absorbed Time

Lambda Architecture - Immutable Data + Views Times Airpor Flight Action tamp t 2015- DUB EL123 take- 01- off 01T10: 00:0 2015- HEL SA45 take- 01- off 01T10: 05:0 2015- AMS BA99 take- 01- off 01T10: Immutable Master Dataset 07:0 2015- LHR LH17 landin 01- g

Lambda Architecture - Immutable Data + Views Timestamp Airport Flight Action air borne: 2307 Map Reduce 2015-01- DUB EL123 take-off 01T10:00:0 air borne per airline: Map Reduce 2015-01- HEL SA45 take-off 01T10:05:0 Airline 2015-01- AMS BA99 take-off Map Reduce airport load: 01T10:07:0 Airport Planes 2015-01- LHR LH17 landing 01T10:09:0 2015-01- CDG AF03 landing SAS 01T10:10:0 Immutable Master Dataset 2015-01- FCO AZ501 take-off AMS 44 01T10:11:0 BA LHR 69

Lambda Architecture - Implementation IMMUTABLE PRECOMPUTE MASTER DATA VIEWS BATCH RECOMPUTE View 1 View 2 View N NEW DATA QUERY MERGE STREAM View 1 View 2 View N PROCESS INCREMENT STREAM VIEWS REAL-TIME INCREMENT

Lambda Architecture - Implementation Hadoop HDFS Apache Hive BATCH RECOMPUTE HBase HBase HBase NEW DATA ● Spark SQL ● R MERGE STREAM ● Presto Storm Storm Storm Bolt Bolt Bolt Apache Kafka Apache Spark REAL-TIME INCREMENT