Swimming in the data river Or, when streaming analytics isnt Gian - - PowerPoint PPT Presentation

Swimming in the data river

Or, when “streaming analytics” isn’t

Gian Merlino

gian@imply.io

Who am I?

Gian Merlino Committer & PMC member on Cofounder at 10 years working on scalable systems

2

Agenda

• From warehouses to rivers
• What can we do with streaming data?
• Streaming analytics
• Enter the Druid
• Do try this at home!

3

Rolling down the river

4

Data warehouses

Tightly coupled architecture with limited flexibility.

5 Data Data Data

Data Sources ETL

Data warehouse

Processing Store and Compute

Analytics Reporting Data mining

Querying

Data lakes

Modern data architectures are more application-centric.

6

• Confidential. Do not redistribute.

Data Data Data

Data Sources MapReduce, Spark Apps ETL

SQL ML/AI TSDB

Data lake Storage

Data rivers

Streaming architectures are true-to-life and enable faster decision cycles.

7

• Confidential. Do not redistribute.

Data Data Data

Data Sources Stream processors Stream hub

Streaming analytics Databases

ETL Storage Apps

Archive to data lake

Streaming data

8

Source: https://www.confluent.io/blog/building-real-time-streaming-etl-pipeline-20-minutes/

Streaming data

9

Source: https://www.confluent.io/blog/building-real-time-streaming-etl-pipeline-20-minutes/ (plus question marks)

? ? ? ? ? ? ?

App or set of requirements? Alerting, charting,

• bservability, APM?

Do search and NoSQL need apps? Analytical or transactional? How does data move between systems?

Streaming data

10

Stream hub App

Direct production

Stream hub library

Streaming data

11

Transactional DB Stream hub App

Change data capture

DB connection Something DB-specific

Streaming data

12

Stream hub

Streaming data pipeline

EDW or data lake

Kafka Connect Kinesis Firehose

Streaming data

13

Microservice communication

Stream hub Service #1 Service #2

Streaming data

14

Streaming data

15

Stream hub

Spark Streaming Flink Storm Samza Kafka Streams Kinesis Data Analytics Azure Stream Analytics Google Cloud Dataflow

?

Streaming data

16

Stream hub

Stream processors

Real-time actions

Alerting, API calls

Streaming data

17

Stream hub

Stream processors

Storage HDFS Cloud storage

Data movement

Streaming data

18

Stream hub Storage HDFS Cloud storage

Data movement + enrichment

Stream processors Stream processors

Continuous query

Streaming data

19

Stream processors

K / V stores HBase Cassandra Redis

Continuous query + write to serving layer

Stream hub

Stream processors

Realtime dashboard

Streaming data

20

K / V stores HBase Cassandra Redis Realtime dashboard

Continuous query + write to serving layer + unemitted state serving

Direct state access

Stream hub

Stream processors Stream processors

Streaming data

21

K / V stores HBase Cassandra Redis

Continuous query + write to serving layer + unemitted state serving

Direct state access

Stream hub

Stream processors Stream processors

Key lookups Short range scans ‘Precomputation’ Continuous query Access unemitted aggregation state, etc Realtime dashboard

22

The problem

23

The problem

24

The problem

• Slice-and-dice for big data streams
• Interactive exploration
• Look under the hood of reports and dashboards
• And we want our data fresh, too

25

Challenges

• Scale: when data is large, we need a lot of servers
• Speed: aiming for sub-second response time
• Complexity: too much fine grain to precompute
• High dimensionality: 10s or 100s of dimensions
• Concurrency: many users and tenants
• Freshness: load from streams

26

27

high performance analytics data store for event-driven data

What is Druid?

• “high performance”: low query latency, high ingest rates
• “analytics”: counting, ranking, groupBy, time trend
• “data store”: the cluster stores a copy of your data
• “event-driven data”: fact data like clickstream, network flows,

user behavior, digital marketing, server metrics, IoT

28

Streaming data

29

Analytical Database (Like Apache Druid!) App

SQL

Stream hub

Stream processors

Continuous load Aggregation + serving layers merged Imply Pivot Apache Superset Looker Your App Here™ Enrichment

Key features

• Column oriented
• High concurrency
• Scalable to 100s of servers, millions of messages/sec
• Continuous, real-time ingest
• Indexes on all dimensions by default
• Query through SQL
• Target query latency sub-second to a few seconds

30

Use cases

• Clickstreams, user behavior
• Digital advertising
• Application performance management
• Network flows
• IoT

31

Powered by Apache Druid

32

Source: http://druid.io/druid-powered.html + many more!

Powered by Apache Druid

“The performance is great ... some of the tables that we have internally in Druid have billions and billions of events in them, and we’re scanning them in under a second.”

33

Source: https://www.infoworld.com/article/2949168/hadoop/yahoo-struts-its-hadoop-stuff.html

From Yahoo:

Architecture

34

Indexer Indexer Indexer

Files

Historical Historical Historical

Streams

Broker Broker Queries

Why this works

• Computers are fast these days
• Indexes help save work and cost
• But don’t be afraid to scan tables — it can be done efficiently

35

36

Integration patterns

Deployment patterns

37

Stream hub Event streams

(millions of events/sec)

• Modern data

architecture

• Centered around

stream hub Enrichment

Deployment patterns

38

Data lake

(Hadoop, S3)

File dumps

(hourly, daily)

• (Slightly less)

modern data architecture

• Centered around

data lake Enrichment

(Spark, Hive)

39

Download

Apache Druid community site (new): https://druid.apache.org/ Apache Druid community site (legacy): http://druid.io/ Imply distribution: https://imply.io/get-started

40

Contribute

41

https://github.com/apache/druid

Stay in touch

42

@druidio

Join the community! http://druid.apache.org/ Follow the Druid project on Twitter!