Intro to Redis Streams IMCSUMMIT - NOVEMBER 2019 | DAVE NIELSEN - - PowerPoint PPT Presentation

▶
intro to redis streams
SMART_READER_LITE
LIVE PREVIEW

Intro to Redis Streams IMCSUMMIT - NOVEMBER 2019 | DAVE NIELSEN - - PowerPoint PPT Presentation

Dec 19, 2022 235 likes •681 views

Intro to Redis Streams IMCSUMMIT - NOVEMBER 2019 | DAVE NIELSEN What is a data stream? What are the typical challenges you face while managing data streams? What is Redis Stream? How Redis Stream addresses the challenges? How

slide-1
SLIDE 1

Intro to Redis Streams

IMCSUMMIT - NOVEMBER 2019 | DAVE NIELSEN

slide-2
SLIDE 2

2

  • What is a data stream?
  • What are the typical challenges you

face while managing data streams?

  • What is Redis Stream?
  • How Redis Stream addresses the

challenges?

  • How to use Redis Streams?
slide-3
SLIDE 3

Who We Are

3

Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source home and commercial provider of Redis Enterprise technology, platform, products & services.

slide-4
SLIDE 4

Redis Top Differentiators

4

Simplicity Extensibility Performance

NoSQL Benchmark

1

Redis Data Structures

2 3

Redis Modules

Lists Hashes Bitmaps Strings Bit field Streams Hyperloglog Sorted Sets Sets Geospatial Indexes

slide-5
SLIDE 5

5

Data Stream Use Cases

slide-6
SLIDE 6

6

What is a Data Stream?

slide-7
SLIDE 7

Simplest form of a Data Stream

7

Consumer Producer

slide-8
SLIDE 8

But Often You Have M Producers and N Consumers

8

Producer 2 Producer m Producer 1 Producer 3 Consumer 1 Consumer n Consumer 2 Consumer 3

slide-9
SLIDE 9

It’s a bit more complex then that

9

slide-10
SLIDE 10

Data Stream Components

10

Ingest Stream Message Broker

  • r

Stream Processor Output Stream Analytics Collect large volume of data arriving in high velocity Store the data temporarily Transform incoming data into one or more formats as required by the consumers Push the data to the consumers Allow backward looking queries to pull data Store the data and perform analytical

  • perations such as

predictive analytics, ML training, ML classification and categorization, recommendations, pattern matching, etc.

IoT Activity logs Messages

slide-11
SLIDE 11

11

slide-12
SLIDE 12

Redis Supports All the Functions of a Data Stream

12

Ingest Stream Message Broker

  • r

Stream Processor Output Stream Analytics

Redis

  • Streams
  • Pub/Sub
  • Lists
  • Streams
  • Pub/Sub
  • Lists
  • Streams
  • Sorted Sets

+

  • Spark Connector
  • ODBC/JDBC Connector
  • Sorted Sets
  • Sets
  • Geo
  • Bitfield

+

  • RedisGraph
  • RediSearch
slide-13
SLIDE 13

13

slide-14
SLIDE 14

Understanding Redis Streams

14

slide-15
SLIDE 15

Let’s look at t the the ch challen llenges es fir irst

15

slide-16
SLIDE 16

How to Support a Variety of Consumers

16

Analytics Data Backup

Consumers

Producer Messaging

Real-time Real-time or Periodic Lookup Periodic Read

slide-17
SLIDE 17

The Catch Up Problem

17

Image Processor Producer Consumption Rate: 100/sec Arrival Rate: 500/sec

Redis Stream

Fast Slow Backlog

slide-18
SLIDE 18

How to Address The Catch Up Problem?

18

Producer Image Processor Arrival Rate: 500/sec Consumption Rate: 500/sec Image Processor Image Processor Image Processor Image Processor

Redis Stream

Scale Out New Problem: Mutual Exclusivity

slide-19
SLIDE 19

Data Recovery from Consumer Failures

19

Producer Image Processor Arrival Rate: 500/sec Consumption Rate: 500/sec Image Processor Image Processor Image Processor Image Processor

Redis Stream

Scale Out Yet Another Problem: Failure Scenarios

slide-20
SLIDE 20

Connection Failures

20

Producer Image Processor Image Processor Image Processor Image Processor Image Processor

Redis Stream

The solution must be resilient to failures

slide-21
SLIDE 21

Here comes Redis Streams

21

slide-22
SLIDE 22

What is Redis Streams?

22

Pub/Sub Lists Sorted Sets It is like Pub/Sub, but with persistence It is like Lists, but decouples producers and consumers It is like Sorted Sets, but asynchronous

+

  • Lifecycle management of streaming data
  • Built-in support for timeseries data
  • A rich choice of options to the consumers to read streaming and static data
  • Super fast lookback queries powered by radix trees
  • Automatic eviction of data based on the upper limit
slide-23
SLIDE 23
  • 1. It enables asynchronous data exchange between producers

and consumers

23

Messaging Producer

Consumer

slide-24
SLIDE 24

24

Analytics Data Backup

Consumers

Producer Messaging

  • 2. You can consume data in real-time as it arrives or lookup

historical data

slide-25
SLIDE 25

25

Producer Image Processor Arrival Rate: 500/sec Consumption Rate: 500/sec Image Processor Image Processor Image Processor Image Processor

Redis Stream

  • 3. With consumer groups, you can scale out and avoid backlogs
slide-26
SLIDE 26

26

Classifier 1 Classifier 2 Classifier n

Consumer Group

XREADGROUP XREAD

Consumers

Producer 2 Producer m Producer 1 Producer 3 XADD XACK

Deep Learning-based Classification

Analytics Data Backup Messaging

  • 4. Simplify data collection, processing and distribution to

support complex scenarios

slide-27
SLIDE 27

Quick Reference Guide https://redislabs.com/docs/getting-started-redis-streams/

Redis Streams Commands

27

slide-28
SLIDE 28

Commands: XADD, XREAD

28

Asynchronous producer-consumer message transfer

Messaging Producer

Consumer

XADD XREAD XADD mystream * name Anna XADD mystream * name Bert XADD mystream * name Cathy XREAD COUNT 100 STREAMS mystream 0 XREAD BLOCK 10000 STREAMS mystream $

slide-29
SLIDE 29

Commands: XRANGE, XREVRANGE

29

Lookup historical data

Analytics Data Backup

Consumers

Producer Messaging

XADD XREAD XRANGE XREVRANGE XRANGE mystream 1518951123450-0 1518951123460-0 COUNT 10 XRANGE mystream - + COUNT 10

slide-30
SLIDE 30

Scaling out using consumer groups

30

Producer Image Processor Arrival Rate: 500/sec Consumption Rate: 500/sec Image Processor Image Processor Image Processor

Redis Stream

slide-31
SLIDE 31

Scaling out using consumer groups

31

Producer Image Processor Arrival Rate: 500/sec Consumption Rate: 500/sec Image Processor Image Processor Image Processor

Redis Stream Consumer Group

Consumer 1 Consumer 2 Consumer 3 Consumer n Unconsumed List Consumers

XADD XGROUP XREADGROUP XACK XPENDING XCLAIM

slide-32
SLIDE 32

Command: XGROUP CREATE

32

Create a consumer group

Unconsumed List

mygroup

Alice Bob

e d c b a

mystream

e d c b a

App A App B

XGROUP CREATE XGROUP CREATE mystream mygroup $ MKSTREAM

slide-33
SLIDE 33

Command: XREADGROUP

33

Read the data

Unconsumed List

mygroup

Alice Bob App A App B

a e d c b

XREADGROUP XREADGROUP GROUP mygroup COUNT 2 Alice STREAMS mystream > XREADGROUP GROUP mygroup COUNT 2 Bob STREAMS mystream >

slide-34
SLIDE 34

Command: XACK

34

Consumers acknowledge that they consumed the data

Unconsumed List

mygroup

Alice Bob App A App B

a c b

XACK XACK mystream mygroup 1526569411111-0 1526569411112-0

slide-35
SLIDE 35

Command: XREADGROUP

35

Repeat the cycle

Unconsumed List

mygroup

Alice Bob App A App B

a c b

XREADGROUP XREADGROUP GROUP mygroup COUNT 2 Alice STREAMS mystream >

slide-36
SLIDE 36

How to claim the data from a consumer that failed while processing the data?

36

Unconsumed List

mygroup

Alice Bob App A App B

c b

Unconsumed List

mygroup

Alice Bob App A App B

c b

XCLAIM

slide-37
SLIDE 37

Commands: XPENDING, XCLAIM

37

Claim pending data from other consumers

Unconsumed List

mygroup

Alice Bob App A App B

c b

XPENDING mystream mygroup - + 10 Bob XCLAIM mystream mygroup Alice 0 1526569411113-0 1526569411114-0

slide-38
SLIDE 38

Now is the time to

38

slide-39
SLIDE 39

Sample Use Case: Social Media Analytics

39

a b x Language classifier XREADGROUP Real-time reports XREAD

Sentiment analyzers

Consumer is x times slower than the producer Location classifier Influencer classifier

Redis Streams

slide-40
SLIDE 40

Demo

40

Redis Streams Twitter Ingest Stream Influencer Classifier https://github.com/redislabsdemo/IngestRedisStreams

slide-41
SLIDE 41

Java Jedis

https://github.com/xetorthio/jedis

Lettuce

https://lettuce.io/

Python redis-py

https://github.com/andymccurdy/redis-py

JavaScript ioredis

https://github.com/luin/ioredis

node_redis

https://github.com/NodeRedis/node_redis

.NET StackExchange.Redis

https://github.com/StackExchange/StackExchange.Redis

Go redigo

https://github.com/gomodule/redigo

go-redis

https://github.com/go-redis/redis

C Hiredis

https://github.com/redis/hiredis

PHP predis

https://github.com/nrk/predis

phpredis

https://github.com/phpredis/phpredis

Ruby redis-rb

https://github.com/redis/redis-rb

Redis Clients that Support Redis Streams

41

slide-42
SLIDE 42

Questions

?

?

?

?

?

?

?

? ?

? ?

slide-43
SLIDE 43

Thank you!

redislabs.com

43