redis cluster or: distributed systems are hard Jan-Erik Rediger - - PowerPoint PPT Presentation

▶
redis cluster
SMART_READER_LITE
LIVE PREVIEW

redis cluster or: distributed systems are hard Jan-Erik Rediger - - PowerPoint PPT Presentation

Sep 27, 2022 346 likes •931 views

redis cluster or: distributed systems are hard Jan-Erik Rediger 28. Mai 2015 Hi, Im Jan-Erik Student of Computer Science, RWTH fjrst Redis-related project: 2010 (an Erlang client) Maintainer of try.redis.io hiredis

slide-1
SLIDE 1

redis cluster

  • r: distributed systems are hard

Jan-Erik Rediger

  • 28. Mai 2015
slide-2
SLIDE 2

who am i?

Hi, I’m Jan-Erik

฀ Student of Computer Science, RWTH ฀ fjrst Redis-related project: 2010 (an Erlang client) ฀ Maintainer of ฀ try.redis.io ฀ hiredis ฀ hiredis-rb, hiredis-py, hiredis-node

slide-3
SLIDE 3

redis

slide-4
SLIDE 4

redis

Redis is an open source, BSD licensed, advanced key-value cache and store. It is

  • ften referred to as a data structure server

since keys can contain strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs.

slide-5
SLIDE 5

redis

SET redis rocks HSET meetup.42 name PHPUGDUS SADD meetups-in-dus 42:PHPUGDUS ZADD meetups-in-nrw 20150628 PHPUGDUS LPUSH trivago "meetup host"

slide-6
SLIDE 6

limits

Must fjt into RAM No redundancy Single-threaded

slide-7
SLIDE 7

limits

Must fjt into RAM No redundancy Single-threaded

slide-8
SLIDE 8

limits

Must fjt into RAM No redundancy Single-threaded

slide-9
SLIDE 9
  • vercoming limits

฀ Sharding ฀ Split data set across nodes ฀ e.g. via Twemproxy or Codis ฀ Replication ฀ Failover for HA ฀ Sentinel or another system ฀ Scale reads to more instances

slide-10
SLIDE 10
  • vercoming limits

฀ Sharding ฀ Split data set across nodes ฀ e.g. via Twemproxy or Codis ฀ Replication ฀ Failover for HA ฀ Sentinel or another system ฀ Scale reads to more instances

slide-11
SLIDE 11
slide-12
SLIDE 12

redis cluster

slide-13
SLIDE 13

redis cluster

฀ It’s still Redis ฀ must be fast ฀ must scale ฀ must be simple to use ฀ must give some guarantees ฀ but it’s a Distributed System

slide-14
SLIDE 14

redis cluster

฀ It’s still Redis ฀ must be fast ฀ must scale ฀ must be simple to use ฀ must give some guarantees ฀ but it’s a Distributed System

slide-15
SLIDE 15

redis cluster

฀ It’s still Redis ฀ must be fast ฀ must scale ฀ must be simple to use ฀ must give some guarantees ฀ but it’s a Distributed System

slide-16
SLIDE 16

redis cluster

฀ It’s still Redis ฀ must be fast ฀ must scale ฀ must be simple to use ฀ must give some guarantees ฀ but it’s a Distributed System

slide-17
SLIDE 17

redis cluster

฀ It’s still Redis ฀ must be fast ฀ must scale ฀ must be simple to use ฀ must give some guarantees ฀ but it’s a Distributed System

slide-18
SLIDE 18

redis cluster

฀ It’s still Redis ฀ must be fast ฀ must scale ฀ must be simple to use ฀ must give some guarantees ฀ but it’s a Distributed System

slide-19
SLIDE 19

cap - pick two

slide-20
SLIDE 20

cp - consistent & partition tolerant

฀ Higher Latency ฀ Wait for majority of ACKs

slide-21
SLIDE 21

ap - available & partition tolerant

฀ Eventual Consistency ฀ might require merges

slide-22
SLIDE 22

so what is it?

slide-23
SLIDE 23

sharding + replication

slide-24
SLIDE 24

replication: asynchronous

slide-25
SLIDE 25

full mesh

฀ Heartbeats ฀ Gossip ฀ Failover ฀ Confjg update

slide-26
SLIDE 26

redirections

slide-27
SLIDE 27

slot configuration

฀ Key space split into 16384 slots ฀ Every node serves a subset of the slots ฀ Every node knows the node฀slot mapping

slide-28
SLIDE 28

single-key operations

> GET redis 1) "rocks" > GET kjdopiqh (error) MOVED 12182 127.0.0.1:7002

slide-29
SLIDE 29

multi-key operations

฀ All keys in same slot: it’s fjne > MGET foo10 foo5406 1) "hello" 1) "world"

slide-30
SLIDE 30

multi-key operations

฀ Keys in difgerent slots: sorry, not possible > MGET foo bar (error) CROSSSLOT Keys in request don't hash to the same slot

slide-31
SLIDE 31

multi-key operations with hash tags

฀ Ensure keys map to the same slot > MGET {foo}.meetup {foo}.city 1) "PHPUG" 2) "Düsseldorf"

slide-32
SLIDE 32

failure detection

slide-33
SLIDE 33

node times out: pfail

slide-34
SLIDE 34

all remaining nodes see pfail

slide-35
SLIDE 35

pfail → fail

slide-36
SLIDE 36

force fail, trigger failover

slide-37
SLIDE 37

failover

slide-38
SLIDE 38

actual failover

slide-39
SLIDE 39

actual failover

slide-40
SLIDE 40

is it consistent?

Eventually…

฀ Last Failover wins ฀ Mechanisms to avoid unbound data loss

slide-41
SLIDE 41

is it consistent?

Eventually…

฀ Last Failover wins ฀ Mechanisms to avoid unbound data loss

slide-42
SLIDE 42

is it consistent?

Eventually…

฀ Last Failover wins ฀ Mechanisms to avoid unbound data loss

slide-43
SLIDE 43

failure modes

slide-44
SLIDE 44

failure: crashes

slide-45
SLIDE 45

failure: network split

slide-46
SLIDE 46

failure: network split

slide-47
SLIDE 47

creating a cluster

slide-48
SLIDE 48

configuration

port 7001 cluster-enabled yes cluster-config-file nodes.conf cluster-node-timeout 5000

slide-49
SLIDE 49

let them meet

CLUSTER MEET 127.0.0.1 7001

* on every node ;)

slide-50
SLIDE 50

more convenient

./redis-trib.rb create --replicas 1 \ 127.0.0.1:7000 127.0.0.1:7001 \ 127.0.0.1:7002 127.0.0.1:7003 \ 127.0.0.1:7004 127.0.0.1:7005

slide-51
SLIDE 51

more commands you want to know

CLUSTER ADDSLOTS 42 43 44 CLUSTER SETSLOT 42 NODE ff7831dfe CLUSTER SETSLOT 42 MIGRATING ff7831dfe CLUSTER SETSLOT 42 IMPORTING ff7831dfe

slide-52
SLIDE 52

and some more

CLUSTER NODES CLUSTER FAILOVER CLUSTER REPLICATE ff7831dfe

slide-53
SLIDE 53

all cluster commands

redis.io/commands#cluster

slide-54
SLIDE 54

nodes.conf

ff7831dfe7fc73f741d5c4663a8020e654f88f22 127.0.0.1:7001 myself,master - 0 0 1 connected 0-5460 94f5710dab37058784824dc9d2ddb27a693f1336 127.0.0.1:7013 slave a46ff091f49bd28149594dfd2272ebb3aedcdd59 0 1415873239710 6 connected vars currentEpoch 6 lastVoteEpoch 0

slide-55
SLIDE 55

available clients

฀ Ruby: redic-cluster ฀ Ruby: redis-rb-cluster ฀ Python: redis-py-cluster ฀ PHP: Predis ฀ Java: Jedis ฀ JavaScript: thunk-redis ฀ .NET: StackExchange.Redis

slide-56
SLIDE 56

sources

฀ redis.io - offjcial site & documentation ฀ cluster-spec - offjcial spec docu ฀ cluster-tutorial - a tutorial ฀ mattsta/redis-cluster-playground - easy to play around with cluster

slide-57
SLIDE 57
slide-58
SLIDE 58

the end

Get the slides here: http://slidr.io/badboy/redis-cluster Reach me on Twitter: @badboy_ Jan-Erik Rediger - 28. Mai 2015 - PHPUGDUS