SLIDE 1
Fault Tolerance at Speed
Todd L. Montgomery @toddlmontgomery
StoneTor
Fault Tolerance at Speed Todd L. Montgomery @toddlmontgomery About - - PowerPoint PPT Presentation
Fault Tolerance at Speed Todd L. Montgomery @toddlmontgomery About - - PowerPoint PPT Presentation
StoneTor Fault Tolerance at Speed Todd L. Montgomery @toddlmontgomery About me What type of Fault Tolerance? What is Clustering? Why Aeron? Design for Speeding Up? What type of Fault Tolerance? What is Clustering? Why Aeron? Design
SLIDE 2
SLIDE 3
What type of Fault Tolerance? What is Clustering? Why Aeron? Design for Speeding Up?
SLIDE 4
What type of Fault Tolerance? What is Clustering? Why Aeron? Design for Speeding Up? Efficiency
SLIDE 5
https://www.forbes.com/sites/forbestechcouncil/2017/12/15/why-energy-is-a-big-and-rapidly-growing-problem-for-data-centers/#344456665a30 https://www.datacenterdynamics.com/opinions/power-consumption-data-centers-global-problem/ https://www.nature.com/articles/d41586-018-06610-y
SLIDE 6
We seem to assume efficiency/security/quality/etc. is a “special” characteristic added … later… if at all
SLIDE 7
Fault Tolerance
SLIDE 8
Service Client
SLIDE 9
Service Client
SLIDE 10
Service Client Service Service
SLIDE 11
Service Client Service Service Client Client
SLIDE 12
Service Client Service Service Client Client
S t a t e
SLIDE 13
SLIDE 14
Service Service Service
State “Storage”
SLIDE 15
Service Client Service Service Client Client
S t a t e
SLIDE 16
Fault Tolerance of State
SLIDE 17
Service Service Service State
Partition Replication
SLIDE 18
Contiguous Log with Snapshot & Replay
SLIDE 19
1 2 3 4 5 6 X
…
SLIDE 20
1 State 2 3 4 5 6 X
…
SLIDE 21
1 State 2 3 4 5 6 X
… Snapshot
SLIDE 22
1 State 2 3 4 5 6 X
… Snapshot
5 6 X
…
Snapshot State
SLIDE 23
Clustered Services
SLIDE 24
Service Service Service
SLIDE 25
Service Service Service Log Archive Log Archive Log Archive
SLIDE 26
Replicated State Machines
https://en.wikipedia.org/wiki/State_machine_replication
SLIDE 27
Each Replicated Service Same event log Same input ordering Log replicated locally
Replicated State Machines
SLIDE 28
Checkpoints / Snapshots Event in the log “Rolling” up previous log events
Replicated State Machines
SLIDE 29
When should a service “consume” (or process) a log event?
SLIDE 30
Service Service Service Archive Archive Archive
1 2 3 4 5 6 1 2 3 4 5 6 7 1 2
SLIDE 31
Once processed, Event can not be altered Only process once event is stable
SLIDE 32
Raft Consensus Event must be recorded at majority
- f Replicas before being consumed
by any Replica
Replicated State Machines
https://raft.github.io/
SLIDE 33
Service Service Service Archive Archive Archive
1 2 3 4 5 6 1 2 3 4 5 6 7 1 2
SLIDE 34
Service Service Service Archive Archive Archive
1 2 3 4 5 6 1 2 3 4 5 6 7 1 2
SLIDE 35
Strong Leader Elected member of the Cluster Orders Input Disseminates Consensus
Raft
SLIDE 36
Service Service Service Archive Archive Archive Consensus Consensus Consensus
SLIDE 37
Raft is An algorithm with formal verification
Replicated State Machines
SLIDE 38
Raft is not A specification Nor A complete system
Replicated State Machines
SLIDE 39
More than Raft Leader timestamps events Async, not RPC-based Timers
The Real World
SLIDE 40
Service Service Service Archive Archive Archive Consensus Consensus Consensus
*Leader
Client
SLIDE 41
Benefits
SLIDE 42
Determinism Log is immutable Log can be played, stopped, & replayed Each event is timestamped Services restarted from snapshot & log
Benefits
SLIDE 43
What Can You Do?
SLIDE 44
Distributed Key/Value Store Distributed Timers Distributed Locks
SLIDE 45
Matching Engines Order Management Market Surveillance P&L, Risk, …
Finance
SLIDE 46
Venue Ticketing / Reservations Auctions
Beyond
Hint - a contended database is a good indicator
SLIDE 47
Why Aeron?
SLIDE 48
Efficient reliable UDP unicast, UDP multicast, and IPC message transport Java, C/C++, C#, Go
Aeron
https://github.com/real-logic/Aeron
SLIDE 49
And a little bit more… Very fast Archival & Replay
Aeron
https://github.com/real-logic/Aeron
SLIDE 50
The “Efficient” bit…
SLIDE 51
All communications Aeron publications & subscriptions Aeron archival & replay Aeron shared counters
SLIDE 52
Consensus based on Aeron stream position
SLIDE 53
Batching Critical to efficient operation Optimizing pipelined throughput
SLIDE 54
Flow Control Critical to correct operation
SLIDE 55
Design for Efficiency?
SLIDE 56
Cache Hit/Miss Ratios Branch Prediction Allocation Rates Garbage Collection Inlining Optimizations
SLIDE 57
Not… Yet…
SLIDE 58
Ownership, Dependency, & Coupling Complexity Layers of Abstraction (ain’t free) Resource Management
SLIDE 59
Closer… But…
- Still. Not. Yet.
SLIDE 60
"AmdahlsLaw" by Daniels220 at English Wikipedia - Own work based on: File:AmdahlsLaw.png. Licensed under CC BY-SA 3.0 via Wikimedia Commons
SLIDE 61
Universal Scalability Law
2 4 6 8 10 12 14 16 18 20 1 2 4 8 16 32 64 128 256 512 1024
Speedup Processors
Amdahl USL
SLIDE 62
Breakdown Interactions Fundamental Sequential Operations
SLIDE 63
Ingress Message, Sequence, Disseminate
Client Follower X Leader
Ingress
Follower Y
Log (multicast or serial unicast) Member Status Log Event Log Event
SLIDE 64
Followers Append
Client Follower X Leader
Ingress
Follower Y
Log (multicast or serial unicast) Member Status Append Position Append Position
SLIDE 65
Commit Message
Client Follower X Leader
Ingress
Follower Y
Log (multicast or serial unicast) Member Status Commit Position Commit Position
SLIDE 66
Breakdown Interactions Pipeline-able Operation & Batching
SLIDE 67
Follower Leader
Log (multicast or serial unicast) Member Status Commit Position @4096 Append Position @6912 Log Event @8192
Stream Positions
Archive Position @8096 Archive Position @7168
Store locally asynchronous to Position processing by Consensus, & Log processing by Service Batching: Log, Appends, Commits
SLIDE 68
Doesn’t this Complicate Recovery?
SLIDE 69
Follower
Recovery Positions
Archive Position @8096 Archive Position @7168
A synchronous system doesn’t make this complexity go away! Election still needs to assert state of the cluster & locally catch-up
Follower Follower
Archive Position @7584 Commit Position @4096 Commit Position @4064 Commit Position @4032 Service Position @4096 Service Position @4064 Service Position @3776
SLIDE 70
Limitations of Efficiency Throughput & Latency
SLIDE 71
Client Followers Leader
Ingress Log (multicast or serial unicast) Member Status Commit Position Append Position Log Event
Client to Service A: 0.5 RTT Client to Service Ox: 1 RTT Client to Service A (on Commit): 1.5 RTT Client to Service Ox (on Commit): 2 RTT
Constant Delay Network
Service A Service Ox
Round-Trip Time (RTT)
SLIDE 72
Client to Service A: 50ns Client to Service Ox: 100ns Client to Service A (on Commit): 150ns Client to Service Ox (on Commit): 200ns
Limits from Constant Delay
Shared Memory RTT <100ns
Client to Service A: 50us Client to Service Ox: 100us Client to Service A (on Commit): 150us Client to Service Ox (on Commit): 200us
DC RTT <100us
Client to Service A: 5us Client to Service Ox: 10us Client to Service A (on Commit): 15us Client to Service Ox (on Commit): 20us
Rack (Kernel Bypass) RTT <10us
SLIDE 73
Measured Latency at Throughput
RTT (us) 75 150 225 300 Percentile Min 0.50 0.90 0.99 0.9999 0.999999 Max
100K msgs/sec 200K msgs/sec
Intel Xeon Gold 5118 (2.30GHz, 12 cores) 32GB DDR4 2400 MHz ECC RAM Intel Optane SSD 900P Series 480GB SolarFlare X2522-PLUS 10GbE NIC All servers are connected to an Arista 7150S CentOS Linux 7.7, kernel 4.4.195-1.el7.elrepo.x86_64 tuned for low-latency workload. Courtesy Mark Price Single client session, bursts of 20x 200B messages, 3-node cluster, Service(s) echo(es) the payload back.
SLIDE 74
Takeways Efficiency is part of design Power of a timestamped, replicated log Replicated State Machines
SLIDE 75
Current Status Aeron Archiving - fully supported Aeron Clustering - pre-release
Sponsored by
https://weareadaptive.com/
SLIDE 76
SLIDE 77
Aeron: https://github.com/real-logic/Aeron Twitter: @toddlmontgomery
Thank You!
Questions?
StoneTor