SLIDE 1
From Distributed Logs to Database Replication
- Dr. Samuel Benz
From Distributed Logs to Database Replication Dr. Samuel Benz How - - PowerPoint PPT Presentation
From Distributed Logs to Database Replication Dr. Samuel Benz How - - PowerPoint PPT Presentation
From Distributed Logs to Database Replication Dr. Samuel Benz How to achieve scalability, fault tolerance and consistency in distributed systems? Distributed applications in theory. . . . . . in practice Introduction Reliable and Scalable
SLIDE 2
SLIDE 3
Distributed applications in theory. . .
SLIDE 4
. . . in practice
SLIDE 5
Introduction Reliable and Scalable Services Distributed Consensus Distributed Log Conclusion
Why do we see such architectures?
Distributed Stateful Components state (vs. stateless) shared > 1 client (isolation) mutable > 0 writer (concurrency) distributed > 1 DB (consistency) geographically > 50km (latency)
5
SLIDE 6
Introduction Reliable and Scalable Services Distributed Consensus Distributed Log Conclusion
Reliable and Scalable Stateful Services
Problem
1 Scalability:
- Size: Internet scale services
- Location: Access latency
- Administration: Multiple organizational units
2 Fault-Tolerance
Solution
1 Distributed Data: Replication 2 Distributed Computing: Coordination
6
SLIDE 7
Introduction Reliable and Scalable Services Distributed Consensus Distributed Log Conclusion
Different Types of Replication
7
SLIDE 8
Introduction Reliable and Scalable Services Distributed Consensus Distributed Log Conclusion
State Machine Replication → Fault-tolerance
8
SLIDE 9
Introduction Reliable and Scalable Services Distributed Consensus Distributed Log Conclusion
State Machine Replication → Consistency
9
SLIDE 10
Introduction Reliable and Scalable Services Distributed Consensus Distributed Log Conclusion
Partitioning → Scalability
10
SLIDE 11
Introduction Reliable and Scalable Services Distributed Consensus Distributed Log Conclusion
Consistent Partitioning
1 The system ensures strong consistency within partitions and
”best-effort” across partitions.
2 The system ensures strong consistency using 2PC across
partitions.
3 The system orders commands before executing them or checks
their order after executing the commands (Atomic Multicast).
11
SLIDE 12
Introduction Reliable and Scalable Services Distributed Consensus Distributed Log Conclusion
Simple Coordination Problem
A B
12
SLIDE 13
Introduction Reliable and Scalable Services Distributed Consensus Distributed Log Conclusion
Consensus Problem
Fundamental Result
No algorithm can solve consensus in an asynchronous system despite a single crash.
FLP impossibility result (after Fischer, Lynch, and Paterson, 1985) 13
SLIDE 14
Introduction Reliable and Scalable Services Distributed Consensus Distributed Log Conclusion
Consensus and Atomic Broadcast
In a crash-stop failure model consensus is defined as follows:
1 Termination: Every correct process eventually decides. 2 Agreement: No two correct processes decide differently. 3 Uniform integrity: Every process decides at most once. 4 Uniform validity: If a process decides v, then v was
proposed by some process. Additionally Atomic Broadcast:
5 Total order: If two correct processes p and q deliver two
messages m and m′, then p delivers m before m′ if and only if q delivers m before m′.
[Chandra et al. Unreliable failure detectors for reliable distributed systems. 1996.] 14
SLIDE 15
Introduction Reliable and Scalable Services Distributed Consensus Distributed Log Conclusion
Consensus and Atomic Broadcast
In a crash-stop failure model consensus is defined as follows:
1 Termination: Every correct process eventually decides. 2 Agreement: No two correct processes decide differently. 3 Uniform integrity: Every process decides at most once. 4 Uniform validity: If a process decides v, then v was
proposed by some process. Additionally Atomic Broadcast:
5 Total order: If two correct processes p and q deliver two
messages m and m′, then p delivers m before m′ if and only if q delivers m before m′.
[Chandra et al. Unreliable failure detectors for reliable distributed systems. 1996.] 14
SLIDE 16
Introduction Reliable and Scalable Services Distributed Consensus Distributed Log Conclusion 15
SLIDE 17
Introduction Reliable and Scalable Services Distributed Consensus Distributed Log Conclusion
Distributed Log
16
SLIDE 18
Introduction Reliable and Scalable Services Distributed Consensus Distributed Log Conclusion
Distributed Transactions
17
SLIDE 19
Introduction Reliable and Scalable Services Distributed Consensus Distributed Log Conclusion
Distributed Data Structures
18
SLIDE 20
Introduction Reliable and Scalable Services Distributed Consensus Distributed Log Conclusion
Kafka Consistency
19
SLIDE 21
Introduction Reliable and Scalable Services Distributed Consensus Distributed Log Conclusion
Kafka Scalability
20
SLIDE 22