Workshop: Implementing Distributed Consensus Dan Ldtke Kordian - PowerPoint PPT Presentation

Workshop: Implementing Distributed Consensus Dan Lüdtke Kordian Bruck danrl@google.com picatso@google.com Disclaimer This work is not affiliated with any company (including Google). This talk is the result of a personal education project!

Agenda ● Part I - Hot Potato at Scale ○ Why we need Distributed Consensus ● Part II - Experiments ○ Introduction to Skinny , an educational distributed lock service ○ How Skinny reaches consensus ○ How Skinny deals with instance failure ● Part III - Implementation ○ A simple Paxos-like protocol ○ Making our protocol more reliable

Part I Hot Potato at Scale

Same potato!

Protocols Implementations ● Paxos ● Chubby ○ Multi-Paxos ○ Coarse grained lock service ○ Cheap Paxos ● etcd ● Raft ○ A distributed key value store ● ZooKeeper Atomic Broadcast ● Apache ZooKeeper ● Proof-of-Work Systems ○ A centralized service for ○ Bitcoin maintaining configuration ● Lockstep Anti-Cheating information, naming, providing ○ Age of Empires distributed synchronization Raft Logo: Attribution 3.0 Unported (CC BY 3.0) Source: https://raft.github.io/#implementations Etcd Logo: Apache 2 Source: https://github.com/etcd-io/etcd/blob/master/LICENSE Zookeeper Logo: Apache 2 Source: https://zookeeper.apache.org/

Want more theory? See paxos-roles.pdf at https://danrl.com/talks/

Part II Distributed Consensus Hands-on

Introducing Skinny ● Paxos-based ● Minimalistic ● Educational ● Lock Service The “Giraffe”, “Beaver”, “Alien”, and “Frame” graphics on the following slides have been released under Creative Commons Zero 1.0 Public Domain License

4 3 5 Instances 2 1

4 3 5 Quorum 2 1

4 3 5 Majority 2 1

4 3 5 Also a majority 2 1

4 3 5 NOT a majority 2 1

Name satsuki Incr. 4 ID 0 Promised 0 4 Holder Name mei Incr. 3 ID 0 Promised 0 Name totoro Holder Incr. 5 ID 0 Promised 0 Holder 3 5 Instance State Information Name kanta Name catbus 2 1 Incr. 2 Incr. 1 ID 0 ID 0 Promised 0 Promised 0 Holder Holder

Instance Name Name mei Unique "Increment" Incr. 3 ID 0 Current Paxos Round Number (ID) Promised 0 Holder foo Promised Paxos Round Number Agreed-on value (Lock Holder) 3

Name satsuki Incr. 4 ID 0 Promised 0 4 Holder Name mei Incr. 3 ID 0 Promised 0 Name totoro Holder Incr. 5 ID 0 Promised 0 Holder 3 5 Client asking for the lock Name kanta Name catbus 2 1 Incr. 2 Incr. 1 ID 0 ID 0 Promised 0 Promised 0 Holder Holder

Let's get used to the lab...

Lab Machine Folder Structure /home/ubuntu/ └── skinny Binaries ├── bin Source of the Skinny CLI tools ├── cmd ├── config Source of the Skinny config parser module ├── doc │ ├── ansible │ ├── examples │ ├── img │ ├── plots Our working directory │ ├── terraform │ └── workshop Lab virtual machine disk image │ ├── ami │ ├── code Lab code/config/scripts │ ├── configs for our experiments │ └── scripts ├── proto Protocol buffer definitions (API definitions) ├── skinny └── vendor Main Skinny source code 3rd party source code

How Skinny reaches consensus

Name satsuki SKINNY QUORUM Incr. 4 ID 0 Promised 0 4 Holder Name mei Incr. 3 ID 0 Promised 0 Name totoro Holder Incr. 5 ID 0 Promised 0 Holder 3 5 Lock please? Name kanta Name catbus 2 1 Incr. 2 Incr. 1 ID 0 ID 0 Promised 0 Promised 0 Holder Holder

Name satsuki PHASE 1A: PROPOSE Incr. 4 ID 0 Promised 0 4 Holder Name mei Incr. 3 ID 0 Promised 0 Name totoro Holder Incr. 5 ID 0 Promised 0 Holder 3 5 Proposal ID 1 Proposal ID 1 Proposal ID 1 Lock please? Proposal Name kanta ID 1 Name catbus 2 1 Incr. 2 Incr. 1 ID 0 ID 0 Promised 0 Promised 1 Holder Holder

Name satsuki PHASE 1B: PROMISE Incr. 4 ID 0 Promised 1 4 Holder Name mei Incr. 3 ID 0 Promised 1 Name totoro Holder Promise Incr. 5 ID 1 ID 0 Promised 1 Holder 3 5 Promise ID 1 Promise ID 1 Promise Name kanta ID 1 Name catbus 2 1 Incr. 2 Incr. 1 ID 0 ID 0 Promised 1 Promised 1 Holder Holder

Name satsuki PHASE 2A: COMMIT Incr. 4 ID 0 Promised 1 4 Holder Name mei Incr. 3 ID 0 Promised 1 Name totoro Holder Commit Incr. 5 ID 1 ID 0 Holder Beaver Promised 1 Holder 3 5 Commit ID 1 Holder Beaver Commit ID 1 Holder Beaver Commit Name kanta ID 1 Name catbus 2 1 Incr. 2 Holder Beaver Incr. 1 ID 0 ID 1 Promised 1 Promised 1 Holder Holder Beaver

Name satsuki PHASE 2B: COMMITTED Incr. 4 ID 1 Promised 1 4 Holder Beaver Name mei Incr. 3 ID 1 Promised 1 Name totoro Holder Beaver Committed Incr. 5 ID 1 Promised 1 Holder Beaver 3 5 Committed Committed Lock acquired! Holder is Beaver. Name kanta Committed Name catbus 2 1 Incr. 2 Incr. 1 ID 1 ID 1 Promised 1 Promised 1 Holder Beaver Holder Beaver

Experiment One Important: Run all commands in folder ~/skinny/doc/workshop/ 1.) Inspect Quorum skinnyctl status 2.) Acquire Lock for "beaver" (using instance catbus ) skinnyctl acquire --instance=catbus beaver 3.) Inspect Quorum skinnyctl status 4.) Release Lock (using random instance) skinnyctl release 5.) Inspect Quorum skinnyctl status Note: Reset the Quorum to initial state to start over! ./scripts/reset-experiment-one.sh

How Skinny deals with Instance Failure

Name satsuki SCENARIO Incr. 4 ID 9 Promised 9 4 Holder Beaver Name mei Incr. 3 ID 9 Promised 9 Name totoro Holder Beaver Incr. 5 ID 9 Promised 9 Holder Beaver 3 5 Name kanta Name catbus 2 1 Incr. 2 Incr. 1 ID 9 ID 9 Promised 9 Promised 9 Holder Beaver Holder Beaver

Name satsuki TWO INSTANCES FAIL Incr. 4 ID 9 Promised 9 4 Holder Beaver Name mei Incr. 3 ID 9 Promised 9 Name totoro Holder Beaver Incr. 5 ID 9 Promised 9 Holder Beaver 3 5 Name kanta Name catbus 2 1 Incr. 2 Incr. 1 ID 9 ID 9 Promised 9 Promised 9 Holder Beaver Holder Beaver

Name satsuki INSTANCES ARE BACK Incr. 4 ID 9 BUT STATE IS LOST Promised 9 4 Holder Beaver Name mei Incr. 3 ID 0 Promised 0 Name totoro Holder Incr. 5 ID 0 Promised 0 Lock Holder 3 5 please? Name kanta Name catbus 2 1 Incr. 2 Incr. 1 ID 9 ID 9 Promised 9 Promised 9 Holder Beaver Holder Beaver

Name satsuki INSTANCES ARE BACK Incr. 4 ID 9 BUT STATE IS LOST Promised 9 4 Holder Beaver Name mei Incr. 3 ID 3 Promised 3 Name totoro Holder Proposal Incr. 5 ID 3 ID 0 Promised 0 Lock Holder 3 5 please? Proposal ID 3 Proposal Proposal ID 3 ID 3 Name kanta Name catbus 2 1 Incr. 2 Incr. 1 ID 9 ID 9 Promised 9 Promised 9 Holder Beaver Holder Beaver

Name satsuki PROPOSAL REJECTED Incr. 4 ID 9 Promised 9 4 Holder Beaver Name mei Incr. 3 ID 3 Promised 3 NOT Promised Name totoro Holder ID 9 Incr. 5 Holder Beaver ID 0 Promised 3 Holder 3 5 Promise ID 3 NOT Promised NOT Promised ID 9 ID 9 Holder Beaver Holder Beaver Name kanta Name catbus 2 1 Incr. 2 Incr. 1 ID 9 ID 9 Promised 9 Promised 9 Holder Beaver Holder Beaver

Name satsuki START NEW PROPOSAL Incr. 4 ID 9 WITH LEARNED VALUES Promised 9 4 Holder Beaver Name mei Incr. 3 ID 9 Promised 12 Name totoro Holder Beaver Proposal Incr. 5 ID 12 ID 0 Promised 3 Holder 3 5 Proposal ID 12 Proposal Proposal ID 12 ID 12 Name kanta Name catbus 2 1 Incr. 2 Incr. 1 ID 9 ID 9 Promised 9 Promised 9 Holder Beaver Holder Beaver

Name satsuki PROPOSAL ACCEPTED Incr. 4 ID 9 Promised 12 4 Holder Beaver Name mei Incr. 3 ID 12 Promised 12 Promise Name totoro Holder Beaver ID 12 Incr. 5 ID 0 Promised 12 Holder 3 5 Promise ID 12 Promise ID 12 Promise ID 12 Name kanta Name catbus 2 1 Incr. 2 Incr. 1 ID 9 ID 9 Promised 12 Promised 12 Holder Beaver Holder Beaver

Name satsuki COMMIT LEARNED VALUE Incr. 4 ID 9 Promised 12 4 Holder Beaver Name mei Incr. 3 ID 12 Promised 12 Name totoro Commit Holder Beaver Incr. 5 ID 12 ID 0 Holder Beaver Promised 12 Holder 3 5 Commit ID 12 Holder Beaver Commit ID 12 Commit Holder Beaver ID 12 Holder Beaver Name kanta Name catbus 2 1 Incr. 2 Incr. 1 ID 9 ID 9 Promised 12 Promised 12 Holder Beaver Holder Beaver

Name satsuki COMMIT ACCEPTED Incr. 4 ID 12 LOCK NOT GRANTED Promised 12 4 Holder Beaver Name mei Incr. 3 ID 12 Promised 12 Committed Name totoro Holder Beaver Incr. 5 ID 12 Promised 12 Lock NOT acquired! Holder Beaver 3 5 Holder is Beaver. Committed Committed Committed Name kanta Name catbus 2 1 Incr. 2 Incr. 1 ID 12 ID 12 Promised 12 Promised 12 Holder Beaver Holder Beaver

Experiment Two Important: Run all commands in folder ~/skinny/doc/workshop/ Screwed up? No worries! 1.) Inspect Quorum skinnyctl status Reset the Quorum to initial state via: ./scripts/reset-experiment-two.sh 2.) Stop instances mei and totoro sudo systemctl stop skinny@mei sudo systemctl stop skinny@totoro 3.) Inspect Quorum. Verify that instances mei and totoro are down! skinnyctl status 4.) Start instances mei and totoro again sudo systemctl start skinny@mei sudo systemctl start skinny@totoro 5.) Inspect Quorum. Verify that instances mei and totoro are out of sync! skinnyctl status 6.) Acquire Lock for "alien" using instance mei skinnyctl acquire --instance=mei alien

Part III Implementing Distributed Consensus

Skinny APIs