WormSpace: A Modular Foundation for Simple, Verifiable Distributed - - PowerPoint PPT Presentation

▶
wormspace a modular foundation for simple verifiable
SMART_READER_LITE
LIVE PREVIEW

WormSpace: A Modular Foundation for Simple, Verifiable Distributed - - PowerPoint PPT Presentation

Mar 24, 2023 535 likes •738 views

WormSpace: A Modular Foundation for Simple, Verifiable Distributed Systems ACM Symposium on Cloud Computing Nov 22, 2019 Ji-Yong Shin 1 Jieung Kim 1 Wolf Honore 1 Hernan Vanzetto 1 Srihari Radhakrishnan 2 Mahesh Balakrishnan 3 Zhong Shao 1 1 Yale

slide-1
SLIDE 1

WormSpace: A Modular Foundation for Simple, Verifiable Distributed Systems

Ji-Yong Shin1 Jieung Kim1 Wolf Honore1 Hernan Vanzetto1 Srihari Radhakrishnan2 Mahesh Balakrishnan3 Zhong Shao1

1Yale University 2Duke University 3Facebook

ACM Symposium on Cloud Computing Nov 22, 2019

slide-2
SLIDE 2

Cloud and Distributed Application Environment

  • Numerous distributed services are readily available
  • New applications are built by combining existing building blocks
  • New services are continuously developed and deployed

2

Amazon EBS Google Dataflow Azure Virtual Network Azure SQL DB Google ML Engine Amazon Digital Publishing

slide-3
SLIDE 3

Amazon EBS Google Dataflow Azure Virtual Network Azure SQL DB Google ML Engine Amazon Digital Publishing

Cloud and Distributed Application Environment

  • Distributed services use and re-implement similar features
  • Distributed systems are complex and difficult to build correctly

3

Exploration for a common, bug-free foundation

Redundant efforts Subtle bugs

slide-4
SLIDE 4

Design Goals

  • 1. Supports common needs for most systems
  • 2. Simple and easy-to-understand APIs
  • 3. Flexible support for optimizations
  • 4. Guaranteed correctness with extensibility

4

System design Formal Verification

slide-5
SLIDE 5

Write once register (WOR)

  • Distributed register

– Replicated by construction (fault tolerance, availability, durability)

  • Write-once-read-many abstraction

– Atomically writes data (consistency) – Only one of concurrent writes succeeds (concurrency control, immutability)

5

WOR

WOR WOR WOR

Writes

  • Logically equivalent to consensus

(Paxos, Chain-replication, PBFT, etc.)

  • Lowest common denominator
slide-6
SLIDE 6

WORs in Existing Systems

  • State machine replication (SMR) and multi-Paxos

– Append / sequential read to WORs

  • Shared log: Corfu, Tango

– Append / random read to WORs

  • Transaction coordinator: 2 phase commit

– Random write / random read to WORs

  • Coordination service: chubby, zookeeper

– File APIs over SMR on WORs

  • Group communication: pub/sub

– Append / sequential read to WORs

slide-7
SLIDE 7

WOR APIs

  • Capture

– Preemptible lock concept – Coordination before write – Returns a capture token

  • Write

– Writes to the WOR – Capture must be valid

  • Read

– Reads the register – Returns data or “empty”

7

WOR A B A B

Paxos: phase 1 prepare PBFT: pre-prepare + prepare Chain-replication: no-op Paxos: phase 2 accept PBFT: commit Chain-replication: write to the chain

slide-8
SLIDE 8

WormSpace (Write-Once-Read-Many Address Space)

  • An address space of WORs
  • Write-once-segment (WOS) for management

– Unit of allocation (alloc) and garbage collection (trim) – Consists of special WORs and data WORs – Support for batch-capture and batch-write to all WORs

8

Paxos S S S S S

  • 1. The complexity of Paxos and

distributed servers are hidden

  • 2. Can use alternative

implementations Distributed servers Client library

WOR WOR WOR WOR WOR WOR WOR WOR WOR WOR WOR WOR WOR WOR WOR

…

Meta WOR Trim WOR WOR WOR WOR Meta WOR Trim WOR WOR WOR WOR Meta WOR Trim WOR WOR WOR WOR

…

WO Segment WO Segment WO Segment

Meta WOR Trim WOR WOR

WormSpace

…

slide-9
SLIDE 9

WormSpace Applications

  • WormPaxos

– Multi-Paxos / state machine replications

  • WormLog

– Corfu / shared-log

  • WormTX

– 2PC variant / non-blocking atomic commit

…

Meta WOR Trim WOR WOR WOR WOR Meta WOR Trim WOR WOR WOR WOR Meta WOR Trim WOR WOR WOR WOR

…

WO Segment WO Segment WO Segment WormSpace

…

9

Please refer to the paper for interesting latency optimizations

slide-10
SLIDE 10

WormPaxos: Flexible Design Choices

  • Multi-Paxos variant for state machine replication
  • Design decisions can be easily configured

– Various single-degree consensus protocols – Leader election: who allocates a WOS and batch captures it?

  • Mencius-like rotating leaders are easy to implement
  • Raft-like leader election can be implemented orthogonally with a timer

– When to call trim call determines durability

10

…

Meta WOR Trim WOR WOR WOR WOR Meta WOR Trim WOR WOR WOR WOR Meta WOR Trim WOR WOR WOR WOR

…

WO Segment WO Segment WO Segment WormSpace

…

  • 1. Paxos
  • 2. Chain-replication
  • 3. Etc.

State Machine Replication Commands

WormSpace APIs are enough and no need to understand Paxos

slide-11
SLIDE 11

Formal Verification

  • WOR is primitive, but encapsulates key distributed properties

– Consistency, durability and availability

  • Concurrent Certified Abstraction Layer (CCAL)

– Divides software into layers – Verifies each layer – Verifies layers interact correctly – Lower layer properties hold in higher layers

11

Can we verify WOR once and reuse it multiple times?

[Gu, et al. PLDI 18]

slide-12
SLIDE 12

Certified Concurrent Abstraction Layer (CCAL)

12

LList C Implementation Specification LQueue (Uses LList) LSched (Uses LQueue) Contextual refinement proof Contextual refinement proof Refinement proof Informally, when we run ANY Program (context) on the Queue, the state reached by the Queue has a matching state with the List which runs the Queue’s impl and the Program. Sched does not need to know about List at all!

slide-13
SLIDE 13
  • 1. Rely-guarantee style concurrency

reasoning

  • 2. Use of a logical network log
  • 3. Proof by induction on the log

Verification Details

13

Data Trusted Computing Base Data Trusted Computing Base Data Trusted Computing Base Data TCB Data Trusted Computing Base Data Trusted Computing Base Data Trusted Computing Base Data TCB Global Layer (distributed system model) Write Once Register (WOR) Write Once Segment (WOS) WormSpace Client Library Layers Server layers Distributed protocol verification (Paxos immutability) Oblivious of distributed nature Preserves distributed protocol correctness CertiKOS (fully verified OS)

[Gu, et al. OSDI 16]

The first end-to-end verification

  • f distributed system

from the OS (x86 ASM to Dist Apps)

WormPaxos WormLog WormPaxos WormLog WormPaxos WormLog WormPaxos WormLog WormPaxos WormPaxos WormPaxos WormTx

Worm Client Paxos Proposer Worm Server Paxos Acceptor Worm Client Paxos Proposer Worm Server Paxos Acceptor Worm Client Paxos Proposer Worm Server Paxos Acceptor Worm Client Paxos Proposer Worm Server Paxos Acceptor

slide-14
SLIDE 14

Experience

14

Data Trusted Computing Base Data Trusted Computing Base Data Trusted Computing Base Data TCB Data Trusted Computing Base Data Trusted Computing Base Data Trusted Computing Base Data TCB Global Layer (distributed system model) Write Once Register (WOR) Write Once Segment (WOS) WormSpace Client Library Layers Server layers

WormPaxos WormLog WormPaxos WormLog WormPaxos WormLog WormPaxos WormLog WormPaxos WormPaxos WormPaxos WormTx

Worm Client Paxos Proposer Worm Server Paxos Acceptor Worm Client Paxos Proposer Worm Server Paxos Acceptor Worm Client Paxos Proposer Worm Server Paxos Acceptor Worm Client Paxos Proposer Worm Server Paxos Acceptor

6 months < 1 month 108K Lines of Coq Proof

4.5K CLoC 359 CLoC 362 CLoC 547 CLoC

1.5 months

  • Simple API and no need to understand distributed protocols
  • Distributed verification is hidden, but verified properties hold
slide-15
SLIDE 15

Evaluation

  • WormPaxos vs Egalitarian Paxos and its calssical multi-Paxos impl.

– Amazon EC2: 3 servers and 16 client nodes – Write-only benchmark – C vs. Go and different internals

15 2 4 6 8 10 12 14 16 5 10 15 20 25 30 Latency (ms) Throughput (KOps/s) WormPaxos EPaxos CPaxos

Verified systems are not slow!

slide-16
SLIDE 16

Evaluation

  • WormSpace over CertiKOS

– Local cloud with same configuration as Amazon EC2 – Over 10X lower throughput and over 1.5X higher latency – Mainly due to inefficiencies in LwIP of CertiKOS

16 1 2 3 4 5 6 5 10 15 20 25 30 Latency (ms) Throughput (KOps/s) Ubuntu+WormPaxos CertiKOS+WormPaxos

slide-17
SLIDE 17

Conclusion

  • Write once registers for programming

– Lowest common denominator for most systems – Source of consistency, availability, and durability

  • Write once register for verification

– Primitive module that encapsulates key distributed system properties – Can be verified once and reused to simplify application verification

  • WormSpace for simple, verifiable distributed systems

– Address space of WOR and with extra APIs – Allows for simple and flexible distributed application designs – Facilitates verification of distributed applications

17

slide-18
SLIDE 18

Thank you

Questions? jiyong.shin@yale.edu

18