Asynchronous Systems Sunghwan Yoo , Charles Killian, Terence Kelly, - - PowerPoint PPT Presentation

▶
asynchronous systems
SMART_READER_LITE
LIVE PREVIEW

Asynchronous Systems Sunghwan Yoo , Charles Killian, Terence Kelly, - - PowerPoint PPT Presentation

Nov 01, 2023 133 likes •492 views

Composable Reliability for Asynchronous Systems Sunghwan Yoo , Charles Killian, Terence Kelly, Hyoun Kyu Cho, Steven Plite Distributed systems: Key-value store X=1 X=1 X=1 X=1 x=1 2 Yoo, Killian, Kelly, Cho and Plite , Composable

slide-1
SLIDE 1

Composable Reliability for Asynchronous Systems

Sunghwan Yoo, Charles Killian, Terence Kelly, Hyoun Kyu Cho, Steven Plite

slide-2
SLIDE 2

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

2

Distributed systems: Key-value store

x=1

X=1 X=1 X=1 X=1

slide-3
SLIDE 3

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

3

Distributed systems: Key-value store

X=1 X=1

#1

X=1

#2 #1 #2

X=1

#3

x=1 x=1

slide-4
SLIDE 4

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

4

Distributed systems: Key-value store

X=1 X=1

#1

X=1

#2 #1 #2

X=1

#3

x=1 x=1

Retransmission Restart upon crash-restart Rollback-recovery protocol

  • Checkpoint-based
  • Message-logging based
slide-5
SLIDE 5

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

5

Distributed systems: Handling failures

slide-6
SLIDE 6

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

6

Distributed systems: Handling failures

slide-7
SLIDE 7

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

7

Distributed systems: Handling failures

Guaranteeing global reliability across independently developed components is difficult.

slide-8
SLIDE 8

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

8

Ken: Crash-restart tolerant protocol

Ken Ken Ken Ken Ken Ken Ken Ken Ken Ken Ken

slide-9
SLIDE 9

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

9

Ken: Crash-restart tolerant protocol

Ken Ken Ken Ken Ken Ken Ken Ken Ken Ken Ken

Ken preserves global reliability when you compose independent components

slide-10
SLIDE 10

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

10

Ken Highlights

Ken makes crash-restarted node look like slow node

  • 1. Reliability

 Uncoordinated rollback recovery protocol  It’s also scalable

  • 2. Composability

 Write locally, work globally

  • 3. Easy Programmability

 Event-driven system (not a new paradigm)  Transparently applicable to Mace

= = =

Ken provides

slide-11
SLIDE 11

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

11

Related Work

 Rollback Recovery

Much research through 1990s

Waterken (1999) : Ken principles in different programming abstractions

Lowell et al. (2000) : Output validity

Computing Surveys (2002): summary of mature field

 Software Persistent Memory (ATC 2012)

 Different approach to orthogonal persistence

 Hardening Crash-Tolerant Systems (ATC 2012)

Detects arbitrary state corruption in event-driven code

Could make Ken-based software more reliable

slide-12
SLIDE 12

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

12

Design of Ken

 Communicating event loop

WORLD KEN KEN KEN

slide-13
SLIDE 13

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

13

Design of Ken

WORLD KEN KEN Time

Changes to memory heap Sending messages Commit

EXTERNALIZER EXTERNALIZER

handler()

Store as checkpoint file Send out messages Event loop begins

slide-14
SLIDE 14

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

14

Design of Ken

WORLD KEN OTHER KEN Time

handler() handler()

Msgs are resent

handler()

Acked msgs are removed

EXTERNALIZER EXTERNALIZER

slide-15
SLIDE 15

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

15

Design of Ken

WORLD KEN OTHER KEN Time

handler() handler()

Msgs are resent

handler()

Acked msgs are removed

EXTERNALIZER EXTERNALIZER

Crashed/recovered nodes look like slow nodes Uncoordinated protocol → scalable

slide-16
SLIDE 16

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

16

Ken: Composable Reliability

Seller Buyer

slide-17
SLIDE 17

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

17

Ken: Composable Reliability

Seller Buyer

Guaranteeing global reliability across independent components is difficult task.

slide-18
SLIDE 18

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

18

Ken: Composable Reliability

Seller Buyer

Ken Ken Ken Ken

When you compose independent components, reliability will be transparently guaranteed by Ken Ken allows decentralized development

slide-19
SLIDE 19

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

19

Ken Illustrated: “Ping-pong Counter”

#include "ken.h" #include "kenapp.h" int ken_handler(void *msg, int len, kenid_t src) { int *cnt; if (NULL == msg) { cnt = ken_malloc(sizeof *cnt); *cnt = 0; ken_set_app_data(cnt); } else { cnt = ken_get_app_data(); *cnt = *(int*)msg + 1; ken_send(src, cnt, sizeof *cnt); } return -1; }

begin transaction end transaction initialization incoming message persistent heap entry point fire & forget analogue of main()

slide-20
SLIDE 20

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

20

Ken Illustrated: “Ping-pong Counter”

#include "ken.h" #include "kenapp.h" int ken_handler(void *msg, int len, kenid_t src) { int *cnt; if (NULL == msg) { cnt = ken_malloc(sizeof *cnt); *cnt = 0; ken_set_app_data(cnt); } else { cnt = ken_get_app_data(); *cnt = *(int*)msg + 1; ken_send(src, cnt, sizeof *cnt); } return -1; }

begin transaction end transaction initialization incoming message persistent heap entry point fire & forget analogue of main() Ken programming is simple

  • 1. Implement ken_handler() instead of main()
  • 2. Use ken_malloc / ken_send instead of malloc / send
  • 3. Use ken_get_app_data / ken_set_app_data for

entry to persistent heap

slide-21
SLIDE 21

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

21

Mace: “Ping-pong Counter”

service PingPong; services { Transport t; } state_variables { int cnt = 0; } messages { pong {int cnt;} } transitions { deliver(src, dest, msg) { cnt = msg.cnt+1; route(src, pong(cnt)); } }

Mace Mace with Ken

programmer makes service define state var & messages incoming message send message end transaction begin transaction in persistent heap fire & forget messaging

  • Event-driven distributed

system language framework

  • Used in many projects
slide-22
SLIDE 22

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

22

Mace: “Ping-pong Counter”

service PingPong; services { Transport t; } state_variables { int cnt = 0; } messages { pong {int cnt;} } transitions { deliver(src, dest, msg) { cnt = msg.cnt+1; route(src, pong(cnt)); } }

Mace Mace with Ken

programmer makes service define state var & messages incoming message send message end transaction begin transaction in persistent heap fire & forget messaging You don’t need to change anything for Ken Reliability and composability comes easily

  • Event-driven distributed

system language framework

  • Used in many projects

No changes are needed

slide-23
SLIDE 23

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

23

Ken: Integration with Mace

Mace Ken Masks failures globally Composable reliability Availability through replication Handles permanent failures MaceKen Packaged with distributed protocols

Ken provides new benefits to legacy Mace applications!

slide-24
SLIDE 24

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

24

Implementation Ken

 C library  Publicly available

MaceKen

 Modifications to existing Mace runtime libraries  No changes to existing Mace services

Linux Container (LXC) environment

 Simulating correct power-failure behavior (in paper)

slide-25
SLIDE 25

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

25

Evaluation

 Microbenchmark

 Performance test

 kBay

 Composable reliability (in paper)

 Distributed analysis of 1.1 TB graph

 Versatility (in paper)

 Bamboo-DHT

 Failure masking & data survival for legacy Mace app

slide-26
SLIDE 26

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

26

Evaluation: Ken Microbenchmark Experimental setup

 16 core, 2.4 GHz Xeon  32GB RAM  Mirrored RAID

 Two 72 GB 15K RPM disks

Test

 Ping-pong counter test between two Ken processes

KEN KEN

 Measure : latency and throughput

1 2 3 4

slide-27
SLIDE 27

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

27

Evaluation: Ken Microbenchmark

1 2 3 4 5 disk sync no sync ramfs sync

Latency (ms)

5,000 10,000 15,000 20,000 25,000 1 2 3 4 5 disk sync no sync ramfs sync

Thruput (events/sec) Latency (ms)

slide-28
SLIDE 28

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

28

Evaluation: Bamboo-DHT

Wide-area network

X=1 X=1 X=1 X=1 X=1 X=1 X=1 X=1 X=1 X=1 X=1

Managed Network

X=1 X=1 X=1

Single administration Colocation

slide-29
SLIDE 29

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

29

Evaluation: Bamboo-DHT

Wide-area network

X=1 X=1 X=1 X=1 X=1 X=1 X=1 X=1 X=1 X=1 X=1

Managed Network

X=1 X=1 X=1

Single administration Colocation

 Bamboo-DHT on Mace will fail  Bamboo-DHT on MaceKen will work  Mace vs. MaceKen: testing rolling-restart

and correlated failure

slide-30
SLIDE 30

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

30

Evaluation: Bamboo-DHT Experimental setup

 12 dual quad-core machines, 8GB RAM  4x1Gb Ethernet connections  300 DHT nodes on 12 machines  Measures  Immediate-Get  Stored-Get

X=1

X=1

X=? X=1

X=1

X=?

slide-31
SLIDE 31

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

31

Bamboo-DHT: Crash-restart Failures

begin experiment Stored GET begins correlated failure very low data survivability

Correlated Failures

bootstrap node spared 100% data survivability

slide-32
SLIDE 32

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

32

Bamboo-DHT: Rolling-restart

churn begins churn ends

Rolling-restart

still low data survivability stored GET begins 100% data survivability

slide-33
SLIDE 33

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

33

Bamboo-DHT: Rolling-restart

churn begins churn ends

Rolling-restart

still low data survivability stored GET begins 100% data survivability

MaceKen provides automated crash resilience to legacy Mace app and 100% data survivability

slide-34
SLIDE 34

Yoo, Killian, Kelly, Cho and Plite, “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012

34

Conclusion

Ken available now: http://ai.eecs.umich.edu/~tpkelly/Ken/ MaceKen available soon: http://www.macesystems.org/maceken/

Thank you! Ken provides

  • 1. Reliability
  • 2. Composability
  • 3. Programmability

MaceKen provides

  • 1. Transparency
  • 2. 100% data survivability