Composable Reliability for Asynchronous Systems Sunghwan Yoo , Charles Killian, Terence Kelly, Hyoun Kyu Cho, Steven Plite
Distributed systems: Key-value store X=1 X=1 X=1 X=1 x=1 2 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Distributed systems: Key-value store #1 #2 #3 X=1 X=1 X=1 X=1 #1 #2 x=1 x=1 3 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Distributed systems: Key-value store #1 #2 #3 X=1 X=1 X=1 X=1 #1 #2 Retransmission Restart upon crash-restart Rollback-recovery protocol - Checkpoint-based - Message-logging based x=1 x=1 4 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Distributed systems: Handling failures 5 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Distributed systems: Handling failures 6 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Distributed systems: Handling failures Guaranteeing global reliability across independently developed components is difficult. 7 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Ken: Crash-restart tolerant protocol Ken Ken Ken Ken Ken Ken Ken Ken Ken Ken Ken 8 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Ken: Crash-restart tolerant protocol Ken Ken Ken Ken Ken preserves global reliability when you compose independent components Ken Ken Ken Ken Ken Ken Ken 9 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Ken Highlights = Ken = makes crash-restarted node look like slow node = 1. Reliability Uncoordinated rollback recovery protocol Ken It’s also scalable provides 2. Composability Write locally, work globally 3. Easy Programmability Event-driven system (not a new paradigm) Transparently applicable to Mace 10 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Related Work Rollback Recovery Much research through 1990s Waterken (1999) : Ken principles in different programming abstractions Lowell et al. (2000) : Output validity Computing Surveys (2002): summary of mature field Software Persistent Memory (ATC 2012) Different approach to orthogonal persistence Hardening Crash-Tolerant Systems (ATC 2012) Detects arbitrary state corruption in event-driven code Could make Ken-based software more reliable 11 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Design of Ken Communicating event loop WORLD KEN KEN KEN 12 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Design of Ken WORLD Commit handler() Send out messages KEN Event loop Changes to EXTERNALIZER begins memory heap Sending messages Store as checkpoint file KEN Time EXTERNALIZER 13 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Design of Ken WORLD handler() KEN EXTERNALIZER Msgs are resent Acked msgs are removed handler() handler() OTHER KEN Time EXTERNALIZER 14 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Design of Ken WORLD handler() Crashed/recovered nodes look like slow nodes KEN Uncoordinated protocol → scalable EXTERNALIZER Msgs are resent Acked msgs are removed handler() handler() OTHER KEN Time EXTERNALIZER 15 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Ken: Composable Reliability Seller Buyer 16 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Ken: Composable Reliability Seller Guaranteeing global reliability across independent components is difficult task. Buyer 17 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Ken: Composable Reliability Seller Ken When you compose independent components, Ken reliability will be transparently guaranteed by Ken Ken allows decentralized development Buyer Ken Ken 18 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Ken Illustrated: “ Ping-pong Counter” #include "ken.h" analogue of main() #include "kenapp.h" int ken_handler (void *msg, int len, kenid_t src) { int *cnt; initialization begin transaction if (NULL == msg) { cnt = ken_malloc (sizeof *cnt); *cnt = 0; persistent heap ken_set_app_data (cnt); } entry point else { incoming message cnt = ken_get_app_data (); *cnt = *(int*)msg + 1; ken_send (src, cnt, sizeof *cnt); } return -1; fire & forget end transaction } 19 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Ken Illustrated: “ Ping-pong Counter” #include "ken.h" analogue of main() #include "kenapp.h" int ken_handler (void *msg, int len, kenid_t src) { int *cnt; Ken programming is simple initialization begin transaction if (NULL == msg) { 1. Implement ken_handler() instead of main() cnt = ken_malloc (sizeof *cnt); *cnt = 0; 2. Use ken_malloc / ken_send instead of malloc / send persistent heap ken_set_app_data (cnt); 3. Use ken_get_app_data / ken_set_app_data for } entry point entry to persistent heap else { incoming message cnt = ken_get_app_data (); *cnt = *(int*)msg + 1; ken_send (src, cnt, sizeof *cnt); } return -1; fire & forget end transaction } 20 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Mace: “ Ping-pong Counter” Mace with Ken Mace Event-driven distributed • programmer makes service service PingPong; system language framework Used in many projects • services { Transport t; } in persistent heap state_variables { int cnt = 0; } messages { pong {int cnt;} } define state var & messages transitions { incoming message deliver(src, dest, msg) { begin transaction cnt = msg.cnt+1; send message route(src, pong(cnt)); fire & forget messaging } end transaction } 21 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Mace: “ Ping-pong Counter” Mace with Ken Mace Event-driven distributed • programmer makes service service PingPong; No changes are needed system language framework Used in many projects • services { Transport t; } in persistent heap state_variables { int cnt = 0; } messages { pong {int cnt;} } define state var & messages You don’t need to change anything for Ken transitions { incoming message Reliability and composability comes easily deliver(src, dest, msg) { begin transaction cnt = msg.cnt+1; send message route(src, pong(cnt)); fire & forget messaging } end transaction } 22 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Ken: Integration with Mace Ken Mace MaceKen Masks failures globally Composable reliability Packaged with distributed protocols Availability through replication Handles permanent failures Ken provides new benefits to legacy Mace applications! 23 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Implementation Ken C library Publicly available MaceKen Modifications to existing Mace runtime libraries No changes to existing Mace services Linux Container (LXC) environment Simulating correct power-failure behavior (in paper) 24 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Evaluation Microbenchmark Performance test kBay Composable reliability (in paper) Distributed analysis of 1.1 TB graph Versatility (in paper) Bamboo-DHT Failure masking & data survival for legacy Mace app 25 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Evaluation: Ken Microbenchmark Experimental setup 16 core, 2.4 GHz Xeon 32GB RAM Mirrored RAID Two 72 GB 15K RPM disks Test Ping-pong counter test between two Ken processes 3 KEN 1 KEN 4 2 Measure : latency and throughput 26 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Evaluation: Ken Microbenchmark 5 5 25,000 Thruput (events/sec) 4 4 20,000 Latency (ms) Latency (ms) 3 3 15,000 2 2 10,000 1 1 5,000 0 0 0 disk sync no sync ramfs sync disk sync no sync ramfs sync 27 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Evaluation: Bamboo-DHT Wide-area network Managed Network X=1 X=1 X=1 X=1 X=1 X=1 X=1 X=1 X=1 X=1 X=1 X=1 X=1 X=1 Single Colocation administration 28 Yoo, Killian, Kelly, Cho and Plite , “Composable Reliability for Asynchronous Systems”, USENIX ATC 2012
Recommend
More recommend
