Manylogs Improving CMR/SMR Disk Bandwidth & Latency Tiratat Patana-anake , Vincentius Martin†, Nora Sandler, Cheng Wu, and Haryadi S. Gunawi †
2 Manylogs @ MSST ‘16 Got 100% of the read bandwidth User 1 Big Read
3 Many Got 50% of the read bandwidth Fair Share User 1 Big Read User 2 Big Read
4 Manylogs @ MSST ‘16 1 Still Fair! 2 1/N bandwidth 3 … N
5 Manylogs @ MSST ‘16 Our Oh no! 5% bandwidth?! User 1 Big Read User 2 Small Durable Writes
6 Manylogs @ MSST ‘16 Our m More Bandwidth Faster Please! Latency Please! User 1 Big Read User 2 Small Durable Writes
7 Manylogs @ MSST ‘16 Ordered Journaling Data Journaling J J J J J J J J Journal Big Read Small Writes Seek
8 Manylogs @ MSST ‘16 Ordered Journaling Data Journaling J J J J J J J J Journal Big Read Small Writes Seek
9 Manylogs @ MSST ‘16 Problems with Current Journaling Ordered Journaling Data Journaling
10 Manylogs @ MSST ‘16 Problems with Current Journaling First Write Ordered Journaling Data Jo Second Write J
11 Manylogs @ MSST ‘16 Introducing Manylogs Single Log J Manylogs 10 MB J J J J J J J 100 MB
12 Manylogs @ MSST ‘16 Small writes made durable to the Manylogs nearest log without seeking J J J J J J J J J J J J Journal Big Read Small Writes Seek
13 Manylogs @ MSST ‘16 Manylogs q Reserved log spaces uniformly across the disk § 10 MB every 100 MB q Follow the disk head (last big I/O) q Redirect Small Writes (e.g. ≤ 256 KB) § Nearest log: log closest to last big I/O q Sequential Writes are left untouched 10 MB J J J J J J J 100 MB
14 Manylogs @ MSST ‘16 Increased Read Throughput Manylogs Reduced Write Latency
15 Manylogs @ MSST ‘16 Where are logs on the disk? 10 MB J J J J J J J 100 MB The log space = whole platter
16 Manylogs @ MSST ‘16 Where are logs on the disk? 10 MB J J J J J J J 100 MB The log space = whole platter
17 Manylogs @ MSST ‘16 Same cylinder = No seek! J J J J J J J Log for others in the same cylinder
18 Manylogs @ MSST ‘16 Latency Ratio of Max (ms) Read Bandwidth User 2 User 1 4KB Random Writes 128MB Sequential Reads At different intensities Ordered vs. Data vs. writes/s writes/s Adaptive vs. Manylogs writes/s • 320 writes/s Disk
19 Manylogs @ MSST ‘16 Adaptive Journaling q Middle ground between ordered journaling and data journaling q Single-log design q Prabhakaran et al., ATC ‘05
20 Manylogs @ MSST ‘16 Ordered Data Results Adaptive Manylogs 100% 80% Ratio of Max Bandwidth 60% 40% 20% 0% 40 80 160 320 Random Writes per Second (IOPS)
21 Manylogs @ MSST ‘16 Ordered Data Results Adaptive Manylogs 100 80% Ratio of Max Bandwidth 57% at 40 IOPS 60% 40% 20% 5% at 320 IOPS 0% 40 80 160 320 Random Writes per Second (IOPS)
22 Manylogs @ MSST ‘16 Ordered Data Results Adaptive Manylogs 100% 80% 73% at 40 IOPS Ratio of Max Bandwidth 60% 40% 20% 9% at 320 IOPS 0% 40 80 160 320 Random Writes per Second (IOPS)
23 Manylogs @ MSST ‘16 Ordered Data Results Adaptive Manylogs 100% 80% Ratio of Max Bandwidth 60% 40% 20% 0% 40 80 160 320 Random Writes per Second (IOPS)
24 Manylogs @ MSST ‘16 Ordered Data Manylogs gives daptive Manylogs the most bandwidth 50% of max 100% bandwidth at extreme IOPS 80% of Max Bandwidth 60% Hey! What about my latency? 40 80 160 320 Random Writes per Second (IOPS)
25 Manylogs @ MSST ‘16 Ordered Data Results Adaptive Manylogs 600 500 Average sync Latency (ms) 400 300 200 100 40 80 160 320 Random Writes per Second (IOPS)
26 Manylogs @ MSST ‘16 Ordered Data Results Adaptive Manylogs 600 500 Average sync Latency (ms) 400 300 200 100 0 40 80 160 320 Random Writes per Second (IOPS)
27 Manylogs @ MSST ‘16 Ordered Data Results Adaptive Manylogs 600 500 Average sync Latency (ms) 400 300 200 100 0 40 80 160 320 Random Writes per Second (IOPS)
28 Manylogs @ MSST ‘16 Ordered Data Results Adaptive Manylogs 600 500 Average sync Latency (ms) 400 300 200 100 0 40 80 160 320 Random Writes per Second (IOPS)
29 Manylogs @ MSST ‘16 Ordered Data Results Adaptive Manylogs WOW! Fast latency 600 at extreme IOPS! 500 Average sync Latency (ms) 400 300 200 100 0 40 80 160 320 Random Writes per Second (IOPS)
30 Manylogs @ MSST ‘16 Ordered Data Results Adaptive Manylogs 100% 600 500 Average sync Latency (ms) 80% Ratio of Max Bandwidth 400 60% 300 40% 200 20% 100 0% 0 40 80 160 320 Random Writes per Second (IOPS)
31 Manylogs @ MSST ‘16 Results
32 Manylogs @ MSST ‘16 Latency Ratio of Max (ms) Read Bandwidth User 2 User 1 “fileserver” 128MB Sequential Reads • Using Filebench • Multi-threaded • 2, 4, 8 instances
33 Manylogs @ MSST ‘16 Ordered Data ogs Manylogs provides the best outcomes! 100% 4000 80% Ratio of Max Bandwidth Operation Latency (ms) 3000 60% 2000 40% 1000 20% 0% 0 2 4 8 fileserver Instances
34 Manylogs @ MSST ‘16 Checkpointing Data Journaling Manylogs q Periodically q “Lazy” or § Usually every 5 “Off-hours” secs q Rarely full q Journal can get because just filled fast small writes are because all redirected writes are in the q Log Swapping journal!
35 Manylogs @ MSST ‘16 Log Swapping J J J J Hot Area! Cold Area! J J J J J J J J Journal Big Read Small Writes Seek
36 Manylogs @ MSST ‘16 Integrations q File System (MLFS) § Durability-Only Mode (O_DUR) q SMR Disk (MLSMR) q RAID
37 Manylogs @ MSST ‘16 Cassandra Write Path Writes Commit SSTable Memtable Log Memory Disk
38 Manylogs @ MSST ‘16 Cassandra Write Path Writes Commit SSTable Memtable Log Memory Disk
39 Manylogs @ MSST ‘16 Cassandra Write Path Requires Fast Ideally SYNC Durability write Writes Commit SSTable Memtable Log But in practice, background Triggered write e.g. every 10 seconds Flush Random writes are the problem 😗 Commit Commit SSTable Memtable Log Log Temp File No location needed Memory Disk
40 Manylogs @ MSST ‘16 open(file, O_DUR); q Need fast durability but not location constraints q Content of files will be put in Manylogs regardless of the write size q Never checkpoint their content q Random writes are not a problem anymore!
41 Manylogs @ MSST ‘16 Latency Ratio of Max (ms) Read Bandwidth User 2 User 1 MongoDB HDFS • 1 instance • 2 instances • 4 instances
42 Manylogs @ MSST ‘16 Data Most Bandwidth & Lowest Manylogs Latency with Manylogs 250 MongoDB Write Latency (ms) 80% 200 Ratio of Max Bandwidth 60% 150 40% 100 20% 50 0% 0 1DB 2DB 4DB MongoDB Instances (w/Default flush period)
43 Manylogs @ MSST ‘16 Manylogs & SMR One non-shingled surface = log space
44 Manylogs @ MSST ‘16 Manylogs & SMR Non-Shingled Tracks Shingled Band
45 Manylogs @ MSST ‘16 Latency Latency (ms) (ms) User 2 User 1 Build Server Back-end Server (WBS) Trace (LM-TBE) Trace
46 Manylogs @ MSST ‘16 Manylogs SMR (MLSMR) SMR Single-log SMR (SLSMR) 100 100 80 80 60 60 Percentile 40 40 20 20 ML-SMR SL-SMR 0 0 0 10 80 30 40 50 Latency (ms)
47 Manylogs @ MSST ‘16 Manylogs & RAID User 2 User 1 Big Read Small Durable Writes Disk #1 Disk #2 Disk #3 Disk #4 Bandwidth drops Max Bandwidth! up to 50% Mingzhe Hao, Gokul Soundararajan, Deepak Kenchammana-Hosekote, Andrew A. Chien, and Haryadi S. Gunawi. "The T ail at Store: A Revelation from Millions of Hours of Disk and SSD Deployments." FAST’16.
48 Manylogs @ MSST ‘16 Latency Ratio of Max (ms) Read Bandwidth User 2 User 1 4KB Random Writes 128MB Sequential Reads At different intensities • 40 writes/s • 80 writes/s • 160 writes/s • 320 writes/s
49 Manylogs @ MSST ‘16 Ordered Data Manylogs 10x Bandwidth Speed-up 14x Latency Speed-up at 320 IOPS! 600 500 Average sync Latency (ms) 80% Ratio of Max Bandwidth 400 60% 300 40% 200 20% 100 0% 0 40 80 160 320 Random Writes per Second (IOPS) per Disk
50 Manylogs @ MSST ‘16 More in the paper q Block-Level Manylogs q Other workloads § Sequential Writes § “varmail” § More Traces q Log Size q Logged Write Size q Mapping Table
51 Manylogs @ MSST ‘16 Manylogs q Reserved log spaces uniformly across the disk q Redirect small writes to the nearest log q Can help with NoSQL , SMR , RAID , and more! q Provide up to 5x speed-up on average J J J J J J J J
52 Manylogs @ MSST ‘16 Manylogs Bandwidth Latency Speed-up Speed-up vs. Ordered 3.7x 5.7x vs. Adaptive 2.7x 2.0x vs. Single-log SMR 1.3x
53 Manylogs @ MSST ‘16 Thank you! Questions? http://ucare.cs.uchicago.edu
Recommend
More recommend
