[PPT] - Manylogs Improving CMR/SMR Disk Bandwidth & Latency Tiratat PowerPoint Presentation

SLIDE 1

Manylogs

Improving CMR/SMR Disk Bandwidth & Latency

† Tiratat Patana-anake, Vincentius Martin†, Nora Sandler, Cheng Wu, and Haryadi S. Gunawi

SLIDE 2 Manylogs @ MSST ‘16 2

User 1 Big Read Got 100% of the read bandwidth

SLIDE 3 Many 3

User 2 Big Read User 1 Big Read Got 50% of the read bandwidth

Fair Share

SLIDE 4 Manylogs @ MSST ‘16 4

1 2 3 N

…

1/N bandwidth Still Fair!

SLIDE 5

Our

Manylogs @ MSST ‘16 5

User 1 Big Read Oh no! 5% bandwidth?! Small Durable Writes User 2

SLIDE 6

Our m

Manylogs @ MSST ‘16 6

User 1 Big Read More Bandwidth Please! Small Durable Writes User 2 Faster Latency Please!

SLIDE 7

J J

Data Journaling

Manylogs @ MSST ‘16 7

Ordered Journaling

Seek Journal Big Read Small Writes J J J J J J

SLIDE 8

Data Journaling

Manylogs @ MSST ‘16 8

Ordered Journaling

J J Seek Journal Big Read Small Writes J J J J J J

SLIDE 9

Data Journaling

Manylogs @ MSST ‘16 9

Ordered Journaling

Problems with Current Journaling

SLIDE 10 Manylogs @ MSST ‘16 10

Problems with Current Journaling

J

Ordered Journaling Data Jo

First Write Second Write

SLIDE 11

Introducing Manylogs

Manylogs @ MSST ‘16 11

Single Log Manylogs

J J J J J J J J 10 MB 100 MB

SLIDE 12

J J J J Seek Journal Big Read Small Writes J J J J

Manylogs

Manylogs @ MSST ‘16 12

J J J J

Small writes made durable to the nearest log without seeking

SLIDE 13

Manylogs

Manylogs @ MSST ‘16 13

q Reserved log spaces uniformly across the

disk

§ 10 MB every 100 MB

q Follow the disk head (last big I/O) q Redirect Small Writes (e.g. ≤ 256 KB)

§ Nearest log: log closest to last big I/O

q Sequential Writes are left untouched

J J J J J J J 10 MB 100 MB

SLIDE 14 Manylogs @ MSST ‘16 14

Manylogs

Increased Read Throughput Reduced Write Latency

SLIDE 15

Where are logs on the disk?

Manylogs @ MSST ‘16 15

J J J J J J J 10 MB 100 MB

The log space = whole platter

SLIDE 16

Where are logs on the disk?

Manylogs @ MSST ‘16 16

J J J J J J J 10 MB 100 MB

The log space = whole platter

SLIDE 17 Manylogs @ MSST ‘16 17

Same cylinder = No seek!

J J J J J J J

Log for others in the same cylinder

SLIDE 18

Disk

User 1 128MB Sequential Reads 4KB Random Writes User 2 Latency (ms) At different intensities writes/s writes/s writes/s

320 writes/s

Ordered vs. Data vs. Adaptive vs. Manylogs

Ratio of Max Read Bandwidth

Manylogs @ MSST ‘16 18

SLIDE 19

Adaptive Journaling

Manylogs @ MSST ‘16 19

q Middle ground between ordered

journaling and data journaling

q Single-log design q Prabhakaran et al., ATC ‘05

SLIDE 20

40 80 160 320 Random Writes per Second (IOPS)

Results

Manylogs @ MSST ‘16 20

Ordered Adaptive Data Manylogs 0% 20% 40% 60% 80% 100% Ratio of Max Bandwidth

SLIDE 21

Results

Manylogs @ MSST ‘16 21

Ordered Adaptive Data Manylogs 0% 20% 40% 60% 80% 100 Ratio of Max Bandwidth Random Writes per Second (IOPS) 40 80 160 320 57% at 40 IOPS 5% at 320 IOPS

SLIDE 22

Results

Manylogs @ MSST ‘16 22

Ordered Adaptive Data Manylogs 0% 20% 40% 60% 80% 100% Ratio of Max Bandwidth Random Writes per Second (IOPS) 40 80 160 320 9% at 320 IOPS 73% at 40 IOPS

SLIDE 23

Results

Manylogs @ MSST ‘16 23

Ordered Adaptive Data Manylogs 0% 20% 40% 60% 80% 100% Ratio of Max Bandwidth Random Writes per Second (IOPS) 40 80 160 320

SLIDE 24 Manylogs @ MSST ‘16 24

Ordered daptive Data Manylogs 60% 80% 100%

f Max Bandwidth

Random Writes per Second (IOPS) 40 80 160 320

Manylogs gives the most bandwidth 50% of max bandwidth at extreme IOPS Hey! What about my latency?

SLIDE 25

Results

Manylogs @ MSST ‘16 25

Ordered Adaptive Data Manylogs 100 200 300 400 500 600 Average sync Latency (ms) Random Writes per Second (IOPS) 40 80 160 320

SLIDE 26

Results

Manylogs @ MSST ‘16 26

100 200 300 400 500 600 Average sync Latency (ms) Random Writes per Second (IOPS) 40 80 160 320 Ordered Adaptive Data Manylogs

SLIDE 27

Results

Manylogs @ MSST ‘16 27

Random Writes per Second (IOPS) 40 80 160 320 Average sync Latency (ms) 100 200 300 400 500 600 Ordered Adaptive Data Manylogs

SLIDE 28

Results

Manylogs @ MSST ‘16 28

Random Writes per Second (IOPS) 40 80 160 320 Average sync Latency (ms) 100 200 300 400 500 600 Ordered Adaptive Data Manylogs

SLIDE 29

Results

Manylogs @ MSST ‘16 29

Random Writes per Second (IOPS) 40 80 160 320 Average sync Latency (ms) 100 200 300 400 500 600

WOW! Fast latency at extreme IOPS!

Ordered Adaptive Data Manylogs

SLIDE 30

Results

Manylogs @ MSST ‘16 30

Ordered Adaptive Data Manylogs 0% 20% 40% 60% 80% 100% Ratio of Max Bandwidth Random Writes per Second (IOPS) 40 80 160 320 Average sync Latency (ms) 100 200 300 400 500 600

SLIDE 31

Results

Manylogs @ MSST ‘16 31

SLIDE 32

User 1 128MB Sequential Reads “fileserver” User 2 Latency (ms)

Using Filebench
Multi-threaded
2, 4, 8 instances

Ratio of Max Read Bandwidth

Manylogs @ MSST ‘16 32

SLIDE 33 Manylogs @ MSST ‘16 33

Ordered Data

gs

0% 20% 40% 60% 80% 100% Ratio of Max Bandwidth fileserver Instances 2 4 8 Operation Latency (ms) 1000 2000 4000 3000

Manylogs provides the best outcomes!

SLIDE 34

q “Lazy” or

“Off-hours”

q Rarely full

because just small writes are redirected

q Log Swapping

Manylogs

Checkpointing

Manylogs @ MSST ‘16 34

q Periodically

§ Usually every 5 secs

q Journal can get

filled fast because all writes are in the journal!

Data Journaling

SLIDE 35

Hot Area! Cold Area!

Log Swapping

Manylogs @ MSST ‘16 35

J J J J Seek Journal Big Read Small Writes J J J J J J J J

SLIDE 36

Integrations

Manylogs @ MSST ‘16 36

q File System (MLFS)

§ Durability-Only Mode (O_DUR)

q SMR Disk (MLSMR) q RAID

SLIDE 37

Cassandra Write Path

Manylogs @ MSST ‘16 37

Writes SSTable Commit Log Memtable Disk Memory

SLIDE 38

SSTable Memtable Commit Log

Cassandra Write Path

Manylogs @ MSST ‘16 38

Writes Disk Memory

SLIDE 39

Disk Memory Ideally SYNC write Flush Triggered SSTable Memtable Commit Log

Cassandra Write Path

Manylogs @ MSST ‘16 39

Writes SSTable Memtable Commit Log Requires Fast Durability Temp File No location needed But in practice, background write e.g. every 10 seconds Commit Log

Random writes are the problem

😗

SLIDE 40

pen(file, O_DUR);

Manylogs @ MSST ‘16 40

q Need fast durability but not location

constraints

q Content of files will be put in Manylogs

regardless of the write size

q Never checkpoint their content q Random writes are not a problem

anymore!

SLIDE 41 Manylogs @ MSST ‘16 41

User 1 HDFS Ratio of Max Read Bandwidth MongoDB User 2 Latency (ms)

1 instance
2 instances
4 instances

SLIDE 42 Manylogs @ MSST ‘16 42

Data Manylogs 0% 20% 40% 60% 80% Ratio of Max Bandwidth MongoDB Write Latency (ms) 50 100 150 200 250 MongoDB Instances (w/Default flush period) 1DB 2DB 4DB

Most Bandwidth & Lowest Latency with Manylogs

SLIDE 43

Manylogs & SMR

Manylogs @ MSST ‘16 43

One non-shingled surface = log space

SLIDE 44

Manylogs & SMR

Manylogs @ MSST ‘16 44

Shingled Band Non-Shingled Tracks

SLIDE 45 Manylogs @ MSST ‘16 45

User 1 Build Server (WBS) Trace Latency (ms) Back-end Server (LM-TBE) Trace User 2 Latency (ms)

SLIDE 46

20 40 60 80 100 ML-SMR SL-SMR

Manylogs @ MSST ‘16 46

Manylogs SMR (MLSMR) Single-log SMR (SLSMR) 20 40 60 80 100 Percentile Latency (ms) 10 80 30 50 40

SMR

SLIDE 47

Manylogs & RAID

Manylogs @ MSST ‘16 47 Mingzhe Hao, Gokul Soundararajan, Deepak Kenchammana-Hosekote, Andrew A. Chien, and Haryadi S.

Gunawi. "The T

ail at Store: A Revelation from Millions of Hours of Disk and SSD Deployments." FAST’16.

User 1 Big Read Small Durable Writes User 2

Disk #1 Disk #2 Disk #3 Disk #4

Max Bandwidth! Bandwidth drops up to 50%

SLIDE 48

User 1 128MB Sequential Reads 4KB Random Writes User 2 Latency (ms) At different intensities

40

writes/s

80

writes/s

160 writes/s
320 writes/s

Ratio of Max Read Bandwidth

Manylogs @ MSST ‘16 48

SLIDE 49 Manylogs @ MSST ‘16 49

Ordered Data Manylogs 0% 20% 40% 60% 80% Ratio of Max Bandwidth Random Writes per Second (IOPS) per Disk 40 80 160 320 Average sync Latency (ms) 100 200 300 400 500 600

10x Bandwidth Speed-up 14x Latency Speed-up at 320 IOPS!

SLIDE 50