SLIDE 1
Scaling Log-Structured KV-Stores featuring Monkey and Dostoevsky - - PowerPoint PPT Presentation
Scaling Log-Structured KV-Stores featuring Monkey and Dostoevsky - - PowerPoint PPT Presentation
Scaling Log-Structured KV-Stores featuring Monkey and Dostoevsky SIGMOD17 / SIGMOD18 Niv Dayan Log-Structured KV-Stores Log-Structured KV-Stores Why Log-Structured KV-Stores? Why Log-Structured KV-Stores? fast writes Why Log-Structured
SLIDE 2
SLIDE 3
Log-Structured KV-Stores
SLIDE 4
Why Log-Structured KV-Stores?
SLIDE 5
Why Log-Structured KV-Stores?
fast writes
SLIDE 6
Why Log-Structured KV-Stores?
memory storage
SLIDE 7
Why Log-Structured KV-Stores?
SLIDE 8
Why Log-Structured KV-Stores?
SLIDE 9
block-addressable byte-addressable
Why Log-Structured KV-Stores?
SLIDE 10
write data
SLIDE 11
write data
SLIDE 12
write data
SLIDE 13
In-Place Writes
write data
SLIDE 14
In-Place Writes
B-trees write data
SLIDE 15
In-Place Writes
write data B-trees
SLIDE 16
Log-Structured Writes
SLIDE 17
Log-Structured Writes
buffer writes
SLIDE 18
Log-Structured Writes
buffer writes
SLIDE 19
Log-Structured Writes
buffer writes
SLIDE 20
Log-Structured Writes
buffer writes
SLIDE 21
Log-Structured Writes
buffer writes
SLIDE 22
Log-Structured KV-Stores
fast writes
buffer writes
SLIDE 23
massive data fast writes fast reads
Log-Structured KV-Stores
SLIDE 24
SLIDE 25
SLIDE 26
SLIDE 27
Background
SLIDE 28
Background The Log-Structured Merge-Tree
buffer
SLIDE 29
Background LSM-tree
buffer
SLIDE 30
buffer
SLIDE 31
buffer writes
SLIDE 32
buffer key value pairs
SLIDE 33
buffer Sherlock: key value Waldo: a fictional detective an inconspicuous traveler
SLIDE 34
buffer gets full
SLIDE 35
buffer 1 level sort & flush
SLIDE 36
buffer 1 level sort & flush sorted runs …
SLIDE 37
buffer 1 2 sort-merge
SLIDE 38
buffer 1 2 3 level exponentially increasing capacities level 1 level 2 level 3
- n
e I / O p e r r u n
SLIDE 39
buffer 1 2 3 level where’s Waldo b i n a r y s e a r c h i n g
SLIDE 40
buffer 1 2 3 level
- n
e I / O p e r r u n pointers where’s Waldo
SLIDE 41
buffer 1 2 3 level Bloom filters pointers where’s Waldo
SLIDE 42
buffer 1 2 3 level
true negative
Bloom filters pointers where’s Waldo
SLIDE 43
buffer 1 2 3 level
false positive true negative
Bloom filters pointers where’s Waldo
SLIDE 44
buffer 1 2 3 level
false positive true positive true negative
Bloom filters pointers where’s Waldo
SLIDE 45
buffer 1 2 3 Bloom filters merging frequency pointers
SLIDE 46
merging writes reads
SLIDE 47
merging writes reads
SLIDE 48
Leveling Tiering
read-optimized write-optimized
merging
SLIDE 49
Leveling Tiering
write-optimized read-optimized
SLIDE 50
Leveling Tiering gather
write-optimized read-optimized
SLIDE 51
Leveling Tiering merge & flush gather
write-optimized read-optimized
SLIDE 52
Leveling Tiering
write-optimized read-optimized
gather
SLIDE 53
Leveling Tiering merge
write-optimized read-optimized
gather
SLIDE 54
Leveling Tiering flush merge
write-optimized read-optimized
gather
SLIDE 55
Leveling Tiering merge
write-optimized read-optimized
gather
SLIDE 56
Leveling Tiering
write-optimized read-optimized
logR(N)
SLIDE 57
Leveling Tiering 1 run per level R runs per level
write-optimized read-optimized
size ratio logR(N)
SLIDE 58
Leveling Tiering
size ratio logR(N)
1 run per level R runs per level
write-optimized read-optimized
SLIDE 59
Leveling Tiering R runs per level 1 run per level
size ratio R
write-optimized read-optimized
SLIDE 60
1 run per level Leveling Tiering 1 run per level
size ratio R
write-optimized read-optimized
SLIDE 61
Leveling Tiering T runs per level 1 run per level
size ratio R
write-optimized read-optimized
SLIDE 62
1 run per level Leveling Tiering log sorted array O(lNl) runs per level
size ratio R
write-optimized read-optimized
SLIDE 63
Tiering Leveling log sorted array
SLIDE 64
Tiering Leveling log sorted array size ratio R
SLIDE 65
Tiering Leveling size ratio R log sorted array
SLIDE 66
Tiering Leveling log sorted array R R size ratio R
SLIDE 67
SLIDE 68
SLIDE 69
SLIDE 70
SLIDE 71
Dostoevsky
Monkey
SLIDE 72
Monkey: Optimal Navigable Key-Value Store SIGMOD17
SLIDE 73
Monkey: Optimal Navigable Key-Value Store SIGMOD17
Niv Dayan Manos Athanassoulis Stratos Idreos
SLIDE 74
Bloom filters data
Monkey: Optimal Navigable Key-Value Store SIGMOD17
SLIDE 75
x x x bits/entry Bloom filters data
SLIDE 76
x x x bits/entry Bloom filters data
SLIDE 77
Bloom filters false positive rate O(e-x) O(e-x) O(e-x) data
SLIDE 78
Bloom filters false positive rate O(e-x) O(e-x) O(e-x) O(e-x · logR(N)) I/O =
SLIDE 79
Bloom filters false positive rate O(e-x) O(e-x) O(e-x) O(e-x · logR(N)) I/O =
SLIDE 80
false positive rate O(e-x) O(e-x) O(e-x) Bloom filters most memory
SLIDE 81
false positive rate O(e-x) O(e-x) O(e-x) Bloom filters most memory saves at most 1 I/O!
SLIDE 82
reallocate
SLIDE 83
reallocate
SLIDE 84
same memory - fewer false positives
reallocate
SLIDE 85
false positive rates relax 0 < p0 < 1 0 < p1 < 1 0 < p2 < 1
SLIDE 86
f(p0, p1 …) f(p0, p1 …) false positive rates relax model 0 < p0 < 1 0 < p1 < 1 0 < p2 < 1 read cost memory footprint = =
SLIDE 87
read cost false positive rates relax model memory footprint 0 < p0 < 1 0 < p1 < 1 0 < p2 < 1
L
∑
1
pi −
L
∑
i
N TL−i ⋅ ln(pi) ln(2)2
= =
SLIDE 88
0 < p0 < 1 0 < p1 < 1 0 < p2 < 1 in terms of p0, p1 … model false positive rates relax
- ptimize
read cost memory footprint = =
L
∑
1
pi −
L
∑
i
N TL−i ⋅ ln(pi) ln(2)2
SLIDE 89
false positive rate O(e-x/R0) O(e-x/R1) O(e-x/R2) p0 ≈ p1 ≈ p2 ≈
SLIDE 90
O(e-x/R0) O(e-x) I/O = geometric progression false positive rate O(e-x/R2) O(e-x/R1)
SLIDE 91
O(e-x) I/O > O(e-x · logR(N))
SLIDE 92
O(e-x · logR(N)) O(e-x) I/O
SLIDE 93
number of entries (log scale) read latency (ms)
RocksDB Monkey
O(e-x · logR(N)) O(e-x) I/O
SLIDE 94
Existing Monkey
SLIDE 95
Existing Monkey Dostoevsky
SLIDE 96
tiering
Monkey
leveling
SLIDE 97
point writes I/O overheads with leveling long range short range
SLIDE 98
exponentially decreasing O(e-x) O(e-x/R) O(e-x/R2) point false positive rates
SLIDE 99
false positive rates point largest level O(e-x) O(e-x/R) O(e-x/R2)
SLIDE 100
largest level O(e-x)
writes long range short range point
SLIDE 101
target key range target range O(s) O(s/R) O(s/R2) long range
SLIDE 102
target key range target range largest level long range O(s) O(s/R) O(s/R2)
SLIDE 103
largest level largest level O(e-x) O(s)
point writes long range short range
SLIDE 104
short range target range 1 1 1
SLIDE 105
all levels target range 1 1 1 O(logR(N)) short range
SLIDE 106
largest level largest level
point
O(e-x) O(s) all levels O(logR(N))
writes long range short range
SLIDE 107
exponentially more work exponentially less frequent writes
SLIDE 108
exponentially more work exponentially less frequent writes
SLIDE 109
= = all levels more work less frequent writes
SLIDE 110
= = all levels writes
SLIDE 111
merge 1 writes
SLIDE 112
merge 2 writes
SLIDE 113
R merge writes
SLIDE 114
O(R) O(R) O(R) write-amplification writes
SLIDE 115
O(R · logR(N)) O(R) O(R) O(R) writes
SLIDE 116
O(e-x) O(s) O(logR(N)) O(R · logR(N)) O(s/R2) O(s/R) 1 1 1 O(s) O(e-x) O(e-x/R) O(e-x/R2) = = = + + + + + + largest level largest level all levels all levels
= + + long range point short range writes
O(R) O(R) O(R)
SLIDE 117
O(s) O(e-x)
largest level largest level all levels writes long range point
O(R) O(R) O(R)
SLIDE 118
O(s) O(e-x) largest level largest level all levels
long range point superfluous
O(R) O(R) O(R)
writes
SLIDE 119
merging at smaller levels is superfluous for point lookups and long range lookups
SLIDE 120
worse as data grows!
SLIDE 121
poor performance
SLIDE 122
poor performance lower device lifetime (on SSD)
SLIDE 123
Dostoevsky
SIGMOD18
SLIDE 124
Dostoevsky: Space-Time Optimized Evolvable Scalable Key-Value Store
SLIDE 125
Dostoevsky: Space-Time Optimized Evolvable Scalable Key-Value Store very write-optimized
SLIDE 126
Leveling Tiering
read-optimized write-optimized
SLIDE 127
Leveling Tiering
read-optimized write-optimized
Lazy Leveling
mixed-optimized
SLIDE 128
Lazy Leveling Leveling Tiering
SLIDE 129
Lazy Leveling merge to have at most 1 run merge when level fills
SLIDE 130
long range short range point writes
SLIDE 131
O(e-x) O(e-x/R2) O(e-x/R3) false positive rates point
SLIDE 132
false positive rates exponentially decreasing point O(e-x) O(e-x/R2) O(e-x/R3)
SLIDE 133
false positive rates largest level point O(e-x) O(e-x/R2) O(e-x/R3)
SLIDE 134
O(e-x) point
SLIDE 135
O(e-x) point O(e-x) O(e-x) O(e-x) O(logR(N) · R · e-x) with uniform FPRs
SLIDE 136
O(e-x)
point long range short range writes
SLIDE 137
target key range target range O(s) O(s/R) O(s/R2) long range
SLIDE 138
target key range target range largest level long range O(s) O(s/R) O(s/R2)
SLIDE 139
O(e-x) O(s)
point writes long range short range
SLIDE 140
1 O(R) O(1+R·(logR(N)-1)) target key range O(R) short range
SLIDE 141
O(1+R·(logR(N)-1)) O(e-x) O(s)
long range point writes short range
SLIDE 142
write-amplification O(1) O(1) O(R) writes
SLIDE 143
write-amplification O( R + logR(N) ) writes O(1) O(1) O(R)
SLIDE 144
O( R + logR(N) ) O(e-x) O(s)
long range point short range writes
O(1+R · (logR(N)-1))
SLIDE 145
O( R + logR(N) ) O(1+R · (logR(N)-1)) O(e-x) O(s)
Lazy Leveling Leveling
O(e-x) O(s) O(logR(N)) O( R · logR(N) )
= = V V long range point short range writes
SLIDE 146
O( R + logR(N) ) O(1+R · (logR(N)-1)) O(e-x) O(s)
Lazy Leveling Leveling
O(e-x) O(s) O(logR(N)) O( R · logR(N) )
Tiering
O( logR(N) ) O(R · logR(N)) O(R · e-x) O(R · s)
V V V V long range point short range writes
SLIDE 147
Leveling writes Lazy Leveling Tiering point
SLIDE 148
Leveling short range Lazy Leveling Tiering writes
SLIDE 149
Leveling long range Lazy Leveling Tiering writes
SLIDE 150
Leveling Tiering Lazy Leveling
SLIDE 151
Leveling Tiering Lazy Leveling
writes
SLIDE 152
Leveling Tiering Lazy Leveling
short range writes
SLIDE 153
Leveling Tiering
short range writes & point
Lazy Leveling
writes
SLIDE 154
Tiering Lazy Leveling
Fluid
Leveling
SLIDE 155
K runs Z runs
LSM-Tree Fluid
SLIDE 156
R runs 1 runs Lazy Leveling
LSM-Tree Fluid
SLIDE 157
R runs 1 runs long range short range point writes Lazy Leveling
SLIDE 158
R runs 1 runs long range short range point writes Lazy Leveling
- ptimize
SLIDE 159
2 runs 1 runs long range short range point writes Lazy Leveling
- ptimize
SLIDE 160
Leveling short range writes 1 runs 1 runs
- ptimize
long range point
SLIDE 161
Leveling short range writes 1 runs 1 runs long range point long range point
- ptimize
SLIDE 162
R runs 1 runs long range short range point writes Lazy Leveling
- ptimize
SLIDE 163
R runs 2 runs long range short range point Lazy Leveling
- ptimize
writes
SLIDE 164
R runs R runs long range short range point Tiering
- ptimize
writes
SLIDE 165
R runs R runs long range short range point Tiering writes
- ptimize
SLIDE 166
R runs 1 runs long range short range point writes Lazy Leveling
- ptimize
SLIDE 167
R runs 1 runs long range short range point writes Lazy Leveling
- ptimize
R size ratio
SLIDE 168
R runs 1 runs long range short range point writes Lazy Leveling
- ptimize
R size ratio
SLIDE 169
Fluid LSM-Tree
R size ratio K runs at smaller levels Z runs at largest level
SLIDE 170
Fluid LSM-Tree Tiering Lazy Leveling Leveling
SLIDE 171
SLIDE 172
0.5 1 normalized throughput (ops/s) 10−2 10−1 point lookups / updates
l e v e l i n g point lookups / updates 0.5 1 normalized throughput 1/100 1/10
SLIDE 173
0.5 1 normalized throughput (ops/s) 10−2 10−1 point lookups / updates
l e v e l i n g point lookups / updates 0.5 1 normalized throughput 1/100 1/10
SLIDE 174
0.5 1 normalized throughput (ops/s) 10−2 10−1 point lookups / updates
l e v e l i n g t i e r i n g point lookups / updates 0.5 1 normalized throughput 1/100 1/10
SLIDE 175
0.5 1 normalized throughput (ops/s) 10−2 10−1 point lookups / updates
l e v e l i n g t i e r i n g lazy leveling point lookups / updates 1/100 1/10 0.5 1 normalized throughput
SLIDE 176
0.5 1 normalized throughput (ops/s) 10−2 10−1 point lookups / updates
l e v e l i n g t i e r i n g lazy leveling Dostoevsky point lookups / updates 0.5 1 normalized throughput 1/100 1/10
SLIDE 177
0.5 1 normalized throughput (ops/s) 10−2 10−1 point lookups / updates
0.5 1 normalized throughput point lookups / updates Dostoevsky 1/100 1/10
SLIDE 178
0.5 1 normalized throughput (ops/s) 10−2 10−1 point lookups / updates
point lookups / updates 0.5 1 normalized throughput Dostoevsky Tuned RocksDB Monkey 1/100 1/10
SLIDE 179
0.5 1 normalized throughput (ops/s) 10−2 10−1 point lookups / updates
point lookups / updates 0.5 1 normalized throughput Dostoevsky Tuned RocksDB Monkey Untuned RocksDB 1/100 1/10
SLIDE 180
Conclusion
SLIDE 181
Conclusion
Bloom filters LSM-tree
SLIDE 182
Conclusion
Bloom filters LSM-tree
- ptimizes
memory allocation
SLIDE 183
Conclusion
Bloom filters LSM-tree removes superfluous merging
- ptimizes
memory allocation
SLIDE 184
Conclusion
SLIDE 185
Conclusion
SLIDE 186
Conclusion
SLIDE 187
Conclusion
t i e r i n g ( 1 9 9 7 ) leveling (1996)
SLIDE 188
Conclusion
little memory t i e r i n g ( 1 9 9 7 ) leveling (1996)
SLIDE 189
Conclusion
ample memory t i e r i n g ( 1 9 9 7 ) leveling (1996)
SLIDE 190
Conclusion
ample memory t i e r i n g ( 1 9 9 7 ) leveling (1996)
SLIDE 191
Conclusion
ample memory t i e r i n g ( 1 9 9 7 ) leveling (1996)
SLIDE 192
Conclusion
ample memory t i e r i n g ( 1 9 9 7 ) leveling (1996)
SLIDE 193
Conclusion
SLIDE 194
Conclusion
SLIDE 195
Conclusion
SLIDE 196
Conclusion
SLIDE 197
SLIDE 198