SSDAlloc: Hybrid SSD/RAM Memory Management Made Easy
Anirudh Badam Vivek S. Pai
Princeton University 03/31/2011
1
SSDAlloc: Hybrid SSD/RAM Memory Management Made Easy Anirudh Badam - - PowerPoint PPT Presentation
SSDAlloc: Hybrid SSD/RAM Memory Management Made Easy Anirudh Badam Vivek S. Pai Princeton University 03/31/2011 1 Memory in Networked Systems 2 Memory in Networked Systems As a cache to reduce pressure on the disk Memcache like
Anirudh Badam Vivek S. Pai
1
2
2
inline data-deduplicators
effectively
2
3
8 16 32 64 128 256 512 1024 37.5 75 112.5 150
$ / GB (Total Cost) Total DRAM
3
8 16 32 64 128 256 512 1024 37.5 75 112.5 150
$ / GB (Total Cost) Total DRAM
3
8 16 32 64 128 256 512 1024 37.5 75 112.5 150
$ / GB (Total Cost)
3
8 16 32 64 128 256 512 1024 37.5 75 112.5 150
$ / GB (Total Cost)
3
4
decreasing
4
decreasing
capacity 7200 rpm drives
4
5
memory
5
memory
5
memory
memory?
5
6
6
6
6
6
7
Disk
Memory
7
Disk Block Addressable
Memory Byte Addressable
7
Disk Block Addressable Directly Addressed
Memory Byte Addressable Virtually Addressed
7
Disk Block Addressable Directly Addressed High Latency
Memory Byte Addressable Virtually Addressed Low Latency
7
Disk Block Addressable Directly Addressed High Latency Non-volatile
Memory Byte Addressable Virtually Addressed Low Latency Volatile
7
Disk Block Addressable Directly Addressed High Latency Non-volatile
Memory Byte Addressable Virtually Addressed Low Latency Volatile
7
Disk Block Addressable Directly Addressed High Latency Non-volatile
Memory Byte Addressable Virtually Addressed Low Latency Volatile
7
Disk Block Addressable Directly Addressed High Latency Non-volatile
Memory Byte Addressable Virtually Addressed Low Latency Volatile Flash has low latency
7
Disk Block Addressable Directly Addressed High Latency Non-volatile
Memory Byte Addressable Virtually Addressed Low Latency Volatile Flash has low latency
7
8
8
performance
performance
was designed as a “dead page storage”
8
1 2 3 4 13 14 15 16 5 6 7 8 9 10 11 12 1 2 3 4 17 18 19 20 33 34 35 36 49 50 51 52 5 6 7 8 21 22 23 24 37 38 39 40 53 54 55 56 9 10 11 12 25 26 27 28 41 42 43 44 57 58 59 60 29 30 31 32 45 46 47 48 61 62 63 64
13 14 15 16 29 30 31 32
9
1 2 3 4 13 14 15 16 5 6 7 8 9 10 11 12 1 2 3 4 17 18 19 20 33 34 35 36 49 50 51 52 5 6 7 8 21 22 23 24 37 38 39 40 53 54 55 56 9 10 11 12 25 26 27 28 41 42 43 44 57 58 59 60 29 30 31 32 45 46 47 48 61 62 63 64
13 14 15 16 read 29 30 31 32
9
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 17 18 19 20 33 34 35 36 49 50 51 52 5 6 7 8 21 22 23 24 37 38 39 40 53 54 55 56 9 10 11 12 25 26 27 28 41 42 43 44 57 58 59 60 29 30 31 32 45 46 47 48 61 62 63 64
13 14 15 16 read 29 30 31 32
9
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 17 18 19 20 33 34 35 36 49 50 51 52 5 6 7 8 21 22 23 24 37 38 39 40 53 54 55 56 9 10 11 12 25 26 27 28 41 42 43 44 57 58 59 60 29 30 31 32 45 46 47 48 61 62 63 64
13 14 15 16 read 29 30 31 32
9
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 17 18 19 20 33 34 35 36 49 50 51 52 5 6 7 8 21 22 23 24 37 38 39 40 53 54 55 56 9 10 11 12 25 26 27 28 41 42 43 44 57 58 59 60 29 30 31 32 45 46 47 48 61 62 63 64
13 14 15 16 read 29 30 31 32
9
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 17 18 19 20 33 34 35 36 49 50 51 52 5 6 7 8 21 22 23 24 37 38 39 40 53 54 55 56 9 10 11 12 25 26 27 28 41 42 43 44 57 58 59 60 29 30 31 32 45 46 47 48 61 62 63 64
13 14 15 16 read 29 30 31 32 write
9
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 17 18 19 20 33 34 35 36 49 50 51 52 5 6 7 8 21 22 23 24 37 38 39 40 53 54 55 56 9 10 11 12 25 26 27 28 41 42 43 44 57 58 59 60 29 30 31 32 45 46 47 48 61 62 63 64
13 14 15 16 read 29 30 31 32 write
9
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 17 18 19 20 33 34 35 36 49 50 51 52 5 6 7 8 21 22 23 24 37 38 39 40 53 54 55 56 29 30 31 32 45 46 47 48 61 62 63 64
13 14 15 16 read 29 30 31 32 write
9
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 17 18 19 20 33 34 35 36 49 50 51 52 5 6 7 8 21 22 23 24 37 38 39 40 53 54 55 56 9 10 11 12 25 26 27 28 41 42 43 44 57 58 59 60 29 30 31 32 45 46 47 48 61 62 63 64
13 14 15 16 read 29 30 31 32 write
9
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 17 18 19 20 33 34 35 36 49 50 51 52 5 6 7 8 21 22 23 24 37 38 39 40 53 54 55 56 9 10 11 12 25 26 27 28 41 42 43 44 57 58 59 60 29 30 31 32 45 46 47 48 61 62 63 64
13 14 15 16 read 29 30 31 32 write free()
9
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 17 18 19 20 33 34 35 36 49 50 51 52 5 6 7 8 21 22 23 24 37 38 39 40 53 54 55 56 9 10 11 12 25 26 27 28 41 42 43 44 57 58 59 60 29 30 31 32 45 46 47 48 61 62 63 64
13 14 15 16 read 29 30 31 32 write free()
9
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 17 18 19 20 33 34 35 36 49 50 51 52 9 10 11 12 25 26 27 28 41 42 43 44 57 58 59 60 29 30 31 32 45 46 47 48 61 62 63 64
13 14 15 16 read 29 30 31 32 write free()
9
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 17 18 19 20 33 34 35 36 49 50 51 52 5 6 7 8 21 22 23 24 37 38 39 40 53 54 55 56 9 10 11 12 25 26 27 28 41 42 43 44 57 58 59 60 29 30 31 32 45 46 47 48 61 62 63 64
13 14 15 16 read 29 30 31 32 write free()
9
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 17 18 19 20 33 34 35 36 49 50 51 52 5 6 7 8 21 22 23 24 37 38 39 40 53 54 55 56 9 10 11 12 25 26 27 28 41 42 43 44 57 58 59 60 29 30 31 32 45 46 47 48 61 62 63 64
13 14 15 16 read 29 30 31 32 write free()
Indirection Table
(log structured page store)
In the OS
9
10
application object granularity
flash memory device
10
application object granularity
flash memory device
10
MyObject* obj = malloc( sizeof( MyObject ) );
free( obj );
MyObjectID oid = createObject( sizeof( MyObject ) ); MyObject* obj = malloc( sizeof( MyObject ) ); readObject( oid, obj );
writeObject( oid, obj ); free( obj );
11
12
12
12
12
information
12
Application Virtual Memory (Object per page - OPP) Physical Memory SSD
13
Application Virtual Memory (Object per page - OPP) Physical Memory SSD
Memory Manager: Creates 64 objects of 1KB size
13
Application Virtual Memory (Object per page - OPP) Physical Memory SSD
Memory Manager: Creates 64 objects of 1KB size
1 2 3 4
61 62 63 64
13
Application Virtual Memory (Object per page - OPP) Physical Memory SSD
Memory Manager: Creates 64 objects of 1KB size
1 2 3 4
61 62 63 64 Page Buffer 1 12 33
13
Application Virtual Memory (Object per page - OPP) Physical Memory SSD
Memory Manager: Creates 64 objects of 1KB size
1 2 3 4
61 62 63 64 Page Buffer 1 12 33 RAM Object Cache
2 3 4 5 6 7 8 9 10 11 13 14 15 16 17 18 19 20 21 22 23 24 25 26
13
Application Virtual Memory (Object per page - OPP) Physical Memory SSD
Memory Manager: Creates 64 objects of 1KB size
1 2 3 4
61 62 63 64 Page Buffer 1 12 33
RAM Object Cache
2 3 4 5 6 7 8 9 10 11 13 14 15 16 17 18 19 20 21 22 23 24 25 26
13
14
Object Per Page (OPP) Memory Page (MP) Data Entity Application Defined Objects 4KB objects (like pages)
14
Object Per Page (OPP) Memory Page (MP) Data Entity Application Defined Objects 4KB objects (like pages) Memory Manager Pool Allocator Coalescing Allocator
14
Object Per Page (OPP) Memory Page (MP) Data Entity Application Defined Objects 4KB objects (like pages) Memory Manager Pool Allocator Coalescing Allocator Virtual Memory
page_size
14
Object Per Page (OPP) Memory Page (MP) Data Entity Application Defined Objects 4KB objects (like pages) Memory Manager Pool Allocator Coalescing Allocator Virtual Memory
page_size Physical Memory Separate Page Buffer & RAM Object Cache No such separation
14
Object Per Page (OPP) Memory Page (MP) Data Entity Application Defined Objects 4KB objects (like pages) Memory Manager Pool Allocator Coalescing Allocator Virtual Memory
page_size Physical Memory Separate Page Buffer & RAM Object Cache No such separation SSD Usage Log-structured Object Store Log-structured Page Store
14
Object Per Page (OPP) Memory Page (MP) Data Entity Application Defined Objects 4KB objects (like pages) Memory Manager Pool Allocator Coalescing Allocator Virtual Memory
page_size Physical Memory Separate Page Buffer & RAM Object Cache No such separation SSD Usage Log-structured Object Store Log-structured Page Store Code Changes Minimal changes restricted to memory allocation No changes needed
14
15
Application Virtual Memory SSD
15
Application Virtual Memory SSD
Page Buffer
15
from RAM object cache/SSD
RAM wastage (from OPP)
Application Virtual Memory SSD
Page Buffer Demand Fetching
15
from RAM object cache/SSD
RAM wastage (from OPP)
Application Virtual Memory SSD
Page Buffer Demand Fetching
15
from RAM object cache/SSD
RAM wastage (from OPP)
Application Virtual Memory SSD
Page Buffer Demand Fetching
15
from RAM object cache/SSD
RAM wastage (from OPP)
Application Virtual Memory SSD
Page Buffer Demand Fetching
15
from RAM object cache/SSD
RAM wastage (from OPP)
handler
Application Virtual Memory SSD
Page Buffer
Demand Fetching
15
from RAM object cache/SSD
RAM wastage (from OPP)
handler
flushes dirty objects to the SSD in LRU order
Application Virtual Memory SSD
Page Buffer Dirty Objects
Demand Fetching
15
16
Virtual Memory SSD RAM Object Cache
Dirty Objects
16
16
16
16
16
17
17
18
Technique Write Logging Access < 4KB Finegrained GC Avoid DRAM Pollution High Performance Programming Ease SSD Swap
SSD Swap (Write Logged) ✔
Application Rewrite
18
Technique Write Logging Access < 4KB Finegrained GC Avoid DRAM Pollution High Performance Programming Ease SSD Swap
SSD Swap (Write Logged) ✔
Application Rewrite
SSDAlloc
18
19
Overhead Source Max Latency TLB Miss (DRAM Read) 0.014 μSec Object Table Lookup 0.046 μSec Page Materialization 0.138 μSec Page Dematerialization 0.172 μSec Signal Handling 0.666 μSec Combined Overhead 0.833 μSec
19
Overhead Source Max Latency TLB Miss (DRAM Read) 0.014 μSec Object Table Lookup 0.046 μSec Page Materialization 0.138 μSec Page Dematerialization 0.172 μSec Signal Handling 0.666 μSec Combined Overhead 0.833 μSec
19
Overhead Source Max Latency TLB Miss (DRAM Read) 0.014 μSec Object Table Lookup 0.046 μSec Page Materialization 0.138 μSec Page Dematerialization 0.172 μSec Signal Handling 0.666 μSec Combined Overhead 0.833 μSec
19
20
20
Intel X25-M
20
Original Modified SSDAlloc-OP c-OPP’s gain vs Application Original LOC Modified LOC Swap SSDAlloc-MP Memcached 11,193 21 5.5 - 17.4x 1.4 - 3.5x B+Tree Index 477 15 4.3 - 12.7x 1.4 - 3.2x Packet Cache 1,540 9 4.8 - 10.1x 1.3 - 2.3x HashCache 20,096 36 5.3 - 17.1x 1.3 - 3.3x
21
Original Modified SSDAlloc-OP c-OPP’s gain vs Application Original LOC Modified LOC Swap SSDAlloc-MP Memcached 11,193 21 5.5 - 17.4x 1.4 - 3.5x B+Tree Index 477 15 4.3 - 12.7x 1.4 - 3.2x Packet Cache 1,540 9 4.8 - 10.1x 1.3 - 2.3x HashCache 20,096 36 5.3 - 17.1x 1.3 - 3.3x
than when compared to the traditional VM style applications
21
22
22
3.75 7.5 11.25 15 SSDAlloc-OPP over Swap SSDAlloc-OPP over SSDAlloc-MP Throughput Gain Factor All Reads 25% Reads 50% Reads 75% Reads All Writes
22
23
23
23
12500 25000 37500 50000 128 256 512 1024 2048 4096 8192 Throughput (req/sec)
SSDAlloc-OPP SSDAlloc-MP Swap
Average Object Size
24
24
7500 15000 22500 30000 SSD-swap SSDAlloc-MP SSDAlloc-OPP Thtoughput (req/sec)
RiData Kingston Intel X25-E Intel X25-V Intel X25-M
24
25
VM system
25
VM system
25
26