SLIDE 1
You are running out of memory
2
Hybrid Indexes Huanchen Zhang You are running out of memory 2 You - - PowerPoint PPT Presentation
Hybrid Indexes Huanchen Zhang You are running out of memory 2 You - - PowerPoint PPT Presentation
Reducing the Storage Overhead of Main-Memory OLTP Databases with Hybrid Indexes Huanchen Zhang You are running out of memory 2 You are running out of memory 2 ? Buy more You are running out of memory 2 TPC-C on -Store Memory Limit =
SLIDE 2
SLIDE 3
You are running out of memory
2
SLIDE 4
Buy more
?
You are running out of memory
2
SLIDE 5
2M 4M 6M 8M 10M 20K 60K
Transactions Executed Throughput TPC-C on
- Store
3
Memory (GB)
Disk tuples In-memory tuples Indexes
4 8 Memory Limit = 5GB
SLIDE 6
4
SLIDE 7
The better way: Use memory more efficiently
5
SLIDE 8
Indexes are LARGE
Benchmark % space for index
TPC-C Voter Articles
58% 55% 34%
Hybrid Index
34% 41% 18%
6
SLIDE 9
Our Contributions The hybrid index architecture The Dual-Stage Transformation Applied to 4 index structures
- B+tree
- Masstree
7
- Skip List
- Adaptive Radix Tree (ART)
Performance Space
30 – 70%
SLIDE 10
2M 4M 6M 8M 10M 20K 60K
Transactions Executed Throughput (txn/s) Did we solve this problem?
TPC-C on
- Store
Stay tuned
8
SLIDE 11
How do hybrid indexes achieve memory savings ?
9
Static
SLIDE 12
dynamic stage static stage
Hybrid Index: a dual-stage architecture
10
SLIDE 13
dynamic stage static stage write merge
Inserts are batched in the dynamic stage
11
SLIDE 14
dynamic stage static stage
Reads search the stages in order
12
SLIDE 15
dynamic stage static stage read
A Bloom filter improves read performance
13
SLIDE 16
dynamic stage static stage read write merge Memory-efficient Skew-aware
14
~ ~ ~ ~ ~ ~ ~ ~
SLIDE 17
dynamic stage static stage merge
The Dual-Stage Transformation
15
SLIDE 18
Compaction Reduction Compression
The Dynamic-to-Static Rules
16
SLIDE 19
2 4 4 1 2 a b 6 8 10 3 c 4 d 5 5 e f 5 g 6 h 7 i 8 j 9 k 10 l 11 m 12 n
Compaction: minimize # of memory blocks
17
SLIDE 20
1 2 3 a b c 4 5 6 d h 7 8 9 i j k 10 11 12 l m n e f g 3 6 9 17
Compaction: minimize # of memory blocks
SLIDE 21
1 2 3 a b c 4 5 6 d h 7 8 9 i j k 10 11 12 l m n e f g 3 6 9
Reduction: minimize structural overhead
18
SLIDE 22
2 4 4 1 2 a b 6 8 10 3 4 c d 5 5 e f 5 6 g h 7 8 i j 9 10 k l 11 12 m n 1 2 3 a b c 4 5 6 d h 7 8 9 i j k 10 11 12 l m n e f g 3 6 9 18
Reduction: minimize structural overhead
SLIDE 23
dynamic stage static stage merge
The Dual-Stage Transformation
19
SLIDE 24
dynamic stage static stage merge
The Dual-Stage Transformation
19
Merge Questions:
1. Partial? 2. When? 3. Blocking?
? ? ?
SLIDE 25
2M 4M 6M 8M 10M 20K 60K
Transactions Executed Throughput (txn/s) Did we solve this problem?
TPC-C on
- Store
B+tree
20
SLIDE 26
2M 4M 6M 8M 10M 20K 60K
Transactions Executed Throughput (txn/s)
60K 20K
Yes, we improved the DBMS’s capacity!
TPC-C on
- Store
B+tree Hybrid
20
SLIDE 27
Transactions Executed Throughput (txn/s)
20K 60K 20K 60K
Memory (GB)
2M 4M 6M 8M 10M
TPC-C on
- Store
4 4 8 8
B+tree Hybrid B+tree Hybrid Disk tuples In-memory tuples Indexes
21
SLIDE 28
Transactions Executed Throughput (txn/s)
20K 60K 20K 60K
Memory (GB)
2M 4M 6M 8M 10M
TPC-C on
- Store
4 4 8 8
B+tree Hybrid B+tree Hybrid Disk tuples In-memory tuples Indexes
21
SLIDE 29
Transactions Executed Throughput (txn/s)
20K 60K 20K 60K
Memory (GB)
2M 4M 6M 8M 10M
TPC-C on
- Store
4 4 8 8
B+tree Hybrid B+tree Hybrid Disk tuples In-memory tuples Indexes
21
SLIDE 30
Transactions Executed Throughput (txn/s)
20K 60K 20K 60K
Memory (GB)
2M 4M 6M 8M 10M
TPC-C on
- Store
4 4 8 8
B+tree Hybrid B+tree Hybrid Disk tuples In-memory tuples Indexes
21
SLIDE 31
Transactions Executed Throughput (txn/s)
20K 60K 20K 60K
Memory (GB)
2M 4M 6M 8M 10M
TPC-C on
- Store
4 4 8 8
B+tree Hybrid B+tree Hybrid Disk tuples In-memory tuples Indexes
21
SLIDE 32
Transactions Executed Throughput (txn/s)
20K 60K 20K 60K
Memory (GB)
2M 4M 6M 8M 10M
TPC-C on
- Store
4 4 8 8
B+tree Hybrid B+tree Hybrid Disk tuples In-memory tuples Indexes
21
Take Away:
Larger working set in memory Higher throughput Memory saved by indexes
SLIDE 33
This is just the BEGINNING
22
SLIDE 34
Conclusions The hybrid index architecture The Dual-Stage Transformation Applied to 4 index structures GENERAL PRACTICAL USEFUL
23
- B+tree
- Masstree
- Skip List
- Adaptive Radix Tree (ART)
SLIDE 35