Hybrid Indexes D a v i d G . A n d e r s e n A n d - - PowerPoint PPT Presentation

Reducing the Storage Overhead of Main-Memory OLTP Databases with

Hybrid Indexes

Huanchen Zhang

C

• a

u t h

• r

s :

D a v i d G . A n d e r s e n A n d r e w P a v l

• M

i c h a e l K a m i n s k y L i n M a R u i S h e n

You are running out of memory

2

You are running out of memory

2

Buy more

?

You are running out of memory

2

2M 4M 6M 8M 10M 20K 60K

Transactions Executed Throughput TPC-C on

• Store

3

Memory (GB)

Disk tuples In-memory tuples Indexes

4 8 Memory Limit = 5GB

4

The better way: Use memory more efficiently

5

Indexes are LARGE

Benchmark % space for index

TPC-C Voter Articles

58% 55% 34%

Hybrid Index

34% 41% 18%

6

Our Contributions The hybrid index architecture The Dual-Stage Transformation Applied to 4 index structures

• B+tree
• Masstree

7

• Skip List
• Adaptive Radix Tree (ART)

Performance Space

30 – 70%

2M 4M 6M 8M 10M 20K 60K

Transactions Executed Throughput (txn/s) Did we solve this problem?

TPC-C on

• Store

Stay tuned

8

How do hybrid indexes achieve memory savings ?

9

Static

dynamic stage static stage

Hybrid Index: a dual-stage architecture

10

dynamic stage static stage write merge

Inserts are batched in the dynamic stage

11

dynamic stage static stage read read

Reads search the stages in order

12

dynamic stage static stage read read

A Bloom filter improves read performance

13

dynamic stage static stage read read write merge Memory-efficient Skew-aware

14

~ ~ ~ ~ ~ ~ ~ ~

dynamic stage static stage merge

The Dual-Stage Transformation

15

Compaction Reduction Compression

The Dynamic-to-Static Rules

16

2 4 4 1 2 a b 6 8 10 3 c 4 d 5 5 e f 5 g 6 h 7 i 8 j 9 k 10 l 11 m 12 n

Compaction: minimize # of memory blocks

17

1 2 3 a b c 4 5 6 d h 7 8 9 i j k 10 11 12 l m n e f g 3 6 9 17

Compaction: minimize # of memory blocks

1 2 3 a b c 4 5 6 d h 7 8 9 i j k 10 11 12 l m n e f g 3 6 9

Reduction: minimize structural overhead

18

2 4 4 1 2 a b 6 8 10 3 4 c d 5 5 e f 5 6 g h 7 8 i j 9 10 k l 11 12 m n 1 2 3 a b c 4 5 6 d h 7 8 9 i j k 10 11 12 l m n e f g 3 6 9 18

Reduction: minimize structural overhead

dynamic stage static stage merge

The Dual-Stage Transformation

19

Merge Questions:

1. Partial? 2. When? 3. Blocking?

? ? ?

2M 4M 6M 8M 10M 20K 60K

Transactions Executed Throughput (txn/s) Did we solve this problem?

TPC-C on

• Store

B+tree

20

2M 4M 6M 8M 10M 20K 60K

Transactions Executed Throughput (txn/s)

60K 20K

Yes, we improved the DBMS’s capacity!

TPC-C on

• Store

B+tree Hybrid

20

Transactions Executed Throughput (txn/s)

20K 60K 20K 60K

Memory (GB)

2M 4M 6M 8M 10M

TPC-C on

• Store

4 4 8 8

B+tree Hybrid B+tree Hybrid Disk tuples In-memory tuples Indexes

21

Transactions Executed Throughput (txn/s)

20K 60K 20K 60K

Memory (GB)

2M 4M 6M 8M 10M

TPC-C on

• Store

4 4 8 8

B+tree Hybrid B+tree Hybrid Disk tuples In-memory tuples Indexes

21

Transactions Executed Throughput (txn/s)

20K 60K 20K 60K

Memory (GB)

2M 4M 6M 8M 10M

TPC-C on

• Store

4 4 8 8

B+tree Hybrid B+tree Hybrid Disk tuples In-memory tuples Indexes

21

Transactions Executed Throughput (txn/s)

20K 60K 20K 60K

Memory (GB)

2M 4M 6M 8M 10M

TPC-C on

• Store

4 4 8 8

B+tree Hybrid B+tree Hybrid Disk tuples In-memory tuples Indexes

21

Transactions Executed Throughput (txn/s)

20K 60K 20K 60K

Memory (GB)

2M 4M 6M 8M 10M

TPC-C on

• Store

4 4 8 8

B+tree Hybrid B+tree Hybrid Disk tuples In-memory tuples Indexes

21

Take Away:

Larger working set in memory Higher throughput Memory saved by indexes

This is just the BEGINNING

22

Conclusions The hybrid index architecture The Dual-Stage Transformation Applied to 4 index structures GENERAL PRACTICAL USEFUL

23

• B+tree
• Masstree
• Skip List
• Adaptive Radix Tree (ART)

1-844-88-CMUDB Toll-Free Hotline: