On Optimally Partitioning Variable-Byte Index Data Giulio Ermanno - - PowerPoint PPT Presentation
On Optimally Partitioning Variable-Byte Index Data Giulio Ermanno Pibiri Rossano Venturini University of Pisa and ISTI-CNR University of Pisa and ISTI-CNR Pisa, Italy Pisa, Italy giulio.pibiri@di.unipi.it rossano.venturini@unipi.it
Melbourne, 17/05/2018
1
Rossano Venturini
University of Pisa and ISTI-CNR Pisa, Italy rossano.venturini@unipi.it
Giulio Ermanno Pibiri
University of Pisa and ISTI-CNR Pisa, Italy giulio.pibiri@di.unipi.it
2
house is red red is always good the the is boy hungry is boy red house is the always hungry
2
house is red red is always good the the is boy hungry is boy red house is the always hungry
T = {always, boy, good, house, hungry, is, red, the}
t1 t2 t3 t4 t5 t6 t7 t8
2
house is red red is always good the the is boy hungry is boy red house is the always hungry
2 1 3 4 5
T = {always, boy, good, house, hungry, is, red, the}
t1 t2 t3 t4 t5 t6 t7 t8
2
house is red red is always good the the is boy hungry is boy red house is the always hungry
2 1 3 4 5
T = {always, boy, good, house, hungry, is, red, the}
t1 t2 t3 t4 t5 t6 t7 t8
Lt1=[1, 3] Lt2=[4, 5] Lt3=[1] Lt4=[2, 3] Lt5=[3, 5] Lt6=[1, 2, 3, 4, 5] Lt7=[1, 2, 4] Lt8=[2, 3, 5]
2
house is red red is always good the the is boy hungry is boy red house is the always hungry
2 1 3 4 5
T = {always, boy, good, house, hungry, is, red, the}
t1 t2 t3 t4 t5 t6 t7 t8
Lt1=[1, 3] Lt2=[4, 5] Lt3=[1] Lt4=[2, 3] Lt5=[3, 5] Lt6=[1, 2, 3, 4, 5] Lt7=[1, 2, 4] Lt8=[2, 3, 5]
2
3
3
house is red red is always good the the is boy hungry is boy red house is the always hungry
T = {always, boy, good, house, hungry, is, red, the}
2 1 3 4 5
Lt1=[1, 3]
t1 t2 t3 t4 t5 t6 t7 t8
Lt2=[4, 5] Lt3=[1] Lt4=[2, 3] Lt5=[3, 5] Lt6=[1, 2, 3, 4, 5] Lt7=[1, 2, 4] Lt8=[2, 3, 5]
3
q = {boy, is, the}
house is red red is always good the the is boy hungry is boy red house is the always hungry
T = {always, boy, good, house, hungry, is, red, the}
2 1 3 4 5
Lt1=[1, 3]
t1 t2 t3 t4 t5 t6 t7 t8
Lt2=[4, 5] Lt3=[1] Lt4=[2, 3] Lt5=[3, 5] Lt6=[1, 2, 3, 4, 5] Lt7=[1, 2, 4] Lt8=[2, 3, 5]
3
q = {boy, is, the}
house is red red is always good the the is boy hungry is boy red house is the always hungry
T = {always, boy, good, house, hungry, is, red, the}
2 1 3 4 5
Lt1=[1, 3]
t1 t2 t3 t4 t5 t6 t7 t8
Lt2=[4, 5] Lt3=[1] Lt4=[2, 3] Lt5=[3, 5] Lt6=[1, 2, 3, 4, 5] Lt7=[1, 2, 4] Lt8=[2, 3, 5]
3
q = {boy, is, the}
house is red red is always good the the is boy hungry is boy red house is the always hungry
T = {always, boy, good, house, hungry, is, red, the}
2 1 3 4 5
Lt1=[1, 3]
t1 t2 t3 t4 t5 t6 t7 t8
Lt2=[4, 5] Lt3=[1] Lt4=[2, 3] Lt5=[3, 5] Lt6=[1, 2, 3, 4, 5] Lt7=[1, 2, 4] Lt8=[2, 3, 5]
3
q = {boy, is, the}
house is red red is always good the the is boy hungry is boy red house is the always hungry
T = {always, boy, good, house, hungry, is, red, the}
2 1 3 4 5
Lt1=[1, 3]
t1 t2 t3 t4 t5 t6 t7 t8
Lt2=[4, 5] Lt3=[1] Lt4=[2, 3] Lt5=[3, 5] Lt6=[1, 2, 3, 4, 5] Lt7=[1, 2, 4] Lt8=[2, 3, 5] q = {good, hungry}
3
q = {boy, is, the}
house is red red is always good the the is boy hungry is boy red house is the always hungry
T = {always, boy, good, house, hungry, is, red, the}
2 1 3 4 5
Lt1=[1, 3]
t1 t2 t3 t4 t5 t6 t7 t8
Lt2=[4, 5] Lt3=[1] Lt4=[2, 3] Lt5=[3, 5] Lt6=[1, 2, 3, 4, 5] Lt7=[1, 2, 4] Lt8=[2, 3, 5] q = {good, hungry}
3
q = {boy, is, the}
house is red red is always good the the is boy hungry is boy red house is the always hungry
T = {always, boy, good, house, hungry, is, red, the}
2 1 3 4 5
Lt1=[1, 3]
t1 t2 t3 t4 t5 t6 t7 t8
Lt2=[4, 5] Lt3=[1] Lt4=[2, 3] Lt5=[3, 5] Lt6=[1, 2, 3, 4, 5] Lt7=[1, 2, 4] Lt8=[2, 3, 5] q = {good, hungry}
4
4
Binary Interpolative Coding Variable-Byte (VByte) Family
5
6
6
SIMD (Single Instruction Multiple Data)
7
7
Sensibly far away from bit-level effectiveness. BIC: 3.8 bpi on Gov2 PEF: 4.1 bpi on Gov2
8
8
13 14 15 16 17 18 19 20 21 22 23 24
9
9
Stores a fixed amount to bits, say F, for each partition.
9
Dynamic Programming (DP)
guarantee
Stores a fixed amount to bits, say F, for each partition.
9
time space Dynamic Programming (DP)
guarantee
Stores a fixed amount to bits, say F, for each partition.
9
DP Approximation
time space Dynamic Programming (DP)
guarantee
Stores a fixed amount to bits, say F, for each partition.
9
DP Approximation
Our solution
time space Dynamic Programming (DP)
guarantee
Stores a fixed amount to bits, say F, for each partition.
10
Why is it so difficult?
10
Why is it so difficult?
10
Why is it so difficult?
splitting Elias-Fano VByte [0,10) 4 bpi [0,5)[5,10) [4][5] bpi [0,6)[6,10) [5][2] bpi
10
Why is it so difficult?
splitting Elias-Fano VByte [0,10) 4 bpi [0,5)[5,10) [4][5] bpi [0,6)[6,10) [5][2] bpi
Different costs to represent the same integers! —> DP
10
Why is it so difficult?
splitting Elias-Fano VByte [0,10) 4 bpi [0,5)[5,10) [4][5] bpi [0,6)[6,10) [5][2] bpi
Different costs to represent the same integers! —> DP
8 bpi [8][8] bpi [8][8] bpi
Costs do NOT change if we consider different splittings.
11
VByte Bin
11
VByte Bin
11
If VByte is winning
have to try any split.
VByte Bin
11
If VByte is winning
have to try any split.
VByte Bin
11
If VByte is winning
have to try any split.
VByte Bin
11
If VByte is winning
have to try any split.
VByte Bin
11
If VByte is winning
have to try any split.
VByte Bin
11
If VByte is winning
have to try any split.
VByte Bin
11
If VByte is winning
have to try any split.
VByte Bin
11
If VByte is winning
have to try any split.
VByte Bin
11
If VByte is winning
have to try any split.
VByte Bin
11
If VByte is winning
have to try any split.
VByte Bin
11
If VByte is winning
have to try any split.
VByte Bin
11
If VByte is winning
have to try any split.
VByte Bin
11
If VByte is winning
have to try any split.
VByte Bin
11
If VByte is winning
have to try any split.
VByte Bin
11
If VByte is winning
have to try any split.
VByte Bin
11
If VByte is winning
have to try any split.
VByte Bin
11
If VByte is winning
have to try any split.
VByte Bin
11
If VByte is winning
have to try any split.
VByte Bin
11
If VByte is winning
have to try any split.
VByte Bin
We pay F bits for each partition!
F bits
12
gain = VByte - Bin g
12
gain = VByte - Bin g
12
gain = VByte - Bin g
Optimal.
12
gain = VByte - Bin g
Optimal.
Valid for ANY point-wise encoder.
12
gain = VByte - Bin g
Optimal.
Valid for ANY point-wise encoder. Very low constant factors.
12
gain = VByte - Bin g
Optimal.
Valid for ANY point-wise encoder. Very low constant factors. Linear time and constant space.
13
14
14
14
14
15
15
Speed NOT affected by partitioning.
15
Speed NOT affected by partitioning.
15
Speed NOT affected by partitioning.
16
16
16
17
18
Dataset Uncompressed Compressed Gov2 46GB 4GB (~11X) ClueWeb09 128GB 14GB (~9X)
19
Dataset Uncompressed Compressed Gov2 46GB 4GB (~11X) ClueWeb09 128GB 14GB (~9X)
19
Can we put everything on disk ? Memory hierarchy!
Dataset Uncompressed Compressed Gov2 46GB 4GB (~11X) ClueWeb09 128GB 14GB (~9X)
19
Can we put everything on disk ? Memory hierarchy!
https://blogs.dropbox.com/tech/2016/09/improving-the-performance-of-full-text-search/ See, for example:
Dataset Uncompressed Compressed Gov2 46GB 4GB (~11X) ClueWeb09 128GB 14GB (~9X)
19
Can we put everything on disk ? Memory hierarchy!
https://blogs.dropbox.com/tech/2016/09/improving-the-performance-of-full-text-search/ See, for example: