Fast Dictionary-based Compression for Inverted Indexes Giulio - - PowerPoint PPT Presentation
Fast Dictionary-based Compression for Inverted Indexes Giulio Ermanno Pibiri Matthias Petri Alistair Mo ff at The University of Pisa The University of Melbourne The University of Melbourne and ISTI-CNR Melbourne, Australia Melbourne,
12/02/2019
Matthias Petri
The University of Melbourne Melbourne, Australia
Giulio Ermanno Pibiri
The University of Pisa and ISTI-CNR Pisa, Italy
Alistair Moffat
The University of Melbourne Melbourne, Australia
house is red red is always good the the is boy hungry is boy red house is the always hungry
V = {always, boy, good, house, hungry, is, red, the}
t1 t2 t3 t4 t5 t6 t7 t8
2 1 3 4 5
Lt1=[1, 3] Lt2=[4, 5] Lt3=[1] Lt4=[2, 3] Lt5=[3, 5] Lt6=[1, 2, 3, 4, 5] Lt7=[1, 2, 4] Lt8=[2, 3, 5]
Interpolative Variable-Byte + SIMD
Space Time
Interpolative Variable-Byte + SIMD
Space Time
Can we inherit both advantages?
RQ
Patterns of d-gaps are repetitive.
Patterns of d-gaps are repetitive.
Gov2
c1 c2 c3 c4 c5 c6 c6 c7 e c8 c9 c10 c11
… … input stream
b b b b
reference of b bits
access
fixed-to-fixed arrangement
2b
l + 1
c1 c2 c3 c4 c5 c6 c6 c7 e c8 c9 c10 c11
… … input stream
b b b b
reference of b bits
access
fixed-to-fixed arrangement
2b
l + 1
c1 c2 c3 c4 c5 c6 c6 c7 e c8 c9 c10 c11
… … input stream
b b b b
reference of b bits
access
fixed-to-fixed arrangement
2b
l + 1
c1 c2 c3 c4 c5 c6 c6 c7 e c8 c9 c10 c11
… … input stream
b b b b
reference of b bits
access
fixed-to-fixed arrangement
2b
l + 1
1/3 of the time is saved
Packed dictionary structure Exploiting string overlap
1
Optimal block parsing
2
Multiple dictionaries
3
C++ code available at https://github.com/jermp/dint Machine Intel Xeon 6144 processor, 512 GiB RAM, Linux 4.13.0 Compiler gcc 7.2.0 (with all optimizations) Datasets
l = 16
b = 16
Chapter 6 and 7 of my Ph.D. thesis. http://pages.di.unipi.it/pibiri/papers/phd_thesis.pdf (more datasets, comparisons, query timings)