succinct trie indexes made practical
play

Succinct Trie Indexes Made Practical Huanchen Zhang David G. - PowerPoint PPT Presentation

Dec 30, 2023 •729 likes •1.49k views

Succinct Trie Indexes Made Practical Huanchen Zhang David G. Andersen, Michael Kaminsky, Andrew Pavlo, Kimberly Keeton DRAM price wont fall forever Price Year Memory-efficient data structures are helpful Smaller data structures More

  1. Succinct Trie Indexes Made Practical Huanchen Zhang David G. Andersen, Michael Kaminsky, Andrew Pavlo, Kimberly Keeton

  2. DRAM price won’t fall forever Price Year

  3. Memory-efficient data structures are helpful Smaller data structures More data resident in faster memory Better performance + lower costs

  4. The limit: information-theoretic lower bound (ITLB) The minimum # of bits required to distinguish any object in a class !"# 2 % bits |S| = n

  5. The limit: information-theoretic lower bound (ITLB) The minimum # of bits required to distinguish any object in a class !"# 2 % bits |S| = n % !"# 2 + − (+ − 1)!"# 2 (+ − 1) bits |n-node trie of degree k| '()* ⁄ +% + 1 = (

  6. The limit: information-theoretic lower bound (ITLB) The minimum # of bits required to distinguish any object in a class !"# 2 % bits |S| = n % !"# 2 + − (+ − 1)!"# 2 (+ − 1) bits |n-node trie of degree k| '()* ⁄ +% + 1 = ( 256 9.44n

  7. The limit: information-theoretic lower bound (ITLB) The minimum # of bits required to distinguish any object in a class !"# 2 % bits |S| = n % !"# 2 + − (+ − 1)!"# 2 (+ − 1) bits |n-node trie of degree k| '()* ⁄ +% + 1 = ( 256 9.44n FST = 10n

  8. Succinct Data Structures Use # of bits close to ITLB Suppose ITLB = L bits Implicit: L + O(1) Succinct: L + o(L) Compact: O(L) FST

  9. Why aren’t succinct data structures popular? Read-only Log-structured design Slow Complex

  10. Existing succinct tries are slow 50M 64-bit integer keys Memory Lookup Latency including key suffixes 3 1.5 2 1 GB us 1 0.5 0 0 ART tx-trie PDT ART tx-trie PDT

  11. Fast Succinct Trie (FST) is fast and small 50M 64-bit integer keys Memory Lookup Latency including key suffixes 3 1.5 2 1 GB us 1 0.5 0 0 ART tx-trie PDT FST ART tx-trie PDT FST

  12. Encoding Mechanism

  13. 3 ways to succinctly encode ordinal trees Ordinal tree: a rooted tree where each node can have an arbitrary # of children in order 0 1 2 3 4 5 6 7 8 9 A B C D E

  14. 3 ways to succinctly encode ordinal trees ! $" ≈ 2' bits |n-node ordinal tree| = C n = " "#! 0 1 2 3 4 5 6 7 8 9 A B C D E

  15. 3 ways to succinctly encode ordinal trees LOUDS: level-ordered unary degree sequence 0 110 1 2 10 110 3 4 5 1110 110 110 6 7 8 9 A B C 0 10 0 0 0 10 0 D E 0 0

  16. 3 ways to succinctly encode ordinal trees LOUDS: 110 10 110 1110 110 110 0 10 0 0 0 10 0 0 0 0 110 1 2 10 110 3 4 5 1110 110 110 6 7 8 9 A B C 0 10 0 0 0 10 0 D E 0 0

  17. 3 ways to succinctly encode ordinal trees BP: balanced parenthesis 0 1 2 3 4 5 6 7 8 9 A B C D E

  18. 3 ways to succinctly encode ordinal trees BP: ( ( ( ( ) ( ( ) ) ( ) ) ) ( ( ( ) ( ) ) ( ( ( ) ) ( ) ) ) ) 0 1 2 3 4 5 6 7 8 9 A B C D E

  19. 3 ways to succinctly encode ordinal trees 3 0 2 BP: ( ( ( ( ) ( ( ) ) ( ) ) ) ( ( ( ) ( ) ) ( ( ( ) ) ( ) ) ) ) 8 6 D 9 A E C 7 B 4 0 1 5 1 2 3 4 5 6 7 8 9 A B C D E

  20. 3 ways to succinctly encode ordinal trees DFUDS: depth-first unary degree sequence 0 1 2 3 4 5 6 7 8 9 A B C D E

  21. 3 ways to succinctly encode ordinal trees DFUDS: ( ( ) ( ) ( ( ( ) ) ( ) ) ) ( ( ) ( ( ) ) ) ( ( ) ( ) ) ) 0 1 2 3 4 5 6 7 8 9 A B C D E

  22. 3 ways to succinctly encode ordinal trees DFUDS: ( ( ) ( ) ( ( ( ) ) ( ) ) ) ( ( ) ( ( ) ) ) ( ( ) ( ) ) ) 0 1 3 6 7 D 8 2 4 9 A 5 B E C 0 1 2 3 4 5 6 7 8 9 A B C D E

  23. LOUDS-Sparse: succinctly encode tries L: f s t $ a o r r s t y p i y $ t e p f t HC: 1010 1 110 100 0 10 000 0 s S: 1001 0 101001 0 10 101 0 $ r o v 1 v 1 v 2 v 3 v 4 v 5 v 6 v 7 v 8 v 9 v 10 v 11 V: a r t y p y v 2 i s $ t e p v 3 v 4 v 5 v 6 v 7 v 8 v 9 v 10 v 11

  24. LOUDS-Sparse: succinctly encode tries L: f s t $ a o r r s t y p i y $ t e p f t HC: 1010 1 110 100 0 10 000 0 s S: 1001 0 101001 0 10 101 0 $ r o v 1 v 1 v 2 v 3 v 4 v 5 v 6 v 7 v 8 v 9 v 10 v 11 V: a Why LOUDS? r t y p y v 2 i s 1. Fast tree nav. 2. Good label locality $ t e p v 3 v 4 v 5 v 6 v 7 3. Easy implementation v 8 v 9 v 10 v 11

  25. Rank & select on bit-vectors 0 5 10 15 bv: 1 0 0 1 0 1 0 1 0 0 1 0 1 0 1 0 1 0 rank(bv, i) = # of 1’s in bv up to position i select(bv, i) = position of the ith 1 in bv Examples: rank(bv, 7) = 4 select(bv, 7) = 14

  26. Compute rank & select in constant time The classic algorithm for computing rank bv

  27. Compute rank & select in constant time The classic algorithm for computing rank !" # $ bits super block = … bv

  28. Compute rank & select in constant time The classic algorithm for computing rank #$ " % bits super block = … … … bv ! "#$% bits basic block =

  29. Compute rank & select in constant time The classic algorithm for computing rank #$ " % bits super block = … … … bv ! "#$% bits basic block = per super block cumulative rank

  30. Compute rank & select in constant time The classic algorithm for computing rank #$ " % bits super block = … … … bv ! "#$% bits basic block = per super block per basic block cumulative rank rank in super block

  31. Compute rank & select in constant time The classic algorithm for computing rank #$ " % bits super block = … … … bv ! "#$% bits basic block = per super block per basic block within super block cumulative rank rank in super block all possible queries

  32. Compute rank & select in constant time The classic algorithm for computing rank #$ " % bits super block = … … … bv ! "#$% bits basic block = per super block per basic block within super block cumulative rank rank in super block all possible queries & '( 2 *

  33. Compute rank & select in constant time The classic algorithm for computing rank #$ " % bits super block = … … … bv ! "#$% bits basic block = per super block per basic block within super block cumulative rank rank in super block all possible queries & & ! '( 2 * " '(*

  34. Compute rank & select in constant time The classic algorithm for computing rank #$ " % bits super block = … … … bv ! "#$% bits basic block = per super block per basic block within super block cumulative rank rank in super block all possible queries & & remaining ! bits '( 2 * " '(*

  35. Compute rank & select in constant time The classic algorithm for computing rank #$ " % bits super block = … … … bv ! "#$% bits basic block = per super block per basic block within super block cumulative rank rank in super block all possible queries & & remaining O (1) time ! bits '( 2 * " '(*

  36. Compute rank & select in constant time The classic algorithm for computing rank #$ " % bits super block = … … … bv ! "#$% bits basic block = per super block per basic block within super block cumulative rank rank in super block all possible queries + + remaining O (1) time ! bits () 2 * " ()* O ( % O ( % #$% ) #$% ()()*) space: o (*) O ( * ()* ()()*)

  37. Compute rank & select in constant time The classic algorithm for computing rank #$ " % bits super block = … … … bv ! "#$% bits basic block = per super block per basic block within super block cumulative rank rank in super block all possible queries + + remaining O (1) time ! bits () 2 * " ()* O ( % O ( % #$% ) #$% ()()*) space: o (*) O ( * ()* ()()*) Select is similar but trickier, often based on rank structures

  38. Tree navigation relies on rank & select 0 5 10 15 L: f s t $ a o r r s t y p i y $ t e p f t s HC: 1010 1 110 100 0 10 000 0 S: 1001 0 101001 0 10 101 0 $ r o v 1 a v 1 v 2 v 3 v 4 v 5 v 6 v 7 v 8 v 9 v 10 v 11 V: child(i) = select(S, rank(HC, i)+1) r t y p y v 2 i s parent(i) = select(S, rank(S, i)-1) value(i) = i - rank(HC, i) $ t e p v 3 v 4 v 5 v 6 v 7 v 8 v 9 v 10 v 11

  39. Tree navigation relies on rank & select 0 5 10 15 L: f s t $ a o r r s t y p i y $ t e p f t s HC: 1010 1 110 100 0 10 000 0 S: 1001 0 101001 0 10 101 0 $ r o v 1 a v 1 v 2 v 3 v 4 v 5 v 6 v 7 v 8 v 9 v 10 v 11 V: child(i) = select(S, rank(HC, i)+1) r t y p y v 2 i s parent(i) = select(S, rank(S, i)-1) value(i) = i - rank(HC, i) $ t e p v 3 v 4 v 5 v 6 v 7 v 8 v 9 v 10 v 11

  40. Tree navigation relies on rank & select 0 5 10 15 L: f s t $ a o r r s t y p i y $ t e p f t s HC: 1010 1 110 100 0 10 000 0 S: 1001 0 101001 0 10 101 0 $ r o v 1 a v 1 v 2 v 3 v 4 v 5 v 6 v 7 v 8 v 9 v 10 v 11 V: child(i) = select(S, rank(HC, i)+1) r t y p y v 2 i s value(i) = i - rank(HC, i) $ t e p v 3 v 4 v 5 v 6 v 7 v 8 v 9 v 10 v 11

  41. Tree navigation relies on rank & select 0 5 10 15 L: f s t $ a o r r s t y p i y $ t e p f t s HC: 1010 1 110 100 0 10 000 0 S: 1001 0 101001 0 10 101 0 $ r o v 1 a v 1 v 2 v 3 v 4 v 5 v 6 v 7 v 8 v 9 v 10 v 11 V: child(i) = select(S, rank(HC, i)+1) r t y p y v 2 i s value(i) = i - rank(HC, i) $ t e p v 3 v 4 v 5 v 6 v 7 v 8 v 9 v 10 v 11

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

ADVANCED DATABASE SYSTEMS OLTP Indexes (Trie Data Structures) @ Andy_Pavlo // 15- 721 //

ADVANCED DATABASE SYSTEMS OLTP Indexes (Trie Data Structures) @ Andy_Pavlo // 15- 721 //

Lect ure # 07 ADVANCED DATABASE SYSTEMS OLTP Indexes (Trie Data Structures) @ Andy_Pavlo // 15- 721 // Spring 2020 2 Latches B+Trees Judy Array ART Masstree 15-721 (Spring 2020) 3 LATCH IM PLEM ENTATIO N GOALS Small memory

1.21k views • 116 slides
ADVANCED DATABASE SYSTEMS OLTP Indexes (Trie Data Structures) @ Andy_Pavlo // 15- 721 //

ADVANCED DATABASE SYSTEMS OLTP Indexes (Trie Data Structures) @ Andy_Pavlo // 15- 721 //

Lect ure # 08 ADVANCED DATABASE SYSTEMS OLTP Indexes (Trie Data Structures) @ Andy_Pavlo // 15- 721 // Spring 2019 CMU 15-721 (Spring 2019) 2 Index Implementation Issues Judy Array ART Masstree CMU 15-721 (Spring 2019) 3 IN DEX IM

769 views • 76 slides
Su SuRF: : PRACTICAL RANGE FILTERING WITH FA FAST ST SU SUCCINCT TRIES Huanchen Zhang Hu

Su SuRF: : PRACTICAL RANGE FILTERING WITH FA FAST ST SU SUCCINCT TRIES Huanchen Zhang Hu

Su SuRF: : PRACTICAL RANGE FILTERING WITH FA FAST ST SU SUCCINCT TRIES Huanchen Zhang Hu Hy Hyeontaek Lim, Viktor r Leis, David G. Anders rsen Michael Kaminsky, Kimberl rly Keeton, Andre rew Pa Pavlo Fi Filters answer approximate

525 views • 50 slides
A Practical Succinct Data Structure for Tree-Like Graphs Johannes Fischer TU Dortmund, Germany

A Practical Succinct Data Structure for Tree-Like Graphs Johannes Fischer TU Dortmund, Germany

A Practical Succinct Data Structure for Tree-Like Graphs Johannes Fischer TU Dortmund, Germany joint work with Daniel Peters (PTB Berlin) leading adj(x,y) neighbours(x) deg(x) terms [bits] n 2 lg arbitrary O(1) O(1) O(#in) or

587 views • 57 slides
In-memory processing of big data via succinct data structures Rajeev Raman University of

In-memory processing of big data via succinct data structures Rajeev Raman University of

In-memory processing of big data via succinct data structures Rajeev Raman University of Leicester SDP Workshop, University of Cambridge Introduction Succinct Data Structuring Succinct Tries Applications & Libraries End Overview

321 views • 18 slides
Module 7: Creating and Maintaining Indexes Overview Creating Indexes Creating Index

Module 7: Creating and Maintaining Indexes Overview Creating Indexes Creating Index

Module 7: Creating and Maintaining Indexes Overview Creating Indexes Creating Index Options Maintaining Indexes Introduction to Statistics Querying the sysindexes Table Setting Up Indexes Using the Index Tuning Wizard

554 views • 27 slides
Indexes 1 Demo 2 Indexes Index = data structure

Indexes 1 Demo 2 Indexes Index = data structure

Indexes 1 Demo 2 Indexes Index = data structure used to speed access to tuples of a rela7on, given values of one or more

1.08k views • 15 slides
Computational Approaches for Stochastic Shortest Path on Succinct MDPs Krishnendu Chatterjee 1

Computational Approaches for Stochastic Shortest Path on Succinct MDPs Krishnendu Chatterjee 1

Computational Approaches for Stochastic Shortest Path on Succinct MDPs Krishnendu Chatterjee 1 Hongfei Fu 2 Amir Goharshady 1 Nastaran Okati 3 1 IST Austria 2 Shanghai Jiao Tong University 3 Ferdowsi University of Mashhad IJCAI 2018 Succinct MDPs

350 views • 32 slides
Module 6: Planning Indexes Overview Introduction to Indexes Index Architecture How

Module 6: Planning Indexes Overview Introduction to Indexes Index Architecture How

Module 6: Planning Indexes Overview Introduction to Indexes Index Architecture How SQL Server Retrieves Stored Data How SQL Server Maintains Index and Heap Structures Deciding Which Columns to Index Introduction to Indexes

352 views • 31 slides
FUNCTIONALLY OBLIVIOUS (AND SUCCINCT) Edward Kmett BUILDING BETTER TOOLS Cache-Oblivious

FUNCTIONALLY OBLIVIOUS (AND SUCCINCT) Edward Kmett BUILDING BETTER TOOLS Cache-Oblivious

FUNCTIONALLY OBLIVIOUS (AND SUCCINCT) Edward Kmett BUILDING BETTER TOOLS Cache-Oblivious Algorithms Succinct Data Structures RAM MODEL Almost everything you do in Haskell assumes this model Good for ADTs, but not a realistic

673 views • 49 slides
LOG-STRUCTURED MERGE-TRIE PART 1 Xingbo Wu and Yuehai Xu, Wayne State University; Zili Shao, The

LOG-STRUCTURED MERGE-TRIE PART 1 Xingbo Wu and Yuehai Xu, Wayne State University; Zili Shao, The

LOG-STRUCTURED MERGE-TRIE PART 1 Xingbo Wu and Yuehai Xu, Wayne State University; Zili Shao, The Hong Kong Polytechnic University; Song Jiang, Wayne State University Presented by: Joel Friberg LSM-Trie Overview 32MB Htable KV-item

264 views • 10 slides
PI is not at least as succinct as MODS Nikolay Kaleyski July 7, 2017 Nikolay Kaleyski PI is not

PI is not at least as succinct as MODS Nikolay Kaleyski July 7, 2017 Nikolay Kaleyski PI is not

PI is not at least as succinct as MODS Nikolay Kaleyski July 7, 2017 Nikolay Kaleyski PI is not at least as succinct as MODS Known results in knowledge compilation A Knowledge Compilation Map, Adnan Darwiche & Pierre Marquis (2002)

814 views • 53 slides
Statistical Encoding of Succinct Data Structures alez 1 Gonzalo Navarro 1 Rodrigo Gonz 1

Statistical Encoding of Succinct Data Structures alez 1 Gonzalo Navarro 1 Rodrigo Gonz 1

Outline Statistical Encoding of Succinct Data Structures alez 1 Gonzalo Navarro 1 Rodrigo Gonz 1 Department of Computer Science Universidad de Chile Combinatorial Pattern Matching, 2006 Gonz alez, Navarro Statistical Encoding of Succinct

1.17k views • 80 slides
RECIPE : Converting Concurrent DRAM Indexes to Persistent-Memory Indexes Se Kwon Lee, Jayashree

RECIPE : Converting Concurrent DRAM Indexes to Persistent-Memory Indexes Se Kwon Lee, Jayashree

RECIPE : Converting Concurrent DRAM Indexes to Persistent-Memory Indexes Se Kwon Lee, Jayashree Mohan, Sanidhya Kashyap * , Taesoo Kim, Vijay Chidambaram *On the job market 1 Persistent Memory (PM) New storage class memory technology

1.01k views • 70 slides
LSM SM-Tr Trie ie: : An An LSM SM-tre ree-base ased d Ultra-Lar arge ge Ke Key-Va Valu

LSM SM-Tr Trie ie: : An An LSM SM-tre ree-base ased d Ultra-Lar arge ge Ke Key-Va Valu

CSE 6350 File and Storage System Infrastructure in Data centers Supporting Internet-wide Services LSM SM-Tr Trie ie: : An An LSM SM-tre ree-base ased d Ultra-Lar arge ge Ke Key-Va Valu lue e St Store e for Small Data (Contd)

221 views • 11 slides
Modern OLTP Indexes (Part 2) 1 / 43 Modern OLTP Indexes (Part 2) Recap Recap 2 / 43 Modern OLTP

Modern OLTP Indexes (Part 2) 1 / 43 Modern OLTP Indexes (Part 2) Recap Recap 2 / 43 Modern OLTP

Modern OLTP Indexes (Part 2) Modern OLTP Indexes (Part 2) 1 / 43 Modern OLTP Indexes (Part 2) Recap Recap 2 / 43 Modern OLTP Indexes (Part 2) Recap Versioned Latch Coupling Optimistic coupling scheme where writers are not blocked on

610 views • 43 slides
Utilizing Precision Navigation and Autonomy to Support Combat Diver CONOPS 1 STIDD and Greensea

Utilizing Precision Navigation and Autonomy to Support Combat Diver CONOPS 1 STIDD and Greensea

Utilizing Precision Navigation and Autonomy to Support Combat Diver CONOPS 1 STIDD and Greensea Partnership STIDD EXPERTS IN DIVER PROPULSION Producing military submersibles since 1998 Most widely used two-man underwater mobility

763 views • 26 slides
Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks Fengda Zhu, Yi Zhu,

Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks Fengda Zhu, Yi Zhu,

MONASH INFORMATION TECHNOLOGY Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks Fengda Zhu, Yi Zhu, Xiaojun Chang, Xiaodan Liang Outline 1. Embodied Navigation 2. Vision-language Navigation Task 3. Related Works 4.

559 views • 21 slides
UAV prototype for terrain dose-rate mapping Eng. Diego Garca (CNEA-UTN)

UAV prototype for terrain dose-rate mapping Eng. Diego Garca (CNEA-UTN)

UAV prototype for terrain dose-rate mapping Eng. Diego Garca (CNEA-UTN) diegogarcia@cae.cnea.gov.ar Eng. Lucio Martinez Garbino (CNEA-UTN) luciojmg@cae.cnea.gov.ar 1 Origins of the project AGE CNEA-CAE requires a surface dose rate map

241 views • 13 slides
Architectural Analysis of Microsoft Dynamics NAV Tom Hvitved hvitved@diku.dk Department of

Architectural Analysis of Microsoft Dynamics NAV Tom Hvitved hvitved@diku.dk Department of

Architectural Analysis of Microsoft Dynamics NAV Tom Hvitved hvitved@diku.dk Department of Computer Science University of Copenhagen 3gERP Workshop, November 18th 2008 Outline Introduction 1 Motivation 2 Architectural Analysis 3 Table

708 views • 31 slides
Mapping FOSDEM for accessibility The team Laura https://codingcatgirl.de/ Created

Mapping FOSDEM for accessibility The team Laura https://codingcatgirl.de/ Created

Mapping FOSDEM for accessibility The team Laura https://codingcatgirl.de/ Created C3NAV Shin Ice Linux Sysadmin (Germany) Fosdem volunteer for different years Johan Van de Wauw Developing sensor solutions at

434 views • 17 slides
Motion Planning for Example Finding Evasive Targets in a Cluttered Environment cleared region

Motion Planning for Example Finding Evasive Targets in a Cluttered Environment cleared region

Motion Planning for Example Finding Evasive Targets in a Cluttered Environment cleared region robot robots visibility region + hidin hiding region 1 2 3 Map Building 4 5 6 1 2 Problem Assumptions Target is unpredictable

184 views • 7 slides
Diagnostics of turbulence and bulk motions in ICM Irina Zhuravleva KIPAC, Stanford University

Diagnostics of turbulence and bulk motions in ICM Irina Zhuravleva KIPAC, Stanford University

Diagnostics of turbulence and bulk motions in ICM Irina Zhuravleva KIPAC, Stanford University E. Churazov, A. Schekochihin, S. Allen, P. Arevalo, A. Fabian, W. Forman, M. Gaspari, A. Kravtsov, E. Lau, D. Nagai, S. Nelson, I. Parrish, J.

463 views • 20 slides
Hao Su Image world Shape world How humans represent 3D in mind? Mental rotation by Roger N.

Hao Su Image world Shape world How humans represent 3D in mind? Mental rotation by Roger N.

Synthesize for Learning: Joint analysis of 2D images and 3D shapes Hao Su Image world Shape world How humans represent 3D in mind? Mental rotation by Roger N. Shepard, National Science Medal Laurate, Stanford and Lynn Cooper, Professor at

1k views • 63 slides

More recommend