inducing suffix and lcp arrays in external memory
play

Inducing Suffix and LCP Arrays in External Memory Timo Bingmann, - PowerPoint PPT Presentation

Mar 16, 2024 •547 likes •1.28k views

Inducing Suffix and LCP Arrays in External Memory Timo Bingmann, Johannes Fischer, and Vitaly Osipov | January 7th, 2013 @ ALENEX13 I NSTITUTE OF T HEORETICAL I NFORMATICS A LGORITHMICS KIT University of the State of Baden-Wuerttemberg

  1. Inducing Suffix and LCP Arrays in External Memory Timo Bingmann, Johannes Fischer, and Vitaly Osipov | January 7th, 2013 @ ALENEX’13 I NSTITUTE OF T HEORETICAL I NFORMATICS – A LGORITHMICS KIT – University of the State of Baden-Wuerttemberg and www.kit.edu National Laboratory of the Helmholtz Association

  2. Abstract We consider text index construction in external memory (EM). Our first contribution is an inducing algorithm for suffix arrays in external memory. Practical tests show that this outperforms the previous best EM suffix sorter [Dementiev et al., ALENEX 2005] by a factor of about two in time and I/O-volume. Our second contribution is to augment the first algorithm to also construct the array of longest common prefixes (LCPs). This yields the first EM construction algorithm for LCP arrays. The overhead in time and I/O volume for this extended algorithm over plain suffix array construction is roughly two. The algorithms scale far beyond problem sizes previously considered in the literature (text size of 80 GiB using only 4 GiB of RAM in our experiments). Timo Bingmann, Johannes Fischer, and Vitaly Osipov – Inducing Suffix and LCP Arrays in External Memory Institute of Theoretical Informatics – Algorithmics January 7th, 2013 2/19

  3. Overview Introduction and Motivation 1 Evolution of Suffix Array Construction Algorithms History of LCP Construction Algorithms Example of the Inducing Step in the eSAIS Algorithm 2 Inducing the Suffix Array Inducing the LCP Array Finding Ranks of S ∗ -Suffixes Implementation and Experimental Results 3 Implementation Highlights Experiments – eSAIS vs. DC3 Timo Bingmann, Johannes Fischer, and Vitaly Osipov – Inducing Suffix and LCP Arrays in External Memory Institute of Theoretical Informatics – Algorithmics January 7th, 2013 3/19

  4. 0 1 2 3 4 5 6 7 8 9 10 11 12 Example T = [ cababcbababb$ ] i LCP i T i 0 0 c a b a b c b a b a b b $ 1 0 a b a b c b a b a b b $ 2 0 b a b c b a b a b b $ 3 0 a b c b a b a b b $ 4 0 b c b a b a b b $ 5 0 c b a b a b b $ 6 0 b a b a b b $ 7 0 a b a b b $ 8 0 b a b b $ 9 0 a b b $ 10 0 b b $ 11 0 b $ 12 - $ Timo Bingmann, Johannes Fischer, and Vitaly Osipov – Inducing Suffix and LCP Arrays in External Memory Institute of Theoretical Informatics – Algorithmics January 7th, 2013 4/19

  5. 0 1 2 3 4 5 6 7 8 9 10 11 12 Example T = [ cababcbababb$ ] SA i LCP i T SA i ... n 12 - $ 7 0 a b a b b $ 1 4 a b a b c b a b a b b $ 9 2 a b b $ 3 2 a b c b a b a b b $ 11 0 b $ 6 1 b a b a b b $ 8 3 b a b b $ 2 3 b a b c b a b a b b $ 10 1 b b $ 4 1 b c b a b a b b $ 0 0 c a b a b c b a b a b b $ 5 1 c b a b a b b $ Timo Bingmann, Johannes Fischer, and Vitaly Osipov – Inducing Suffix and LCP Arrays in External Memory Institute of Theoretical Informatics – Algorithmics January 7th, 2013 5/19

  6. 0 1 2 3 4 5 6 7 8 9 10 11 12 Example T = [ cababcbababb$ ] SA i LCP i T SA i ... n 12 - $ 7 0 a b a b b $ 1 4 a b a b c b a b a b b $ 9 2 a b b $ 3 2 a b c b a b a b b $ 11 0 b $ 6 1 b a b a b b $ 8 3 b a b b $ 2 3 b a b c b a b a b b $ 10 1 b b $ 4 1 b c b a b a b b $ 0 0 c a b a b c b a b a b b $ 5 1 c b a b a b b $ Timo Bingmann, Johannes Fischer, and Vitaly Osipov – Inducing Suffix and LCP Arrays in External Memory Institute of Theoretical Informatics – Algorithmics January 7th, 2013 5/19

  7. 0 1 2 3 4 5 6 7 8 9 10 11 12 Example T = [ cababcbababb$ ] SA i LCP i T SA i ... n 12 - $ 7 0 a b a b b $ 1 4 a b a b c b a b a b b $ 9 2 a b b $ 3 2 a b c b a b a b b $ 11 0 b $ 6 1 b a b a b b $ 8 3 b a b b $ 2 3 b a b c b a b a b b $ 10 1 b b $ 4 1 b c b a b a b b $ 0 0 c a b a b c b a b a b b $ 5 1 c b a b a b b $ Timo Bingmann, Johannes Fischer, and Vitaly Osipov – Inducing Suffix and LCP Arrays in External Memory Institute of Theoretical Informatics – Algorithmics January 7th, 2013 5/19

  8. F MM [PST07] BW O ( n ) tree original updated BWT S IT 1999 LS 1/2 copy A/B copy 2000 runs BK diffcover KSPP KS 2003 mod2 split DC3 KA L/S split MaF Na KJP 2004 deep-shallow succinct fixed Σ SS M 2005 bpr chains Mo MP 2006 NZ divsufsort cache aware O ( n log | Σ | ) 2007 Doubling NZC AN SAIS/SADS Prefix SFE-coding 2009 N Induced Copying Recursion 2011 OSACA

  9. Suffix Sorting 256 MiB of Gutenberg Text F MM [PST07] BW in RAM on Intel i7 2.67 GHz O ( n ) tree original updated BWT S IT 1999 LS 1/2 copy A/B copy 2000 runs 396 s BK diffcover KSPP 106 s KS 2003 66 s mod2 split DC3 KA L/S split MaF Na KJP 2004 deep-shallow 50 s succinct fixed Σ SS M 2005 bpr chains Mo MP 2006 NZ divsufsort cache aware O ( n log | Σ | ) 2007 39 s 129 / 189 s Doubling NZC AN SAIS/SADS Mo: 57 s Prefix SFE-coding 2009 98 s N Induced Copying Recursion 2011 OSACA

  10. Suffix Sorting 256 MiB of Gutenberg Text F MM [PST07] BW in RAM on Intel i7 2.67 GHz O ( n ) tree original updated BWT S IT 1999 LS 1/2 copy A/B copy 2000 runs 396 s BK diffcover KSPP 106 s KS 2003 66 s mod2 split DC3 KA L/S split MaF Na KJP 2004 deep-shallow 50 s succinct fixed Σ SS M 2005 bpr chains Mo MP 2006 NZ divsufsort cache aware O ( n log | Σ | ) 2007 39 s 129 / 189 s Doubling NZC AN SAIS/SADS Mo: 57 s Prefix SFE-coding 2009 98 s N Induced Copying Recursion 2011 OSACA

  11. Suffix Sorting 256 MiB of Gutenberg Text F MM [PST07] BW in RAM on Intel i7 2.67 GHz O ( n ) tree original updated BWT S IT 1999 LS 1/2 copy A/B copy 2000 runs 396 s BK diffcover KSPP 106 s KS 2003 66 s mod2 split DC3 KA L/S split MaF Na KJP 2004 deep-shallow 50 s succinct fixed Σ SS M 2005 bpr chains Mo MP 2006 NZ divsufsort cache aware O ( n log | Σ | ) 2007 39 s 129 / 189 s Doubling NZC AN SAIS/SADS Mo: 57 s Prefix SFE-coding 2009 98 s N Induced Copying Recursion 2011 OSACA

  12. LCP Construction Algorithms Algorithm Construction Time Space T → SA,LCP O ( n log n ) MM 1993 9 n KLAAP 2001 T,SA → LCP O ( n ) 13 n T → SA,LCP O ( n ) O ( n ) KS 2003 EM T,SA → LCP O ( n ) M 2004 9 n / 5 n 6 n + O ( n PT 2008 T,SA → v -LCP O ( nv ) v + v ) √ 5 n + 3 Φ -KMP 2009 T,SA → PLCP O ( n log n ) 8 n O ( n 2 ) T,SA,BWT,LF → LCP GO 2011 11 n T → SA,LCP O ( n ) F 2011 9 n Timo Bingmann, Johannes Fischer, and Vitaly Osipov – Inducing Suffix and LCP Arrays in External Memory Institute of Theoretical Informatics – Algorithmics January 7th, 2013 7/19

  13. 0 1 2 3 4 5 6 7 8 9 10 11 12 Example T = [ cababcbababb$ ] SA i T i − 1 T SA i ... n 12 b $ 7 b a b a b b $ 1 c a b a b c b a b a b b $ 9 b a b b $ 3 b a b c b a b a b b $ 11 b b $ 6 c b a b a b b $ 8 a b a b b $ 2 a b a b c b a b a b b $ 10 a b b $ 4 a b c b a b a b b $ 0 - c a b a b c b a b a b b $ 5 b c b a b a b b $ Timo Bingmann, Johannes Fischer, and Vitaly Osipov – Inducing Suffix and LCP Arrays in External Memory Institute of Theoretical Informatics – Algorithmics January 7th, 2013 8/19

  14. 0 1 2 3 4 5 6 7 8 9 10 11 12 Example T = [ cababcbababb$ ] SA i T i − 1 T SA i ... n 12 b $ 7 b a b a b b $ 1 c a b a b c b a b a b b $ 9 b a b b $ 3 b a b c b a b a b b $ 11 b b $ 6 c b a b a b b $ 8 a b a b b $ 2 a b a b c b a b a b b $ 10 a b b $ 4 a b c b a b a b b $ 0 - c a b a b c b a b a b b $ 5 b c b a b a b b $ Timo Bingmann, Johannes Fischer, and Vitaly Osipov – Inducing Suffix and LCP Arrays in External Memory Institute of Theoretical Informatics – Algorithmics January 7th, 2013 8/19

  15. 0 1 2 3 4 5 6 7 8 9 10 11 12 Example T = [ cababcbababb$ ] SA i T i − 1 T SA i ... n 12 b $ 7 b a b a b b $ 1 c a b a b c b a b a b b $ 9 b a b b $ 3 b a b c b a b a b b $ 11 b b $ 6 c b a b a b b $ 8 a b a b b $ 2 a b a b c b a b a b b $ 10 a b b $ 4 a b c b a b a b b $ 0 - c a b a b c b a b a b b $ 5 b c b a b a b b $ Timo Bingmann, Johannes Fischer, and Vitaly Osipov – Inducing Suffix and LCP Arrays in External Memory Institute of Theoretical Informatics – Algorithmics January 7th, 2013 8/19

  16. 0 1 2 3 4 5 6 7 8 9 10 11 12 Example T = [ cababcbababb$ ] SA i T i − 1 T SA i ... n 12 b $ 7 b a b a b b $ 1 c a b a b c b a b a b b $ 9 b a b b $ 3 b a b c b a b a b b $ 11 b b $ 6 c b a b a b b $ 8 a b a b b $ 2 a b a b c b a b a b b $ 10 a b b $ 4 a b c b a b a b b $ 0 - c a b a b c b a b a b b $ 5 b c b a b a b b $ Timo Bingmann, Johannes Fischer, and Vitaly Osipov – Inducing Suffix and LCP Arrays in External Memory Institute of Theoretical Informatics – Algorithmics January 7th, 2013 8/19

  17. 0 1 2 3 4 5 6 7 8 9 10 11 12 Example T = [ cababcbababb$ ] SA i T i − 1 T SA i ... n 12 b $ 7 b a b a b b $ 1 c a b a b c b a b a b b $ 9 b a b b $ 3 b a b c b a b a b b $ 11 b b $ 6 c b a b a b b $ 8 a b a b b $ 2 a b a b c b a b a b b $ 10 a b b $ 4 a b c b a b a b b $ 0 - c a b a b c b a b a b b $ 5 b c b a b a b b $ Timo Bingmann, Johannes Fischer, and Vitaly Osipov – Inducing Suffix and LCP Arrays in External Memory Institute of Theoretical Informatics – Algorithmics January 7th, 2013 8/19

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Lecture 15: Suffix trees, suffix arrays, and their applica8ons

Lecture 15: Suffix trees, suffix arrays, and their applica8ons

Lecture 15: Suffix trees, suffix arrays, and their applica8ons Spring 2017 April 4, 2017 1 Trie A tree represen8ng a set of strings. {aeef, ad,

298 views • 29 slides
Suffix arrays: A new method for on-line string searches Udi Manber 1 Gene Myers 2 Department of

Suffix arrays: A new method for on-line string searches Udi Manber 1 Gene Myers 2 Department of

Suffix arrays: A new method for on-line string searches Udi Manber 1 Gene Myers 2 Department of Computer Science University of Arizona Tucson, AZ 85721 May 1989 Revised August 1991 Abstract A new and conceptually simple data structure,

196 views • 16 slides
Algorithms for Dysfluency Detection in Symbolic Sequences using Suffix Arrays alfy 1 , 2 chal 1

Algorithms for Dysfluency Detection in Symbolic Sequences using Suffix Arrays alfy 1 , 2 chal 1

Algorithms for Dysfluency Detection in Symbolic Sequences using Suffix Arrays alfy 1 , 2 chal 1 J. P J. Posp 1 Slovak University of Technology, Faculty of Informatics and Information Technologies, Bratislava, Slovakia 2 Slovak Academy of

258 views • 22 slides
Algorithms in Bioinformatics: A Practical Introduction Suffix tree Overview What is suffix

Algorithms in Bioinformatics: A Practical Introduction Suffix tree Overview What is suffix

Algorithms in Bioinformatics: A Practical Introduction Suffix tree Overview What is suffix tree? Simple application of suffix tree Linear time algorithm for constructing suffix tree Suffix array FM-index 1-mismatch search

1.53k views • 111 slides
Arrays Memory a b c d start 1-dimensional array x = [a, b, c, d] map into

Arrays Memory a b c d start 1-dimensional array x = [a, b, c, d] map into

1D Array Representation In Java, C, and C++ Arrays Memory a b c d start 1-dimensional array x = [a, b, c, d] map into contiguous memory locations location(x[i]) = start + i Space Overhead 2D Arrays Memory a b c d The

512 views • 4 slides
Arrays Memory a b c d start 1-dimensional array x = [a, b, c, d] map into

Arrays Memory a b c d start 1-dimensional array x = [a, b, c, d] map into

1D Array Representation In Java, C, and C++ Arrays Memory a b c d start 1-dimensional array x = [a, b, c, d] map into contiguous memory locations location(x[i]) = start + i Space Overhead 2D Arrays Memory a b c d The

427 views • 6 slides
CENG 4480 L10 Memory 3 Bei Yu Reference : Chapter 11 Memories CMOS VLSI DesignA

CENG 4480 L10 Memory 3 Bei Yu Reference : Chapter 11 Memories CMOS VLSI DesignA

CENG 4480 L10 Memory 3 Bei Yu Reference : Chapter 11 Memories CMOS VLSI DesignA Circuits and Systems Perspective by H.E.Weste and D.M.Harris 1 Memory Arrays Memory Arrays Content Addressable Memory Random Access Memory Serial

485 views • 34 slides
Arrays LAST TODAY NEXT Integers Array mechanics Searching arrays Memory model Correctness

Arrays LAST TODAY NEXT Integers Array mechanics Searching arrays Memory model Correctness

Arrays LAST TODAY NEXT Integers Array mechanics Searching arrays Memory model Correctness Aliasing Safety Dataypes so far Basic (a.k.a. small types) : int, string, bool, char Integers represent numbers (rendered using 2s

712 views • 33 slides
Data, memory, pointer Pointers and arrays 1-1 Data, memory memory address: every byte

Data, memory, pointer Pointers and arrays 1-1 Data, memory memory address: every byte

Data, memory, pointer Pointers and arrays 1-1 Data, memory memory address: every byte is identified by a numeric address in the memory. once a variable is defined , one cannot predict its memory address, entirely determined by

976 views • 70 slides
capitalise Suffix terrorise fertilise ise suffix words are usually just created by simply

capitalise Suffix terrorise fertilise ise suffix words are usually just created by simply

This week, we are again looking at how nouns and adjectives can be changed into verbs using suffixes with spelling words such as. suffix capitalise Suffix terrorise fertilise ise suffix words are usually just created by simply adding

1.15k views • 19 slides
Pointers and Arrays Pointers and Arrays We've seen examples of both of these in our LC-3

Pointers and Arrays Pointers and Arrays We've seen examples of both of these in our LC-3

Chapter 16 Pointers and Arrays Pointers and Arrays We've seen examples of both of these in our LC-3 programs; now we'll see them in C. Pointer Address of a variable in memory Allows us to indirectly access variables in other words,

1.1k views • 29 slides
Implementing Logic in FPGA Embedded Memory Arrays: Heterogeneous Memory Architectures Steve

Implementing Logic in FPGA Embedded Memory Arrays: Heterogeneous Memory Architectures Steve

Implementing Logic in FPGA Embedded Memory Arrays: Heterogeneous Memory Architectures Steve Wilton University of British Columbia Vancouver, B.C., Canada stevew@ece.ubc.ca As FPGAs Get Bigger... Embedded Memory is becoming critical

539 views • 31 slides
Approximate Pattern Matching Using Suffix Tries Hendrik Nigul nigulh@math.ut.ee University of

Approximate Pattern Matching Using Suffix Tries Hendrik Nigul nigulh@math.ut.ee University of

Approximate Pattern Matching Using Suffix Tries Hendrik Nigul nigulh@math.ut.ee University of Tartu Veskisilla, Oct 3 2004 p. 1 Overview Introduction, problem description Suffix tries What is a suffix trie How to create suffix tries

1.03k views • 99 slides
CENG 4480 L09 Memory 2 Bei Yu Reference : Chapter 11 Memories CMOS VLSI DesignA

CENG 4480 L09 Memory 2 Bei Yu Reference : Chapter 11 Memories CMOS VLSI DesignA

CENG 4480 L09 Memory 2 Bei Yu Reference : Chapter 11 Memories CMOS VLSI DesignA Circuits and Systems Perspective by H.E.Weste and D.M.Harris 1 Memory Arrays Memory Arrays Random Access Memory Serial Access Memory Content

744 views • 35 slides
Suffix Trees Construction and Applications Joo Carreira 2008 Outline Why Suffix Trees?

Suffix Trees Construction and Applications Joo Carreira 2008 Outline Why Suffix Trees?

Suffix Trees Construction and Applications Joo Carreira 2008 Outline Why Suffix Trees? Definition Ukkonen's Algorithm (construction) Applications Why Suffix Trees? Why Suffix Trees? Asymptotically fast. Why Suffix Trees?

692 views • 68 slides
Structures, Strings, and Such What about arrays? Arrays are declared: 0 arr[0] int arr[8];

Structures, Strings, and Such What about arrays? Arrays are declared: 0 arr[0] int arr[8];

Structures, Strings, and Such What about arrays? Arrays are declared: 0 arr[0] int arr[8]; 0 arr[1] This declares an array of 8 0 arr[2] integers 0 arr[3] Memory allocated for it, but 5 0 arr[4] not initialized 0 arr[5]

149 views • 4 slides
GPU Primitives - Case Study: Hair Rendering Ulf Assarsson, Markus Billeter, Ola Olsson, Erik

GPU Primitives - Case Study: Hair Rendering Ulf Assarsson, Markus Billeter, Ola Olsson, Erik

GPU Primitives - Case Study: Hair Rendering Ulf Assarsson, Markus Billeter, Ola Olsson, Erik Sintorn Chalmers University of Technology Gothenburg, Sweden 1 Beyond Programmable Shading Parallelism Programming massively parallel systems 2

1.28k views • 82 slides
Verifying and enforcing network paths with ICING Jad Naous , Michael Walfish, Antonio Nicolosi,

Verifying and enforcing network paths with ICING Jad Naous , Michael Walfish, Antonio Nicolosi,

Verifying and enforcing network paths with ICING Jad Naous , Michael Walfish, Antonio Nicolosi, David Mazires, Michael Miller, and Arun Seehra 1 Today: New protocol for every feature Remote VPN, Private WANs, Specifying QoS, Firewalls,

581 views • 38 slides
Orthogonal Time Frequency Space (OTFS) Modulation and Applications Tutorial at SPCOM 2020, IISc,

Orthogonal Time Frequency Space (OTFS) Modulation and Applications Tutorial at SPCOM 2020, IISc,

Orthogonal Time Frequency Space (OTFS) Modulation and Applications Tutorial at SPCOM 2020, IISc, Bangalore, July, 2020 Yi Hong, Emanuele Viterbo, Raviteja Patchava Department of Electrical and Computer Systems Engineering Monash University,

4.48k views • 86 slides
vDC: Virtual Data Center Powered with AS Alliance for Enabling Cost-Effective Business

vDC: Virtual Data Center Powered with AS Alliance for Enabling Cost-Effective Business

vDC: Virtual Data Center Powered with AS Alliance for Enabling Cost-Effective Business Continuity and Coverage Yuichiro Hei* Akihiro Nakao Tomohiko Ogishi* Toru Hasegawa* Shu Yamamoto KDDI R&D Laboratories, Inc. * The

184 views • 16 slides
The DESQ Framework for Declarative and Scalable Frequent Sequence Mining Kaustubh Beedkar 1 Rainer

The DESQ Framework for Declarative and Scalable Frequent Sequence Mining Kaustubh Beedkar 1 Rainer

The DESQ Framework for Declarative and Scalable Frequent Sequence Mining Kaustubh Beedkar 1 Rainer Gemulla 2 Alexander Renz-Wieland 1 1 Technische Universit at Berlin 2 Universit at Mannheim INFORMATIK 19, Kassel September 24 th , 2019

254 views • 23 slides
Functional programming and hardware design: where to now?? Wouter Swierstra, Koen Claessen, Carl

Functional programming and hardware design: where to now?? Wouter Swierstra, Koen Claessen, Carl

Functional programming and hardware design: where to now?? Wouter Swierstra, Koen Claessen, Carl Seger, Emily Shriver, Mary Sheeran with thanks to Andy Gill (U Kansas) Behavioural Hawk (Cook, Launchbury, Matthews) Chalk Lava (Bjesse,

405 views • 36 slides
Simpler and More General Minimization for Weighted Finite-State Automata Jason Eisner Department

Simpler and More General Minimization for Weighted Finite-State Automata Jason Eisner Department

Appeared in Proceedings of the Joint Meeting of the Human Language Technology Conference and the North American Chapter of the Association for Computational Linguistics (HLT-NAACL 2003) , pp. 64-71, 2003. Simpler and More General Minimization for

290 views • 8 slides
Discrete Morse Theory and Generalized Factor Order Bruce Sagan Department of Mathematics,

Discrete Morse Theory and Generalized Factor Order Bruce Sagan Department of Mathematics,

Discrete Morse Theory and Generalized Factor Order Bruce Sagan Department of Mathematics, Michigan State U. East Lansing, MI 48824-1027, sagan@math.msu.edu www.math.msu.edu/ sagan and Robert Willenbring Department of Mathematics, Michigan

1.68k views • 111 slides

More recommend