database cracking
play

Database cracking Stratos Idreos, Martin Kersten and Stefan Manegold - PowerPoint PPT Presentation

May 23, 2023 •318 likes •627 views

Database cracking Stratos Idreos, Martin Kersten and Stefan Manegold CWI Amsterdam, The Netherlands The open problem - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Dynamic

  1. Database cracking Stratos Idreos, Martin Kersten and Stefan Manegold CWI Amsterdam, The Netherlands

  2. The open problem - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Dynamic environments What kind of indices should be used, when and on which data? Database experts or special tools monitor the system More difficult in databases with huge datasets

  3. Database cracking - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - We explore self-organization Each query triggers physical re-organization of the database We designed and implemented a DBMS using database cracking We work on top of MonetDB, a column oriented database system

  4. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - select A>5 and A<10 17 3 8 6 2 12 13 4 15

  5. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - select A>5 and A<10 17 3 3 4 8 2 6 2 12 13 4 15

  6. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - select A>5 and A<10 17 3 3 4 8 2 6 8 2 6 12 13 4 15

  7. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - select A>5 and A<10 17 3 <= 5 3 4 8 2 6 8 > 5 2 6 12 12 13 15 >= 10 4 17 15 13

  8. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - select A>5 and A<10 17 3 <= 5 3 4 8 2 6 8 > 5 2 6 12 12 13 15 >= 10 4 17 15 13

  9. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Improve data access for future queries select A>5 and A<10 17 3 <= 5 3 4 8 2 6 8 > 5 2 6 12 12 13 15 >= 10 4 17 15 13

  10. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Improve data access for future queries select A>5 and A<10 select A>2 and A<15 17 3 <= 5 3 4 8 2 6 8 > 5 2 6 12 12 13 15 >= 10 4 17 15 13

  11. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Improve data access for future queries select A>5 and A<10 select A>2 and A<15 17 3 2 <= 5 3 4 8 2 6 8 > 5 2 6 12 12 13 15 >= 10 4 17 15 13

  12. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Improve data access for future queries select A>2 and A<15 select A>5 and A<10 2 17 3 <= 5 3 4 4 8 2 3 6 8 > 5 2 6 12 12 13 15 >= 10 4 17 15 13

  13. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Improve data access for future queries select A>2 and A<15 select A>5 and A<10 2 17 3 <= 5 3 4 4 8 2 3 6 8 8 > 5 2 6 6 12 12 13 15 >= 10 4 17 15 13

  14. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Improve data access for future queries select A>2 and A<15 select A>5 and A<10 2 17 3 <= 5 3 4 4 8 2 3 6 8 8 > 5 2 6 6 12 12 12 13 13 15 >= 10 4 17 15 13

  15. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Improve data access for future queries select A>2 and A<15 select A>5 and A<10 <= 2 2 17 3 3 4 4 > 2 8 2 3 6 8 8 > 5 > 5 2 6 6 12 12 12 >= 10 13 13 15 >= 10 17 4 17 >= 15 15 13 15

  16. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Improve data access for future queries select A>2 and A<15 select A>5 and A<10 <= 2 2 17 3 3 4 4 > 2 8 2 3 6 8 8 > 5 > 5 2 6 6 12 12 12 >= 10 13 13 15 >= 10 17 4 17 >= 15 15 13 15

  17. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Improve data The more we access for crack the more future queries we learn select A>2 and A<15 select A>5 and A<10 <= 2 2 17 3 3 4 4 > 2 8 2 3 6 8 8 > 5 > 5 2 6 6 12 12 12 >= 10 13 13 15 >= 10 17 4 17 >= 15 15 13 15

  18. Cracking algorithms - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - There are two types of cracking algorithms Split a piece in Split a piece in two new pieces three new pieces

  19. Design - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - The first time a range query is posed on an attribute A , a cracking DBMS makes a copy of column A, called the cracker column of A A cracker column is continuously physically re-organized based on queries that need to touch attribute such as the result is in a contiguous space For each cracker column, there is a cracker index

  20. The cracker select operator - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  The simple select operator: 1. Scans the column 2. Return a new column that contains qualifying values  The crackers select operator: 1. Searches the cracker index 2. Physically re-organizes pieces found 3. Update the cracker index 4. Return a slice of the cracker column as result  More steps but faster because we analyze less data

  21. Testing the select operator - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

  22. Research and opportunities for cracking - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  Optimization  optimal piece size / granularity / index (avl) depth  Exploit cracking for join queries, aggregate queries etc.  Concurrency issues  Cracking histograms  Distributed cracking  A priori cracking  ...

  23. Research and opportunities for cracking - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  Optimization  optimal piece size / granularity / index (avl) depth  Exploit cracking for join queries, aggregate queries etc.  Concurrency issues  Cracking histograms  Distributed cracking  A priori cracking THANK YOU!  ...

  24. Cracking vs Indices and Sorting - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  How cracking compares to a sorting strategy?  Sort data upfront and then use binary search  For a sorting strategy we have to make an investment upfront  Sorting needs prior knowledge of query workload  Similar arguments stand for indices

  25. Impact on query plan - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - select R.c from R where 5 < R.a < 10 and 9 < R.b < 20 Rb1 Ra1 algebra.select(Ra, 5, 10) algebra.select(Rb, 9, 20) Ra2 algebra.OIDintersect(Ra1, Rb1) algebra.fetch(Ra2, Rc)

  26. Impact on query plan - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - select R.c from R where 5 < R.a < 10 and 9 < R.b < 20 Rb1 Ra1 crackers.select(Rb, 9, 20) crackers.select(Ra, 5, 10) algebra.select(Ra, 5, 10) algebra.select(Rb, 9, 20) Ra2 algebra.OIDintersect(Ra1, Rb1) algebra.fetch(Ra2, Rc)

  27. Impact on query plan - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - select R.c from R where 5 < R.a < 10 and 9 < R.b < 20 Rb1 Ra1 crackers.select(Rb, 9, 20) crackers.select(Ra, 5, 10) algebra.select(Ra, 5, 10) algebra.select(Rb, 9, 20) Becomes Ra2 algebra.OIDintersect(Ra1, Rb1) expensive algebra.fetch(Ra2, Rc)

  28. Impact on query plan - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - select R.c from R where 5 < R.a < 10 and 9 < R.b < 20 Ra1 crackers.select(Ra, 5, 10) algebra.select(Ra, 5, 10) Ra2 algebra.joinselect(Ra1, Rb,9,20) algebra.OIDintersect(Ra1, Rb1) algebra.fetch(Ra2, Rc)

  29. Scalability - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

  30. TPC-H query 6 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

DATABASE CRACKING: Fancy Scan, not Poor Mans Sort! Hardware Folks Cracking Folks Don Holger

DATABASE CRACKING: Fancy Scan, not Poor Mans Sort! Hardware Folks Cracking Folks Don Holger

DATABASE CRACKING: Fancy Scan, not Poor Mans Sort! Hardware Folks Cracking Folks Don Holger Pirk Eleni Petraki Strato Idreos Stefan Manegold Martin Kersten EVALUATING RANGE PREDICATES COMPLEXITY ON PAPER Scanning: O(n) Sorting:

910 views • 43 slides
Database Cracking September 7, 2016 CSE 662 - Database Languages &amp; Runtimes 1 Row Stores

Database Cracking September 7, 2016 CSE 662 - Database Languages &amp; Runtimes 1 Row Stores

Database Cracking September 7, 2016 CSE 662 - Database Languages &amp; Runtimes 1 Row Stores A1 B1 C1 D1 A2 B2 C2 D2 A3 B3 C3 D3 A4 B4 C4 D4 Traditional DB: Lay out data on disk in rows CSE 662 - Database Languages &amp;

753 views • 41 slides
Database Cracking Languages and Runtimes for Big Data CSE 662 - Database Languages &amp; Runtimes

Database Cracking Languages and Runtimes for Big Data CSE 662 - Database Languages &amp; Runtimes

Database Cracking Languages and Runtimes for Big Data CSE 662 - Database Languages &amp; Runtimes 1 Row Stores A1 B1 C1 D1 A2 B2 C2 D2 A3 B3 C3 D3 A4 B4 C4 D4 Traditional DB: Lay out data on disk in rows CSE 662 - Database

544 views • 41 slides
Project Seeds Languages &amp; Runtimes for Big Data Reminder Homework 1: Database Cracking

Project Seeds Languages &amp; Runtimes for Big Data Reminder Homework 1: Database Cracking

Project Seeds Languages &amp; Runtimes for Big Data Reminder Homework 1: Database Cracking Read the paper (linked from the course page) Submit 2 discussion points (strength and weakness of the work) or make a counterargument to

588 views • 38 slides
Cracking the code Cracking the code Houston, we have a problem. Consanguineous Marriage Tay

Cracking the code Cracking the code Houston, we have a problem. Consanguineous Marriage Tay

Cracking the code Cracking the code Houston, we have a problem. Consanguineous Marriage Tay Brain T2-Hyperintensity Sachs Stem Pelizaeus- B12 Metabolism U-Fibres Merzbacher Hypomyelination T1-Hypointensity CSF Salla LEUKODYSTROPHY

918 views • 62 slides
IAPF TRUSTEE NETWORK EVENT CRACKING THE DC CODE: VALUE FOR MONEY WELCOME Cracking the DC Code:

IAPF TRUSTEE NETWORK EVENT CRACKING THE DC CODE: VALUE FOR MONEY WELCOME Cracking the DC Code:

IAPF TRUSTEE NETWORK EVENT CRACKING THE DC CODE: VALUE FOR MONEY WELCOME Cracking the DC Code: Value for Money Rickard Mills Council Member, IAPF THANK YOU SPONSOR Cracking the DC Code: Value for Money HOUSEKEEPING Cracking the DC Code:

1.04k views • 54 slides
twenty six diagonal cracking shear f c http://www.bam.de concrete construction:

twenty six diagonal cracking shear f c http://www.bam.de concrete construction:

A RCHITECTURAL S TRUCTURES : Concrete in Compression F ORM, B EHAVIOR, AND D ESIGN ARCH 331 crushing D R. A NNE N ICHOLS vertical cracking S PRING 2018 tension lecture twenty six diagonal cracking shear f c

137 views • 3 slides
Cracking Wireless Ryan Curtin LUG@GT Ryan Curtin Cracking Wireless - p. 1 Goals By the end of

Cracking Wireless Ryan Curtin LUG@GT Ryan Curtin Cracking Wireless - p. 1 Goals By the end of

Cracking Wireless Ryan Curtin LUG@GT Ryan Curtin Cracking Wireless - p. 1 Goals By the end of this presentation (if you stay awake), you will: Goals Setting Up Checking Injection Understand the different types of wireless keys

447 views • 13 slides
Merging Whats Cracked, Cracking Whats Merged Adaptive Indexing in Main-Memory Column-Stores

Merging Whats Cracked, Cracking Whats Merged Adaptive Indexing in Main-Memory Column-Stores

Merging Whats Cracked, Cracking Whats Merged Adaptive Indexing in Main-Memory Column-Stores Stratos Idreos, Stefan Manegold, (CWI) Harumi Kuno, Goetz Graefe (HP Labs) PVLDB 2011, 4(9) Physical design problem Database systems perform

1.24k views • 122 slides
Designing for durability and strength Cracking &amp; durability Local studies on corrosion &amp;

Designing for durability and strength Cracking &amp; durability Local studies on corrosion &amp;

Designing for durability and strength Cracking &amp; durability Local studies on corrosion &amp; cracking Other aspects Presented by: Professor Mark Alexander Concrete Materials and Structural Integrity Research Unit (CoMSIRU)

718 views • 29 slides
PicoCrack: The Art of Efficient Cracking FPGAs (Field Programmable Gate Arrays) allow custom

PicoCrack: The Art of Efficient Cracking FPGAs (Field Programmable Gate Arrays) allow custom

PicoCrack: The Art of Efficient Cracking FPGAs (Field Programmable Gate Arrays) allow custom silicon to be implemented easily. The result is a chip that can be built specifically for cracking passwords. This presentation focuses on uncovering

606 views • 47 slides
Distributed GPU password cracking Alexander Kasabov &amp; Jochem van Kerkwijk System and

Distributed GPU password cracking Alexander Kasabov &amp; Jochem van Kerkwijk System and

Distributed GPU password cracking Alexander Kasabov &amp; Jochem van Kerkwijk System and Network Engineering { akasabov | jkerkwijk } @os3.nl February 2, 2011 Outline Introduction Password cracking Graphics processing unit Distributed

501 views • 16 slides
Physical and Metallurgical Analysis Aids in Understanding Recovery Boiler Tube Cracking

Physical and Metallurgical Analysis Aids in Understanding Recovery Boiler Tube Cracking

Physical and Metallurgical Analysis Aids in Understanding Recovery Boiler Tube Cracking November, 2014 PdM Technology Infrared Thermography Gas Find IR Example of LNG Leak Safety Minute Economizer Tube Cracking Assessment

447 views • 18 slides
ramp. Photo 2. Looking along the guardrail. Cracking has protruded through asphalt patch,

ramp. Photo 2. Looking along the guardrail. Cracking has protruded through asphalt patch,

SHOP SLIDE Photo 1. Cracking across the footpath at the end of the Hwy 2 off ramp. Photo 2. Looking along the guardrail. Cracking has protruded through asphalt patch, with a drop depression in the asphalt and significant bow in the

247 views • 10 slides
Seamless Asphalt Pavement Preservation Featuring Types of asphalt failure Alligator cracking

Seamless Asphalt Pavement Preservation Featuring Types of asphalt failure Alligator cracking

36 th Annual Utah Asphalt Conference Seamless Asphalt Pavement Preservation Featuring Types of asphalt failure Alligator cracking Joint cracking Raveling Rutting Potholes Typical Repair Approaches Sawcut, removal, and

445 views • 27 slides
Database Utilities 10/17/2007 DC/Win Database Utilities Opening Database Utilities From File on

Database Utilities 10/17/2007 DC/Win Database Utilities Opening Database Utilities From File on

Database Utilities 10/17/2007 DC/Win Database Utilities Opening Database Utilities From File on the Menu Bar DC/Win Database Utilities Options Vary Depending on Database Two Types of Databases Sequel (SQL) Database Access Database

460 views • 6 slides
Winning The Race Against Winning The Race Against Software Pirates Software Pirates Software

Winning The Race Against Winning The Race Against Software Pirates Software Pirates Software

Winning The Race Against Winning The Race Against Software Pirates Software Pirates Software Pirates Software Pirates Next Generation Copy Protection 1 Yours Truly Yours Truly Erik Simon Erik Simon Veteran of the German Development

613 views • 40 slides
Preparation Begets Safe Performance Planning for an Underwater Munitions Investigation 2015 M2S2

Preparation Begets Safe Performance Planning for an Underwater Munitions Investigation 2015 M2S2

Preparation Begets Safe Performance Planning for an Underwater Munitions Investigation 2015 M2S2 Webinar Series April 23, 2015 Former Raritan Arsenal The former Raritan Arsenal covers approximately 3,200 acres of central New Jersey

462 views • 15 slides
Network provider Security Markus Peuhkuri 2005-04-28 Lecture topics Basics of security

Network provider Security Markus Peuhkuri 2005-04-28 Lecture topics Basics of security

Network provider Security Markus Peuhkuri 2005-04-28 Lecture topics Basics of security Security threats Regulators and ISP security Some headlines Davie-Besse nuclear reactor control network

482 views • 17 slides
A Balloon-borne Soft Gamma-ray Polarimeter Mark Pearce Dept. of Physics, KTH, Stockholm, Sweden

A Balloon-borne Soft Gamma-ray Polarimeter Mark Pearce Dept. of Physics, KTH, Stockholm, Sweden

A Balloon-borne Soft Gamma-ray Polarimeter Mark Pearce Dept. of Physics, KTH, Stockholm, Sweden P o G OLite SLAC / Stanford- KIPAC, Hawaii KTH, Stockholm University Tokyo Institute of Technology, Hiroshima University, ISAS/JAXA,

394 views • 15 slides
Attack Frameworks and Tools Pranav Jagdish Betreuer: Nadine Herold Seminar Innovative Internet

Attack Frameworks and Tools Pranav Jagdish Betreuer: Nadine Herold Seminar Innovative Internet

Network Architectures and Services, Georg Carle Faculty of Informatics Technische Universitt Mnchen, Germany Attack Frameworks and Tools Pranav Jagdish Betreuer: Nadine Herold Seminar Innovative Internet Technologies and Mobile

584 views • 30 slides
Password Cracking Prof. Tom Austin San Jos State University How should you store users'

Password Cracking Prof. Tom Austin San Jos State University How should you store users'

CS 166: Information Security Password Cracking Prof. Tom Austin San Jos State University How should you store users' passwords? Reverse-lookup tables A cryptographic hash is irreversible. However, you can make a table of common hashes.

359 views • 11 slides
Denial of Service A/acks Cunsheng Ding Department of CSE HKUST, Hong Kong Acknowledgements:

Denial of Service A/acks Cunsheng Ding Department of CSE HKUST, Hong Kong Acknowledgements:

Denial of Service A/acks Cunsheng Ding Department of CSE HKUST, Hong Kong Acknowledgements: Materials are taken from the Internet COMP4631 1 Agenda of this lecture Zombie computers, bots, botnets Denial of service (DoS) a/acks

630 views • 45 slides
In collabora*on with IDPF and BISG Session 4: DRM Moderator: Oliver

In collabora*on with IDPF and BISG Session 4: DRM Moderator: Oliver

In collabora*on with IDPF and BISG Session 4: DRM Moderator: Oliver Brooks Gerardo Capiel Minhyung Ko Jim Dovey Valobox Benetech Samsung Kobo Ill talk about general

642 views • 28 slides

More recommend