data systems that are easy to design tune and use
play

Data Systems that are Easy to Design, Tune and Use Stratos Idreos - PowerPoint PPT Presentation

Dec 28, 2022 •370 likes •1.56k views

Data Systems that are Easy to Design, Tune and Use Stratos Idreos applications api/sql algorithms/operators cpu data data data memory hierarchy data system kernel Stratos Idreos design space it all starts with how we store data every

  1. column A 4 4 13 Q1: 9 2 16 select R.A 2 1 4 piece1: A<=7 from R piece1: 7 3 9 where R.A>10 A<=10 1 6 2 and R.A<14 3 7 12 8 9 7 piece2: 7<A<=10 6 8 1 Q2: 13 13 19 piece2: select R.A piece3: 10<A<14 12 12 3 10<A<14 from R 11 11 14 where R.A>7 16 14 11 and R.A<=16 piece3: 19 16 8 A>=14 14 19 6 dynamically/on-the-fly within the select-operator Database Cracking CIDR 2007 Stratos Idreos

  2. column A 4 4 13 Q1: 9 2 16 select R.A 2 1 4 piece1: A<=7 from R piece1: 7 3 9 where R.A>10 A<=10 1 6 2 and R.A<14 3 7 12 8 9 7 piece2: 7<A<=10 6 8 1 Q2: 13 13 19 piece2: select R.A piece3: 10<A<14 12 12 3 10<A<14 from R 11 11 14 where R.A>7 16 14 11 piece4: 14<=A<=16 and R.A<=16 piece3: 19 16 8 piece5: A>16 A>=14 14 19 6 dynamically/on-the-fly within the select-operator Database Cracking CIDR 2007 Stratos Idreos

  3. column A 4 4 13 Q1: 9 2 16 select R.A 2 1 4 piece1: A<=7 from R piece1: 7 3 9 where R.A>10 A<=10 1 6 2 and R.A<14 3 7 12 8 9 7 piece2: 7<A<=10 6 8 1 Q2: 13 13 19 result piece2: select R.A piece3: 10<A<14 12 12 3 10<A<14 from R 11 11 14 where R.A>7 16 14 11 piece4: 14<=A<=16 and R.A<=16 piece3: 19 16 8 piece5: A>16 A>=14 14 19 6 dynamically/on-the-fly within the select-operator Database Cracking CIDR 2007 Stratos Idreos

  4. the more we crack, the more we learn column A 4 4 13 Q1: 9 2 16 select R.A 2 1 4 piece1: A<=7 from R piece1: 7 3 9 where R.A>10 A<=10 1 6 2 and R.A<14 3 7 12 8 9 7 piece2: 7<A<=10 6 8 1 Q2: 13 13 19 result piece2: select R.A piece3: 10<A<14 12 12 3 10<A<14 from R 11 11 14 where R.A>7 16 14 11 piece4: 14<=A<=16 and R.A<=16 piece3: 19 16 8 piece5: A>16 A>=14 14 19 6 dynamically/on-the-fly within the select-operator Database Cracking CIDR 2007 Stratos Idreos

  5. Database Cracking CIDR 2007 Stratos Idreos

  6. select [15,55] Database Cracking CIDR 2007 Stratos Idreos

  7. select [15,55] Database Cracking CIDR 2007 Stratos Idreos

  8. select [15,55] 40 50 30 60 10 20 Database Cracking CIDR 2007 Stratos Idreos

  9. select [15,55] 40 50 30 60 10 20 select [15,55] Database Cracking CIDR 2007 Stratos Idreos

  10. select [15,55] 40 50 30 60 10 20 select [15,55] Database Cracking CIDR 2007 Stratos Idreos

  11. touch at most two pieces at a time pieces become smaller and smaller select [15,55] 40 50 30 60 10 20 select [15,55] Database Cracking CIDR 2007 Stratos Idreos

  12. continuous adaptation set-up Scan 100K random selections 1000 random selectivity random value ranges 100 in a 10 million integer column Response time (secs) 10 Crack 1 0.1 Full Index 0.01 0.001 1 10 100 1000 10000 100000 Query sequence (x1000) Database Cracking CIDR 2007 Stratos Idreos

  13. continuous adaptation set-up Scan 100K random selections 1000 random selectivity random value ranges 100 in a 10 million integer column Response time (secs) 10 Crack 1 almost no 0.1 initialization overhead Full Index 0.01 0.001 1 10 100 1000 10000 100000 Query sequence (x1000) Database Cracking CIDR 2007 Stratos Idreos

  14. continuous adaptation set-up Scan 100K random selections 1000 random selectivity random value ranges 100 in a 10 million integer column Response time (secs) 10 Crack 1 almost no 0.1 initialization overhead Full Index 0.01 continuous improvement 0.001 1 10 100 1000 10000 100000 Query sequence (x1000) Database Cracking CIDR 2007 Stratos Idreos

  15. continuous adaptation set-up Scan 100K random selections 1000 random selectivity random value ranges 100 in a 10 million integer column Response time (secs) 10 Crack 1 almost no 0.1 initialization overhead Full Index 0.01 continuous improvement 0.001 1 10 100 1000 10000 100000 Query sequence (x1000) Database Cracking CIDR 2007 Stratos Idreos

  16. continuous adaptation set-up 200 Full Index 100 10K random selections Cumulative average response time (secs) selectivity 10% random value ranges 10 in a 30 million integer column Scan 1 0.1 0.01 Crack 0.004 0.001 1 10 100 1000 10000 Query sequence Database Cracking CIDR 2007 Stratos Idreos

  17. continuous adaptation set-up 200 Full Index 100 10K random selections Cumulative average response time (secs) selectivity 10% random value ranges 10 in a 30 million integer column Scan 1 0.1 0.01 Crack 0.004 0.001 1 10 100 1000 10000 Query sequence Database Cracking CIDR 2007 Stratos Idreos

  18. continuous adaptation set-up 200 Full Index 100 10K random selections Cumulative average response time (secs) selectivity 10% random value ranges 10 in a 30 million integer column Scan 1 0.1 0.01 Crack 10K queries later, 0.004 Full Index still has not 0.001 amortized the initialization costs 1 10 100 1000 10000 Query sequence Database Cracking CIDR 2007 Stratos Idreos

  19. table1 A Stratos Idreos

  20. table1 A B C D ... ... Stratos Idreos

  21. select R.A from R where R.A>10 and R.A<14 table1 A B C D ... ... Stratos Idreos

  22. select R.A from R where R.A>10 and R.A<14 select max(R.A),max(R.B),max(S.A),max(S.B) from R,S where v1 <R.C<v2 and v3 <R.D<v4 and v5 <R.E<v6 and k1 <S.C<k2 and k3 <S.D<k4 and k5 <S.E<k6 and R.F = S.F table1 A B C D ... ... Stratos Idreos

  23. select R.A from R where R.A>10 and R.A<14 select max(R.A),max(R.B),max(S.A),max(S.B) from R,S where v1 <R.C<v2 and v3 <R.D<v4 and v5 <R.E<v6 and k1 <S.C<k2 and k3 <S.D<k4 and k5 <S.E<k6 and R.F = S.F table1 updates A B C D joins ... concurrency control ... ... Stratos Idreos

  24. cracking databases basics (CIDR07) >1 storage- updates >1 columns columns restrictions (SIGMOD07) (SIGMOD09) (SIGMOD09) benchmarking robustness algorithms robustne (TPCTC10) (PVLDB12) (PVLDB11) concurrency control multi-cores time-series (PVLDB12) (SIGMOD15) (SIGMOD14) adaptive storage (SIGMOD14) b-trees hadoop (HP Labs) (Yale/Saarland)

  25. cracking tangram base data as queries arrive... table 1 A B C D table 2 A B C D Stratos Idreos

  26. cracking tangram base data as queries arrive... table 1 table 1 A B C D A B C D table 2 table 2 A B C D A B C D Stratos Idreos

  27. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D table 2 table 2 A B C D A B C D Stratos Idreos

  28. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D partial indexing table 2 table 2 A B C D A B C D Stratos Idreos

  29. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D partial indexing continuous adaptation table 2 table 2 A B C D A B C D Stratos Idreos

  30. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D x partial indexing continuous adaptation storage adaptation table 2 table 2 x A B C D A B C D x Stratos Idreos

  31. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D partial indexing continuous adaptation storage adaptation table 2 table 2 A B C D A B C D Stratos Idreos

  32. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D partial indexing continuous adaptation storage adaptation no tuple reconstruction table 2 table 2 A B C D A B C D Stratos Idreos

  33. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D partial indexing continuous adaptation storage adaptation no tuple reconstruction adaptive alignment table 2 table 2 A B C D A B C D Stratos Idreos

  34. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D partial indexing continuous adaptation storage adaptation no tuple reconstruction adaptive alignment table 2 table 2 A B C D A B C D Stratos Idreos

  35. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D partial indexing continuous adaptation storage adaptation no tuple reconstruction adaptive alignment table 2 table 2 sort in caches A B C D A B C D Stratos Idreos

  36. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D partial indexing continuous adaptation storage adaptation no tuple reconstruction adaptive alignment table 2 table 2 sort in caches A B C D A B C D crack joins Stratos Idreos

  37. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D partial indexing continuous adaptation storage adaptation no tuple reconstruction adaptive alignment table 2 table 2 sort in caches A B C D A B C D q1 crack joins q2 lightweight locking Stratos Idreos

  38. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D partial indexing continuous adaptation storage adaptation no tuple reconstruction adaptive alignment table 2 table 2 sort in caches A B C D A B C D query crack joins random lightweight locking stochastic cracking Stratos Idreos

  39. MonetDB Sel. Crack MySQL Presorted Sid. Crack Presorted 10000 Response time (milli secs) 1000 764 TPC-H Query 15 330 420 300 250 200 150 100 70 0 5 10 15 20 25 30 Query sequence Sideways Cracking, SIGMOD 09 Stratos Idreos

  40. MonetDB Sel. Crack MySQL Presorted Sid. Crack Presorted 10000 Response time (milli secs) 1000 764 TPC-H Query 15 330 420 300 normal MonetDB 250 200 selection cracking 150 100 70 0 5 10 15 20 25 30 Query sequence Sideways Cracking, SIGMOD 09 Stratos Idreos

  41. MonetDB Sel. Crack MySQL Presorted Sid. Crack Presorted 10000 Response time (milli secs) 1000 764 TPC-H Query 15 330 420 300 normal MonetDB 250 200 selection cracking 150 100 70 presorted MonetDB 0 5 10 15 20 25 30 preparation cost Query sequence 3-14 minutes Sideways Cracking, SIGMOD 09 Stratos Idreos

  42. MonetDB Sel. Crack MySQL Presorted Sid. Crack Presorted 10000 Response time (milli secs) 1000 764 TPC-H Query 15 330 420 300 normal MonetDB 250 200 selection cracking 150 MonetDB with 100 sideways cracking 70 presorted MonetDB 0 5 10 15 20 25 30 preparation cost Query sequence 3-14 minutes Sideways Cracking, SIGMOD 09 Stratos Idreos

  43. MonetDB Sel. Crack MySQL Presorted Sid. Crack Presorted 10000 Response time (milli secs) 1000 764 TPC-H Query 15 330 420 300 normal MonetDB 250 200 selection cracking 150 MonetDB with 100 sideways cracking 70 presorted MonetDB 0 5 10 15 20 25 30 preparation cost Query sequence 3-14 minutes Sideways Cracking, SIGMOD 09 Stratos Idreos

  44. MonetDB Sel. Crack MySQL Presorted Sid. Crack Presorted 10000 Response time (milli secs) 1000 764 TPC-H Query 15 330 420 300 normal MonetDB 250 200 selection cracking 150 MonetDB with 100 sideways cracking 70 presorted MonetDB 0 5 10 15 20 25 30 preparation cost Query sequence 3-14 minutes Sideways Cracking, SIGMOD 09 Stratos Idreos

  45. MonetDB Sel. Crack MySQL Presorted Sid. Crack Presorted 10000 Response time (milli secs) 1000 764 TPC-H Query 15 330 420 300 normal MonetDB 250 200 selection cracking 150 MonetDB with 100 sideways cracking 70 presorted MonetDB 0 5 10 15 20 25 30 preparation cost Query sequence 3-14 minutes Sideways Cracking, SIGMOD 09 Stratos Idreos

  46. cracking on Skyserver (4TB) (Sloan Digital Sky Survey, www.sdss.org) cracking answers 160.000 queries while full indexing is still half way creating one index Stochastic Cracking, PVLDB 12 Stratos Idreos

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

7 Habits of highly effective woodworkers Workshop tune-up/makeover Workbench tune-up

7 Habits of highly effective woodworkers Workshop tune-up/makeover Workbench tune-up

FWW references 7 Habits of highly effective woodworkers Workshop tune-up/makeover Workbench tune-up Storage of tools Tune-up of squares Handplane tune-up Sharpening- bevel angles Cabinet scrapers

477 views • 26 slides
Tune In &amp; Tune Up San Joaquin Valley, CA Helping Immigrant Families Build Financial

Tune In &amp; Tune Up San Joaquin Valley, CA Helping Immigrant Families Build Financial

Tune In &amp; Tune Up San Joaquin Valley, CA Helping Immigrant Families Build Financial Capability Webinar August 25 th , 2016 San Joaquin Valley, CA Epicenter of pollution and poverty Population of 4 million people Largest rural

509 views • 15 slides
API Design made easy with the GraphQL Whiteboard Jrgen Albert Data In Motion Consulting GmbH

API Design made easy with the GraphQL Whiteboard Jrgen Albert Data In Motion Consulting GmbH

API Design made easy with the GraphQL Whiteboard Jrgen Albert Data In Motion Consulting GmbH 1 About Us Founded in 2010 Located in Jena/Thuringia - Germany Consulting, Independen RnD, Development, Training Assisted

346 views • 15 slides
To tune or not to tune Thomas Pasquier tfjmp@cs.ubc.ca https://tfjmp.org The team - Ayat Fekry

To tune or not to tune Thomas Pasquier tfjmp@cs.ubc.ca https://tfjmp.org The team - Ayat Fekry

To tune or not to tune Thomas Pasquier tfjmp@cs.ubc.ca https://tfjmp.org The team - Ayat Fekry , PhD student - Lucian Carata , Senior Research Associate - Andrew Rice , Professor - Andy Hopper , Professor 2 About me - Assistant Professor

626 views • 37 slides
Computer Science 240 Principles of Software Design Goals of Software Design Create systems

Computer Science 240 Principles of Software Design Goals of Software Design Create systems

Computer Science 240 Principles of Software Design Goals of Software Design Create systems that Work Easy as possible to understand, debug, and maintain Hold up well under changes Have reusable components Design is

470 views • 20 slides
Easy-to-Use Easy-to-Install Easy on the Budget orecx.com Easy-to-Use

Easy-to-Use Easy-to-Install Easy on the Budget orecx.com Easy-to-Use

Easy-to-Use Easy-to-Install Easy on the Budget orecx.com Easy-to-Use Easy-to-Install Easy on the Budget OrecX Call Recording The issue is simple: You want to comply with regulatory requirements and improve customer service by

204 views • 9 slides
PRESENTATION OF DATA Data summarization Is the organization of data in a way for easy

PRESENTATION OF DATA Data summarization Is the organization of data in a way for easy

Tabular presentation PRESENTATION OF DATA Data summarization Is the organization of data in a way for easy understanding. It is the first step of data interpretation (analysis). Consists of the following steps: 1) Data entering. 2)

932 views • 71 slides
ELECTRONIC MEDICAL RECORDS FOR EMERGENCIES / PROJECT BUENDIA WORKS EVERYWHERE EXTREMELY

ELECTRONIC MEDICAL RECORDS FOR EMERGENCIES / PROJECT BUENDIA WORKS EVERYWHERE EXTREMELY

Fabian Tamp @capnfabs ELECTRONIC MEDICAL RECORDS FOR EMERGENCIES / PROJECT BUENDIA WORKS EVERYWHERE EXTREMELY ADAPTABLE Photos: MSF PORTABLE EASY TO ANALYSE EASY TO BACK UP EASY TO ACCESS HISTORICAL DATA BETTER DESIGN FOR DECISION SUPPORT

631 views • 15 slides
Snake orbit effect on the spin tune in RHIC M. Bai, V. Ptitsyn, T. Roser Spin tune versus snake

Snake orbit effect on the spin tune in RHIC M. Bai, V. Ptitsyn, T. Roser Spin tune versus snake

Snake orbit effect on the spin tune in RHIC M. Bai, V. Ptitsyn, T. Roser Spin tune versus snake orbit angle Making use of the depolarization by 0.75 Example of measurements done in Blue at 250 GeV (second order) spin resonance: - vertical

381 views • 4 slides
Augmented Cognition Augmented Cognition systems use real-time cognitive state data to adapt

Augmented Cognition Augmented Cognition systems use real-time cognitive state data to adapt

Platform-Based Design of Augmented Cognition Systems Latosha Marshall &amp; Colby Raley ENSE623 Fall 2004 Design &amp; implementation of Augmented Cognition systems: Modular design can make it possible Platform-based design makes it feasible

185 views • 15 slides
Design Space Optimization of Embedded Memory Design Space Optimization of Embedded Memory Systems

Design Space Optimization of Embedded Memory Design Space Optimization of Embedded Memory Systems

Design Space Optimization of Embedded Memory Design Space Optimization of Embedded Memory Systems via Data Remapping Systems via Data Remapping Krishna V. Palem, Rodric M. Rabbah, Vincent J. Mooney III, Pinar Korkmaz and Kiran Puttaswamy Center

775 views • 38 slides
Data Driven Algorithm Design Maria-Florina (Nina) Balcan Carnegie Mellon University Analysis and

Data Driven Algorithm Design Maria-Florina (Nina) Balcan Carnegie Mellon University Analysis and

Data Driven Algorithm Design Maria-Florina (Nina) Balcan Carnegie Mellon University Analysis and Design of Algorithms Classic algo design: solve a worst case instance. Easy domains, have optimal poly time algos. E.g., sorting, shortest

768 views • 43 slides
TS 83 DORMA DORMA TS 83 Easy-action Door Closer Easy-action door closer Data and features TS

TS 83 DORMA DORMA TS 83 Easy-action Door Closer Easy-action door closer Data and features TS

Easy-action Door Closer TS 83 DORMA DORMA TS 83 Easy-action Door Closer Easy-action door closer Data and features TS 83 with thinking backcheck Variable Spring EN EN closing force strength 2 6 7 Easy to fix and even

178 views • 6 slides
Data Movement Instructions Systems Design &amp; Programming CMPE 310 Intel Assembly Data

Data Movement Instructions Systems Design &amp; Programming CMPE 310 Intel Assembly Data

Data Movement Instructions Systems Design &amp; Programming CMPE 310 Intel Assembly Data Movement Instruction: mov (covered already) push, pop lea (mov and offset) lds, les, lfs, lgs, lss movs, lods, stos ins, outs

406 views • 11 slides
Learning Faster from Easy Data II Wouter Koolen Tim van Erven Aim of the Workshop

Learning Faster from Easy Data II Wouter Koolen Tim van Erven Aim of the Workshop

Learning Faster from Easy Data II Wouter Koolen Tim van Erven Aim of the Workshop Minimax analysis gives robust algorithms But in common easy cases these are overly conservative Large gap between performance predicted by

347 views • 32 slides
Design Innovation in Complex Systems Design: Integrating Design Thinking and Systems Thinking

Design Innovation in Complex Systems Design: Integrating Design Thinking and Systems Thinking

ASME International Design Engineering Technical Conference, Design Theory and Methodology (DTM) Conference, Presentation and Extended Abstract, 52938, August 2020. Design Innovation in Complex Systems Design: Integrating Design Thinking and

646 views • 5 slides
CSAR CEBALLOS PhD student Supervisor DR. PROF. ANTNIO CARLOS ROQUE Department of Physics

CSAR CEBALLOS PhD student Supervisor DR. PROF. ANTNIO CARLOS ROQUE Department of Physics

CSAR CEBALLOS PhD student Supervisor DR. PROF. ANTNIO CARLOS ROQUE Department of Physics DR. PROF. RICARDO LEO Department of Physiology University of So Paulo SUBTHRESHOLD ACTIVE CONDUCTANCES SHAPE THE EPSPS IN CORTICAL NEURONS

536 views • 15 slides
School and teacher characteristics vs. student progress Sandra Sousa, Miguel Portela and Carla S

School and teacher characteristics vs. student progress Sandra Sousa, Miguel Portela and Carla S

School and teacher characteristics vs. student progress Sandra Sousa, Miguel Portela and Carla S September 2017 Stata UGM 2017, Portugal Universidade do Porto 1 / 37 School characteristics Motivation The decision to invest in education

649 views • 37 slides
Flavor Violation and Electroweak Baryogenesis Jing Shu (arXiv:1609.09849) ITP-CAS Oct 27, 2017

Flavor Violation and Electroweak Baryogenesis Jing Shu (arXiv:1609.09849) ITP-CAS Oct 27, 2017

Flavor Violation and Electroweak Baryogenesis Jing Shu (arXiv:1609.09849) ITP-CAS Oct 27, 2017 Jing Shu | Oct 27, 2017 1 / 42 The Matter/Energy Budget of our Universe Cosmological Parameters from Planck Planck 2015 Fit of the base CDM at

800 views • 43 slides
7/7/15 JaeRan Kim, Pact Camp, July 4, 2015

7/7/15 JaeRan Kim, Pact Camp, July 4, 2015

7/7/15 JaeRan Kim, Pact Camp, July 4, 2015 1 7/7/15 2 7/7/15 3 7/7/15 Study by Raleigh &amp; Kreider (in press) Using

615 views • 26 slides
s infrastructure for s infrastructure for Linguateca Linguateca Portuguese... and how it

s infrastructure for s infrastructure for Linguateca Linguateca Portuguese... and how it

s infrastructure for s infrastructure for Linguateca Linguateca Portuguese... and how it allows the Portuguese... and how it allows the detailed study of language varieties detailed study of language varieties Diana Santos Information

742 views • 41 slides
14.54 International Trade Lecture 22: Trade Policy (III) 14.54 Week 14 Fall 2016 14.54

14.54 International Trade Lecture 22: Trade Policy (III) 14.54 Week 14 Fall 2016 14.54

14.54 International Trade Lecture 22: Trade Policy (III) 14.54 Week 14 Fall 2016 14.54 (Week 14) Trade Policy (III) Fall 2016 1 / 23 Todays Plan Trade Policy as a Second Best Instrument 1 Strategic Trade Policy 2 The WTO 3

530 views • 23 slides
&quot;Strategic Subsidies for Green Goods&quot; Carolyn Fischer Stphane Caprice TSE-INRA

&quot;Strategic Subsidies for Green Goods&quot; Carolyn Fischer Stphane Caprice TSE-INRA

&quot;Strategic Subsidies for Green Goods&quot; Carolyn Fischer Stphane Caprice TSE-INRA September, 2015 Stphane Caprice (TSE-INRA) Discussion September, 2015 1 / 8 Content International trade Relevant markets: Green goods (like

300 views • 8 slides
Simplified high-speed high-distance list decoding for alternant codes cr.yp.to/papers.html

Simplified high-speed high-distance list decoding for alternant codes cr.yp.to/papers.html

Simplified high-speed high-distance list decoding for alternant codes cr.yp.to/papers.html #simplelist D. J. Bernstein University of Illinois at Chicago Thanks to: Cisco University Research Program And thanks to: NIST grant 60NANB10D263

580 views • 20 slides

More recommend