buffer pools
play

Buffer Pools Lecture # 05 Database Systems Andy Pavlo AP AP - PowerPoint PPT Presentation

Dec 22, 2023 •10 likes •770 views

Buffer Pools Lecture # 05 Database Systems Andy Pavlo AP AP Computer Science 15-445/15-645 Carnegie Mellon Univ. Fall 2018 2 UPCO M IN G DATABASE EVEN TS Relational AI Talk Wednesday Sep 12 th @ 4:00pm GHC 8102 MapD Talk

  1. Buffer Pools Lecture # 05 Database Systems Andy Pavlo AP AP Computer Science 15-445/15-645 Carnegie Mellon Univ. Fall 2018

  2. 2 UPCO M IN G DATABASE EVEN TS Relational AI Talk → Wednesday Sep 12 th @ 4:00pm → GHC 8102 MapD Talk → Thursday Sept 20 th @ 12pm → CIC 4 th Floor CMU 15-445/645 (Fall 2018)

  3. 3 DATABASE STO RAGE Problem #1: How the DBMS represents the database in files on disk. Problem #2: How the DBMS manages its memory and move data back-and-forth from disk. CMU 15-445/645 (Fall 2018)

  4. 4 DATABASE STO RAGE Spatial Control: → Where to write pages on disk. → The goal is to keep pages that are used together often as physically close together as possible on disk. Temporal Control: → When to read pages into memory, and when to write them to disk. → The goal is minimize the number of stalls from having to read data from disk. CMU 15-445/645 (Fall 2018)

  5. 5 DISK- O RIEN TED DBM S Get page #2 Execution Engine Buffer Pool Directory Header 2 Pointer to page #2 Memory Database File Directory Header Header Header Header Header … 1 2 3 4 5 Pages Disk CMU 15-445/645 (Fall 2018)

  6. 6 TO DAY'S AGEN DA Buffer Pool Manager Replacement Policies Allocation Policies Other Memory Pools CMU 15-445/645 (Fall 2018)

  7. 7 BUFFER PO O L O RGAN IZATIO N Buffer Memory region organized as an array Pool of fixed-size pages. frame1 page1 An array entry is called a frame. frame2 page3 frame3 When the DBMS requests a page, an exact copy is placed into one of these frame4 frames. page1 page2 page3 page4 On-Disk File CMU 15-445/645 (Fall 2018)

  8. 8 BUFFER PO O L M ETA- DATA Page Buffer The page table keeps track of pages Table Pool that are currently in memory. page1 frame1 page1 page3 frame2 page3 Also maintains additional meta-data frame3 per page: → Dirty Flag frame4 → Pin/Reference Counter page1 page2 page3 page4 On-Disk File CMU 15-445/645 (Fall 2018)

  9. 8 BUFFER PO O L M ETA- DATA Page Buffer The page table keeps track of pages Table Pool that are currently in memory. page1 frame1 page1 page3 frame2 page3 Also maintains additional meta-data frame3 per page: → Dirty Flag frame4 → Pin/Reference Counter page1 page2 page3 page4 On-Disk File CMU 15-445/645 (Fall 2018)

  10. 8 BUFFER PO O L M ETA- DATA Page Buffer The page table keeps track of pages Table Pool that are currently in memory. page1 frame1 page1 page3 frame2 page3 Also maintains additional meta-data frame3 per page: → Dirty Flag frame4 → Pin/Reference Counter page1 page2 page3 page4 On-Disk File CMU 15-445/645 (Fall 2018)

  11. 8 BUFFER PO O L M ETA- DATA Page Buffer The page table keeps track of pages Table Pool that are currently in memory. page1 frame1 page1 page3 frame2 page3 Also maintains additional meta-data frame3 page2 per page: → Dirty Flag frame4 → Pin/Reference Counter page1 page2 page3 page4 On-Disk File CMU 15-445/645 (Fall 2018)

  12. 8 BUFFER PO O L M ETA- DATA Page Buffer The page table keeps track of pages Table Pool that are currently in memory. page1 frame1 page1 page3 frame2 page3 Also maintains additional meta-data frame3 page2 per page: → Dirty Flag page2 frame4 → Pin/Reference Counter page1 page2 page3 page4 On-Disk File CMU 15-445/645 (Fall 2018)

  13. 9 LO CKS VS. LATCH ES Locks: → Protects the database's logical contents from other transactions. → Held for transaction duration. → Need to be able to rollback changes. Latches: → Protects the critical sections of the DBMS's internal data structure from other threads. → Held for operation duration. → Do not need to be able to rollback changes. CMU 15-445/645 (Fall 2018)

  14. 10 PAGE TABLE VS. PAGE DIRECTO RY The page directory is the mapping from page ids to page locations in the database files. → All changes must be recorded on disk to allow the DBMS to find on restart. The page table is the mapping from page ids to a copy of the page in buffer pool frames. → This is an in-memory data structure that does not need to be stored on disk. CMU 15-445/645 (Fall 2018)

  15. 11 M ULTIPLE BUFFER PO O LS The DBMS does not always have a single buffer pool for the entire system. → Multiple buffer pool instances → Per-database buffer pool → Per-page type buffer pool Helps reduce latch contention and improve locality. CMU 15-445/645 (Fall 2018)

  16. 12 PRE- FETCH IN G Disk Pages The DBMS can also prefetch pages based on a query plan. Q1 page0 → Sequential Scans page1 → Index Scans page2 Buffer Pool page3 page4 page5 CMU 15-445/645 (Fall 2018)

  17. 12 PRE- FETCH IN G Disk Pages The DBMS can also prefetch pages based on a query plan. Q1 page0 → Sequential Scans page1 → Index Scans page2 Buffer Pool page3 page0 page4 page5 CMU 15-445/645 (Fall 2018)

  18. 12 PRE- FETCH IN G Disk Pages The DBMS can also prefetch pages based on a query plan. page0 → Sequential Scans page1 → Index Scans Q1 page2 Buffer Pool page3 page0 page4 page1 page5 CMU 15-445/645 (Fall 2018)

  19. 12 PRE- FETCH IN G Disk Pages The DBMS can also prefetch pages based on a query plan. page0 → Sequential Scans page1 → Index Scans Q1 page2 Buffer Pool page3 page0 page4 page1 page5 CMU 15-445/645 (Fall 2018)

  20. 12 PRE- FETCH IN G Disk Pages The DBMS can also prefetch pages based on a query plan. page0 → Sequential Scans page1 → Index Scans Q1 page2 Buffer Pool page3 page3 page4 page1 page5 page2 CMU 15-445/645 (Fall 2018)

  21. 12 PRE- FETCH IN G Disk Pages The DBMS can also prefetch pages based on a query plan. page0 → Sequential Scans page1 → Index Scans page2 Q1 Buffer Pool page3 page3 page4 page1 page5 page2 CMU 15-445/645 (Fall 2018)

  22. 12 PRE- FETCH IN G Disk Pages The DBMS can also prefetch pages based on a query plan. page0 → Sequential Scans page1 → Index Scans page2 Buffer Pool page3 page3 page4 page4 Q1 page5 page5 CMU 15-445/645 (Fall 2018)

  23. 13 PRE- FETCH IN G index-page0 Disk Pages index-page1 index-page4 index-page0 index-page2 index-page3 index-page5 index-page6 index-page1 index-page2 Buffer Pool index-page3 index-page4 index-page5 CMU 15-445/645 (Fall 2018)

  24. 13 PRE- FETCH IN G index-page0 Disk Pages Q1 index-page1 index-page4 index-page0 index-page2 index-page3 index-page5 index-page6 index-page1 index-page2 Buffer Pool index-page3 index-page0 index-page4 index-page5 CMU 15-445/645 (Fall 2018)

  25. 13 PRE- FETCH IN G index-page0 Disk Pages index-page1 index-page4 index-page0 index-page2 index-page3 index-page5 index-page6 index-page1 Q1 index-page2 Buffer Pool index-page3 index-page0 index-page4 index-page1 index-page5 CMU 15-445/645 (Fall 2018)

  26. 13 PRE- FETCH IN G index-page0 Disk Pages index-page1 index-page4 index-page0 index-page2 index-page3 index-page5 index-page6 index-page1 Q1 index-page2 Buffer Pool index-page3 index-page0 index-page4 index-page1 index-page5 CMU 15-445/645 (Fall 2018)

  27. 14 SCAN SH ARIN G Queries are able to reuse data retrieved from storage or operator computations. → This is different from result caching. Allow multiple queries to attach to a single cursor that scans a table. → Queries do not have to be exactly the same. → Can also share intermediate results. CMU 15-445/645 (Fall 2018)

  28. 15 SCAN SH ARIN G If a query starts a scan and if there one already doing this, then the DBMS will attach to the second query's cursor. → The DBMS keeps track of where the second query joined with the first so that it can finish the scan when it reaches the end of the data structure. Fully supported in IBM DB2 and MSSQL. Oracle only supports cursor sharing for identical queries. CMU 15-445/645 (Fall 2018)

  29. 16 SCAN SH ARIN G Disk Pages Q1 SELECT SUM (val) FROM A Q1 page0 page1 page2 Buffer Pool page3 page4 page5 CMU 15-445/645 (Fall 2018)

  30. 16 SCAN SH ARIN G Disk Pages Q1 SELECT SUM (val) FROM A Q1 page0 page1 page2 Buffer Pool page3 page0 page4 page5 CMU 15-445/645 (Fall 2018)

  31. 16 SCAN SH ARIN G Disk Pages Q1 SELECT SUM (val) FROM A page0 page1 page2 Q1 Buffer Pool page3 page0 page4 page1 page5 page2 CMU 15-445/645 (Fall 2018)

  32. 16 SCAN SH ARIN G Disk Pages Q1 SELECT SUM (val) FROM A page0 page1 page2 Buffer Pool Q1 page3 page0 page4 page1 page5 page2 CMU 15-445/645 (Fall 2018)

  33. 16 SCAN SH ARIN G Disk Pages Q1 SELECT SUM (val) FROM A page0 page1 page2 Buffer Pool Q1 page3 page3 page4 page1 page5 page2 CMU 15-445/645 (Fall 2018)

  34. 16 SCAN SH ARIN G Disk Pages Q1 SELECT SUM (val) FROM A Q2 page0 Q2 SELECT AVG (val) FROM A page1 page2 Buffer Pool Q1 page3 page3 page4 page1 page5 page2 CMU 15-445/645 (Fall 2018)

  35. 16 SCAN SH ARIN G Disk Pages Q1 SELECT SUM (val) FROM A page0 Q2 SELECT AVG (val) FROM A page1 page2 Buffer Pool Q1 page3 Q2 page3 page4 page1 page5 page2 CMU 15-445/645 (Fall 2018)

  36. 16 SCAN SH ARIN G Disk Pages Q1 SELECT SUM (val) FROM A page0 Q2 SELECT AVG (val) FROM A page1 page2 Buffer Pool page3 page3 page4 page4 Q2 Q1 page5 page5 CMU 15-445/645 (Fall 2018)

  37. 16 SCAN SH ARIN G Disk Pages Q1 SELECT SUM (val) FROM A Q2 page0 Q2 SELECT AVG (val) FROM A page1 page2 Buffer Pool page3 page3 page4 page4 page5 page5 CMU 15-445/645 (Fall 2018)

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Buffer Manager Each relation is can be mapped onto many files (each file containing data from one

Buffer Manager Each relation is can be mapped onto many files (each file containing data from one

Buffer Management The database buffer is the mediator between the basic file system and the tuple-oriented file system. The buffer managers task is to make the pages addressable in main memory and to coordinate the writing of pages to disk

1.12k views • 8 slides
Introduction Buffer Overflows Buffer overflows were the most common form of security

Introduction Buffer Overflows Buffer overflows were the most common form of security

Introduction Buffer Overflows Buffer overflows were the most common form of security vulnerability in the 90s. Buffer overflows dominate in the area of remote network penetration vulnerabilities. CS 465 They represent one of the

651 views • 4 slides
Buffer Overflows with Content 2 A Process Stack Buffer Overflow Common Techniques employed

Buffer Overflows with Content 2 A Process Stack Buffer Overflow Common Techniques employed

1 Buffer Overflows with Content 2 A Process Stack Buffer Overflow Common Techniques employed in buffer overflow exploits to create backdoors Execution of additional network services via the INETD daemon The addition of new users

497 views • 30 slides
External buffer Raslan Darawsheh Mellanox External buffer First was introduced by Olivier

External buffer Raslan Darawsheh Mellanox External buffer First was introduced by Olivier

x External buffer Raslan Darawsheh Mellanox External buffer First was introduced by Olivier in his presentation in 2016. The support for External buffer has been there since DPDK 18.05 2 External buffer: Contd 3 External

544 views • 12 slides
Lab 2: Buffer Overflows Fengwei Zhang SUSTech CS 315 Computer Security 1 Buffer Overflows

Lab 2: Buffer Overflows Fengwei Zhang SUSTech CS 315 Computer Security 1 Buffer Overflows

Lab 2: Buffer Overflows Fengwei Zhang SUSTech CS 315 Computer Security 1 Buffer Overflows One of the most common vulnerabilities in software Programming languages commonly associated with buffer overflows including C and C++

1.14k views • 17 slides
Buffer Overflows and Defenses CS 419: Computer Security Lecture 10 and 11 Buffer Overflow a

Buffer Overflows and Defenses CS 419: Computer Security Lecture 10 and 11 Buffer Overflow a

Buffer Overflows and Defenses CS 419: Computer Security Lecture 10 and 11 Buffer Overflow a very common attack mechanism from 1988 Morris Worm to Code Red, Slammer, Sasser and many others prevention techniques known still of major concern

873 views • 53 slides
TinyOS Determine when Fill message Specify Pass buffer message buffer Network Communication

TinyOS Determine when Fill message Specify Pass buffer message buffer Network Communication

Inter-Node Communication General idea: Sender: TinyOS Determine when Fill message Specify Pass buffer message buffer Network Communication buffer with data Recipients to OS can be reused Computer Network Receiver:

176 views • 3 slides
Week 03 Lectures PostgreSQL Buffer Manager 1/95 PostgreSQL buffer manager: provides a shared

Week 03 Lectures PostgreSQL Buffer Manager 1/95 PostgreSQL buffer manager: provides a shared

Week 03 Lectures PostgreSQL Buffer Manager 1/95 PostgreSQL buffer manager: provides a shared pool of memory buffers for all backends all access methods get data from disk via buffer manager Buffers are located in a large region of shared

505 views • 25 slides
Riparian Buffer or Riparian Forest Buffer Equivalency Demonstration and Offsetting Water

Riparian Buffer or Riparian Forest Buffer Equivalency Demonstration and Offsetting Water

Implementation of Act 162 of 2014 Riparian Buffer or Riparian Forest Buffer Equivalency Demonstration and Offsetting Water Resources Advisory Committee August 12, 2015 Tom Wolf, Governor John Quigley, Secretary Agenda 1. Overview of Act 162

530 views • 37 slides
Chapter 10 Buffer Overflow Buffer Overflow Common attack mechanism first wide use by

Chapter 10 Buffer Overflow Buffer Overflow Common attack mechanism first wide use by

Chapter 10 Buffer Overflow Buffer Overflow Common attack mechanism first wide use by the Morris Worm in 1988 Prevention techniques known NX bit, stack canaries, ASLR Still of major concern Recent examples:

354 views • 21 slides
Patent Pools, Litigation and Innovation Jay Pil Choi (UNSW, Michigan State) Heiko Gerlach (U of

Patent Pools, Litigation and Innovation Jay Pil Choi (UNSW, Michigan State) Heiko Gerlach (U of

Patent Pools, Litigation and Innovation Jay Pil Choi (UNSW, Michigan State) Heiko Gerlach (U of Queensland) 2nd ATE Symposium at UNSW Sydney December 15, 2014 Patent Pools, Litigation and Innovation 1 Patent Pools and Antitrust patent

730 views • 44 slides
Buffer Trees Lars Arge. The Buffer Tree: A New Technique for Optimal I/O Algorithms . In

Buffer Trees Lars Arge. The Buffer Tree: A New Technique for Optimal I/O Algorithms . In

Buffer Trees Lars Arge. The Buffer Tree: A New Technique for Optimal I/O Algorithms . In Proceedings of Fourth Workshop on Algorithms and Data Structures (WADS), Lecture Notes in Computer Science Vol. 955, Springer-Verlag, 1995, 334-345. 1

518 views • 27 slides
AKC Education Webinar Series: Social Media Resources Buffer: What Is It? Buffer is a software

AKC Education Webinar Series: Social Media Resources Buffer: What Is It? Buffer is a software

AKC Education Webinar Series: Social Media Resources Buffer: What Is It? Buffer is a software application for the web and mobile, designed to manage accounts in social networks, by providing the means for a user to schedule posts to Twitter,

506 views • 35 slides
More Vulnerabilities (buffer overreads, format string, integer overflow, heap overflows) Chester

More Vulnerabilities (buffer overreads, format string, integer overflow, heap overflows) Chester

More Vulnerabilities (buffer overreads, format string, integer overflow, heap overflows) Chester Rebeiro Indian Institute of Technology Madras Buffer Overreads 2 Buffer Overread Example 3 Buffer Overread Example len read from command line

906 views • 76 slides
Buffer Software Security overflows and other memory safety vulnerabilities Buffer overflow

Buffer Software Security overflows and other memory safety vulnerabilities Buffer overflow

Last time We continued By launching our 1st section: Buffer Software Security overflows and other memory safety vulnerabilities Buffer overflow fundamentals This time We will finish up By looking at Buffer Overflow overflows

1.33k views • 103 slides
k + -buffer: Fragment Synchronized k-buffer I3D 2014 Andreas A. Vasilakis & Ioannis Fudos {

k + -buffer: Fragment Synchronized k-buffer I3D 2014 Andreas A. Vasilakis & Ioannis Fudos {

k + -buffer: Fragment Synchronized k-buffer I3D 2014 Andreas A. Vasilakis & Ioannis Fudos { abasilak,fudos } @cs.uoi.gr Department of Computer Science & Engineering, University of Ioannina, Greece 16 March 2014 Introduction

782 views • 44 slides
Using RDMA Efficiently for Key-Value Services Anuj Kalia (CMU) Michael Kaminsky (Intel Labs),

Using RDMA Efficiently for Key-Value Services Anuj Kalia (CMU) Michael Kaminsky (Intel Labs),

Using RDMA Efficiently for Key-Value Services Anuj Kalia (CMU) Michael Kaminsky (Intel Labs), David Andersen (CMU) 1 RDMA Remote Direct Memory Access: A network feature that allows direct access to the memory of a remote computer. 2 HERD

1.13k views • 34 slides
Practical Attacks against Mobile Device Management Solutions Michael Shaulov, CEO

Practical Attacks against Mobile Device Management Solutions Michael Shaulov, CEO

Practical Attacks against Mobile Device Management Solutions Michael Shaulov, CEO michael@lacoon.com Daniel Brodie, Sr Security Researcher daniel@lacoon.com About: Daniel Security researcher for nearly a decade From PC to Mobile

693 views • 54 slides
Banked Multiported Register Files for High-Frequency Superscalar Microprocessors Jessica H. Tseng

Banked Multiported Register Files for High-Frequency Superscalar Microprocessors Jessica H. Tseng

Banked Multiported Register Files for High-Frequency Superscalar Microprocessors Jessica H. Tseng and Krste Asanovi MIT Laboratory for Computer Science, Cambridge, MA 02139, USA ISCA2003 1 Motivation Increasing demand on number of

661 views • 30 slides
XDP (eXpress Data Path) as a building block for other FOSS projects Jesper Dangaard Brouer (Red

XDP (eXpress Data Path) as a building block for other FOSS projects Jesper Dangaard Brouer (Red

XDP (eXpress Data Path) as a building block for other FOSS projects Jesper Dangaard Brouer (Red Hat) Magnus Karlsson (Intel) FOSDEM 2019 Brussels, Feb 2019 1 Framing XDP XDP: new in-kernel programmable (eBPF) layer before netstack Similar

689 views • 34 slides
for General Purpose Processors Zhengrong Wang Prof. Tony Nowatzki 1 Computation & Memory

for General Purpose Processors Zhengrong Wang Prof. Tony Nowatzki 1 Computation & Memory

Stream-based Memory Specialization for General Purpose Processors Zhengrong Wang Prof. Tony Nowatzki 1 Computation & Memory Specialization SIMD Dataflow + + + + + + New ISA abstraction for / certain computation pattern. Core

774 views • 32 slides
universal composability from essentially any trusted setup Mike Rosulek | | CRYPTO 2012 .

universal composability from essentially any trusted setup Mike Rosulek | | CRYPTO 2012 .

universal composability from essentially any trusted setup Mike Rosulek | | CRYPTO 2012 . Example: Set intersection A B ( function evaluation ) Generate a fair coin toss ( randomized ) Online poker without a dealer ( reactive ) secure

882 views • 48 slides
Ticagrelor vs Aspirin in Patients undergoing Coronary- Artery Bypass Grafting Herib eribert

Ticagrelor vs Aspirin in Patients undergoing Coronary- Artery Bypass Grafting Herib eribert

The TiCAB Trial Ticagrelor vs Aspirin in Patients undergoing Coronary- Artery Bypass Grafting Herib eribert Schun unker ert, M , MD on behalf of the e TiCAB Investigators TiCAB an investigator initiated trial Sponsor Deutsches

822 views • 26 slides
WEN ETA JB? A 2 million dollars problem Date: 05/06/2019 For: SSTIC Presenters: Eloi

WEN ETA JB? A 2 million dollars problem Date: 05/06/2019 For: SSTIC Presenters: Eloi

WEN ETA JB? A 2 million dollars problem Date: 05/06/2019 For: SSTIC Presenters: Eloi Benoist-Vanderbeken, Fabien Perigaud Who are we Eloi Benoist-Vanderbeken Fabien Perigaud @elvanderb @0xf4b Working for Synacktiv: Offensive

883 views • 40 slides

More recommend