write amplification an analysis of in memory database
play

Write Amplification: An Analysis of In-Memory Database Durability - PowerPoint PPT Presentation

Jan 21, 2023 •31 likes •421 views

Write Amplification: An Analysis of In-Memory Database Durability Techniques Jaemyung Kim , Kenneth Salem, Khuzaima Daudjee University of Waterloo IMDM 2015 Durability Matters OLTP IMDB Orders of magnitude faster! I/O is unavoidable! (ACID,

  1. Write Amplification: An Analysis of In-Memory Database Durability Techniques Jaemyung Kim , Kenneth Salem, Khuzaima Daudjee University of Waterloo IMDM 2015

  2. Durability Matters OLTP IMDB Orders of magnitude faster! I/O is unavoidable! (ACID, Durability) Write I/O efficiency of in-memory DBMS is an important issue. cliparts from openclipart.org 2

  3. Write Amplification? Application λ In-memory Database λ P On-disk Database Persistent Storage 3

  4. Goal of Write Amplification Model Quantify and compare the I/O efficiency of the persistent storage management schemes Provide us with some insight into the different natures of update-in-place and copy-on-write storage managers Lower cost for operating a database management system (contributed by improved I/O efficiency) Lead to better system performance in situations that I/O capacity is constrained (restart recovery) The following is not our goals: Emulate a specific storage manager implementation Compare specific implemenations: e.g., Hekaton is better than H-Store 4

  5. Architectural Diversity in IMDB SM Two broad classes: Update In-Place and Copy-On-Write Update In-Place (UIP) UIP : conventional page-based (e.g., Shore-MT) random writes for checkpointing device sensitive: e.g., HDD vs. SSD UIP-S : snapshot checkpointing (e.g., H-Store, SiloR) Copy-On-Write (COW) COW-D : logging only (log-structured) database (e.g., Hekaton) COW-M : log-structured memory and disk datbases (e.g., RAMCloud) 5

  6. UIP: Page-level Checkpoint Example Space Constraint ( α ) = 1 . 2 × DBSize, PageSize=2 Example Update Sequence: I,D,B Memory: A B C D E F G H I J Disk: A B C D E F G H I J 0 0 DB LOG LOG I/O History: DB I/O History: I/O Per Update = # DBIO +# LogIO = # Updates 6

  7. UIP: Page-level Checkpoint Example Space Constraint ( α ) = 1 . 2 × DBSize, PageSize=2 Example Update Sequence: I,D,B Memory: A B C D E F G H I J Disk: A B C D E F G H I J I 0 DB LOG LOG I/O History: I DB I/O History: I/O Per Update = # DBIO +# LogIO = 0+1 = 1 # Updates 1 6

  8. UIP: Page-level Checkpoint Example Space Constraint ( α ) = 1 . 2 × DBSize, PageSize=2 Example Update Sequence: I,D,B Memory: A B C D E F G H I J Disk: A B C D E F G H I J I D DB LOG LOG I/O History: I,D DB I/O History: I/O Per Update = # DBIO +# LogIO = 0+2 = 1 # Updates 2 6

  9. UIP: Page-level Checkpoint Example Space Constraint ( α ) = 1 . 2 × DBSize, PageSize=2 Example Update Sequence: I,D,B Memory: A B C D E F G H I J Disk: A B C D E F G H I J 0 0 DB LOG LOG I/O History: I,D DB I/O History: C,D,I,J I/O Per Update = # DBIO +# LogIO = 4+2 = 3 # Updates 2 6

  10. UIP: Page-level Checkpoint Example Space Constraint ( α ) = 1 . 2 × DBSize, PageSize=2 Example Update Sequence: I,D,B Memory: A B C D E F G H I J Disk: A B C D E F G H I J B 0 DB LOG LOG I/O History: I,D,B DB I/O History: C,D,I,J I/O Per Update = # DBIO +# LogIO = 4+3 ≈ 2 . 33 # Updates 3 6

  11. UIP: Page-level Checkpoint Example Space Constraint ( α ) = 1 . 2 × DBSize, PageSize=2 Example Update Sequence: I,D,B Memory: A B C D E F G H I J Disk: A B C D E F G H I J B 0 DB LOG LOG I/O History: I,D,B DB I/O History: C,D,I,J I/O Per Update = # DBIO +# LogIO = 4+3 ≈ 2 . 33 # Updates 3 6

  12. UIP-S: Snapshot Checkpoint Example Space Constraint ( α ) = 1 . 2 × DB Size Example Update Sequence: I,D,B Memory: A B C D E F G H I J Disk: A B C D E F G H I J I D DB LOG LOG I/O History: I,D DB I/O History: I/O Per Update = # DBIO +# LogIO = 0+2 = 1 # Updates 2 7

  13. UIP-S: Snapshot Checkpoint Example Space Constraint ( α ) = 1 . 2 × DB Size Example Update Sequence: I,D,B Memory: A B C D E F G H I J Disk: A B C D E F G H I J 0 0 DB LOG LOG I/O History: I,D DB I/O History: A,B,C,D,E,F,G,H,I,J I/O Per Update = # DBIO +# LogIO = 10+2 = 6 # Updates 2 7

  14. UIP-S: Snapshot Checkpoint Example Space Constraint ( α ) = 1 . 2 × DB Size Example Update Sequence: I,D,B Memory: A B C D E F G H I J Disk: A B C D E F G H I J B 0 DB LOG LOG I/O History: I,D,B DB I/O History: A,B,C,D,E,F,G,H,I,J I/O Per Update = # DBIO +# LogIO = 10+3 ≈ 4 . 33 # Updates 3 7

  15. UIP-S: Snapshot Checkpoint Example Space Constraint ( α ) = 1 . 2 × DB Size Example Update Sequence: I,D,B Memory: A B C D E F G H I J Disk: A B C D E F G H I J B 0 DB LOG LOG I/O History: I,D,B DB I/O History: A,B,C,D,E,F,G,H,I,J I/O Per Update = # DBIO +# LogIO = 10+3 ≈ 4 . 33 # Updates 3 7

  16. Architectural Diversity in IMDB SM Two broad classes: Update In-Place and Copy-On-Write Update In-Place (UIP) UIP : conventional page-based (e.g., Shore-MT) random writes for checkpointing device sensitive: e.g., HDD vs. SSD UIP-S : snapshot checkpointing (e.g., H-Store, SiloR) Copy-On-Write (COW) COW-D : logging only (log-structured) database (e.g., Hekaton) COW-M : log-structured memory and disk datbases (e.g., RAMCloud) 8

  17. COW-D: Log-structured Disk Example Space Constraint ( α ) = 1 . 2 × DB Size Example Update Sequence: I,D,B Memory: A B C D E F G H I J Disk: H A E I F G D C B J 0 0 Log-structured DB DB I/O History: Read: Write: I/O Per Update = # ReadIO +# WriteIO = # Updates 9

  18. COW-D: Log-structured Disk Example Space Constraint ( α ) = 1 . 2 × DB Size Example Update Sequence: I,D,B Memory: A B C D E F G H I J Disk: H A E I F G D C B J I 0 Log-structured DB DB I/O History: Read: Write: I I/O Per Update = # ReadIO +# WriteIO = 0+1 = 1 # Updates 1 9

  19. COW-D: Log-structured Disk Example Space Constraint ( α ) = 1 . 2 × DB Size Example Update Sequence: I,D,B Memory: A B C D E F G H I J Disk: H A E I F G D C B J I D Log-structured DB DB I/O History: Read: Write: I,D I/O Per Update = # ReadIO +# WriteIO = 0+2 = 1 # Updates 2 9

  20. COW-D: Log-structured Disk Example Space Constraint ( α ) = 1 . 2 × DB Size Example Update Sequence: I,D,B Memory: A B C D E F G H I J Disk: I F G D C B J I D H A E Log-structured DB DB I/O History: Read: H,A,E Write: I,D,H,A,E I/O Per Update = # ReadIO +# WriteIO = 3+5 = 4 # Updates 2 9

  21. COW-D: Log-structured Disk Example Space Constraint ( α ) = 1 . 2 × DB Size Example Update Sequence: I,D,B Memory: A B C D E F G H I J Disk: F G D C B J I D H A E 0 Log-structured DB DB I/O History: Read: H,A,E,I Write: I,D,H,A,E I/O Per Update = # ReadIO +# WriteIO = 4+5 = 4 . 5 # Updates 2 9

  22. COW-D: Log-structured Disk Example Space Constraint ( α ) = 1 . 2 × DB Size Example Update Sequence: I,D,B Memory: A B C D E F G H I J Disk: F G D C B J I D H A E B Log-structured DB DB I/O History: Read: H,A,E,I Write: I,D,H,A,E,B I/O Per Update = # ReadIO +# WriteIO = 4+6 ≈ 3 . 33 # Updates 3 9

  23. COW-D: Log-structured Disk Example Space Constraint ( α ) = 1 . 2 × DB Size Example Update Sequence: I,D,B Memory: A B C D E F G H I J Disk: F G D C B J I D H A E B Log-structured DB DB I/O History: Read: H,A,E,I Write: I,D,H,A,E,B I/O Per Update = # ReadIO +# WriteIO = 4+6 ≈ 3 . 33 # Updates 3 9

  24. COW-M: Log-structured Memory Example Space Constraint ( α ) = 1 . 2 × DB Size Example Update Sequence: I,D,B Memory: H A E I F G D C B J I Log-structured IMDB Disk: H A E I F G D C B J 0 0 Log-structured DB DB I/O History: Write: I/O Per Update = # WriteIO # Updates = 10

  25. COW-M: Log-structured Memory Example Space Constraint ( α ) = 1 . 2 × DB Size Example Update Sequence: I,D,B Memory: H A E F G D C B J I D Log-structured IMDB Disk: H A E I F G D C B J I 0 Log-structured DB DB I/O History: Write: I I/O Per Update = # WriteIO # Updates = 1 1 = 1 10

  26. COW-M: Log-structured Memory Example Space Constraint ( α ) = 1 . 2 × DB Size Example Update Sequence: I,D,B Memory: H A E F G C B J I D B Log-structured IMDB Disk: H A E I F G D C B J I D Log-structured DB DB I/O History: Write: I,D I/O Per Update = # WriteIO # Updates = 2 2 = 1 10

  27. COW-M: Log-structured Memory Example Space Constraint ( α ) = 1 . 2 × DB Size Example Update Sequence: I,D,B Memory: F G C B J I D H A E B Log-structured IMDB Disk: F G D C B J I D H A E 0 Log-structured DB DB I/O History: Write: I,D,H,A,E I/O Per Update = # WriteIO # Updates = 5 2 = 2 . 5 10

  28. COW-M: Log-structured Memory Example Space Constraint ( α ) = 1 . 2 × DB Size Example Update Sequence: I,D,B Memory: F G C J I D H A E B Log-structured IMDB Disk: F G D C B J I D H A E B Log-structured DB DB I/O History: Write: I,D,H,A,E,B I/O Per Update = # WriteIO # Updates = 6 3 = 2 10

  29. COW-M: Log-structured Memory Example Space Constraint ( α ) = 1 . 2 × DB Size Example Update Sequence: I,D,B Memory: F G C J I D H A E B Log-structured IMDB Disk: F G D C B J I D H A E B Log-structured DB DB I/O History: Write: I,D,H,A,E,B I/O Per Update = # WriteIO # Updates = 6 3 = 2 10

  30. What We Can Do Using WAF Model Compare UIP and COW Analyze the effect of persistent storage utilization ( α ) Analyze the effect of update workload (uniform vs. skew) Analyze the effect of page size on UIP Find best page size for storage-specific performance characteristic Analyze the effect of SSD vs. HDD persistent storage Weight random and sequential I/O differently, depending on the device type Application λ In-memory Database I/O Per Update = λ P λ λ P On-disk Database Persistent Storage 11

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

MatrixKV: Reducing Write Stalls and Write Amplification in LSM-tree Based KV Stores with a Matrix

MatrixKV: Reducing Write Stalls and Write Amplification in LSM-tree Based KV Stores with a Matrix

MatrixKV: Reducing Write Stalls and Write Amplification in LSM-tree Based KV Stores with a Matrix Container in NVM Ting Yao 1 , Yiwen Zhang 1 , Jiguang Wan 1 , Qiu Cui 2 , Liu Tang 2 , Hong Jiang 3 , Changsheng Xie 1 , and Xubin He 4 1 Huazhong

545 views • 27 slides
Reducing Write Amplification of Flash Storage through Cooperative Data Management with NVM 32nd

Reducing Write Amplification of Flash Storage through Cooperative Data Management with NVM 32nd

Chungbuk National University Reducing Write Amplification of Flash Storage through Cooperative Data Management with NVM 32nd International Conference on Massive Storage Systems and Technology (MSST) May, 2016 Eunji Lee , Chungbuk Natational

268 views • 22 slides
DATA ANALYSIS AND DEEP LEARNING CS 8803 // FALL 2018 // Sneha Venkatachalam Main Memory

DATA ANALYSIS AND DEEP LEARNING CS 8803 // FALL 2018 // Sneha Venkatachalam Main Memory

DATA ANALYSIS AND DEEP LEARNING CS 8803 // FALL 2018 // Sneha Venkatachalam Main Memory Database Systems: An Overview IEEE 1992 1 TODAYS PAPER Main Memory Database Systems: An Overview AUTHORS Hector Garcia-Molina and Kenneth

729 views • 43 slides
Main Memory by J. Nelson Amaral Types of Memories

Main Memory by J. Nelson Amaral Types of Memories

Main Memory by J. Nelson Amaral Types of Memories Read/Write Memory (RWM): we can store and retrieve data. the time required to read or Random Access Memory (RAM): write a bit of memory is independent of

847 views • 43 slides
An In-Depth Analysis of Data Aggregation Cost Factors in a Columnar In-Memory Database Stephan

An In-Depth Analysis of Data Aggregation Cost Factors in a Columnar In-Memory Database Stephan

An In-Depth Analysis of Data Aggregation Cost Factors in a Columnar In-Memory Database Stephan Mller, Hasso Plattner Enterprise Platform and Integration Concepts Hasso Plattner Institute, Potsdam (Germany) DOLAP 2012 November 2nd Agenda

319 views • 20 slides
1 Implementation Snoop Caches Implementing Snooping Caches Write Races: Multiple processors

1 Implementation Snoop Caches Implementing Snooping Caches Write Races: Multiple processors

An Example Snoopy Protocol Invalidation protocol, write-back cache Each block of memory is in one state: Cache Coherence and Memory Clean in all caches and up-to-date in memory (Shared) Consistency OR Dirty in exactly one cache

507 views • 3 slides
CS411 than $50,000 and pay taxes in Champaign-Urbana. Database Systems Could write in

CS411 than $50,000 and pay taxes in Champaign-Urbana. Database Systems Could write in

How do we query (specify what info we want from) the database? Find all the employees who earn more CS411 than $50,000 and pay taxes in Champaign-Urbana. Database Systems Could write in C++/Java, but who would want to? 05: Relational

367 views • 8 slides
Lehrstuhl fr Systemsicherheit Amplification DDoS Attacks Marc Khrer SPRING 9 Bochum, 31.

Lehrstuhl fr Systemsicherheit Amplification DDoS Attacks Marc Khrer SPRING 9 Bochum, 31.

Lehrstuhl fr Systemsicherheit Amplification DDoS Attacks Marc Khrer SPRING 9 Bochum, 31. Juli 2014 Outline Background UDP-based Amplification TCP-based Amplification Comparison UDP- / TCP-based Amplification 2 AMPLIFICATION

731 views • 27 slides
CS4617 Computer Architecture Lecture 4: Memory Hierarchy 2 Dr J Vaughan September 17, 2014 1/25

CS4617 Computer Architecture Lecture 4: Memory Hierarchy 2 Dr J Vaughan September 17, 2014 1/25

CS4617 Computer Architecture Lecture 4: Memory Hierarchy 2 Dr J Vaughan September 17, 2014 1/25 Write stall Occurs when processor must wait for writes to complete during write-through Write stalls can be reduced by using a write buffer

479 views • 25 slides
1 Executive Summary Database Proxies: Improves SQL read/write performance and

1 Executive Summary Database Proxies: Improves SQL read/write performance and

1 Executive Summary Database Proxies: Improves SQL read/write performance and reliability Deployment requires no application changes Demo 2 IMDG vs. Database Proxies Feature ProxySQL Automated Failover

335 views • 17 slides
UMBC A B M A L T F O U M B C I M Y O R T 1 (12/8/04) I E S R C E O V U

UMBC A B M A L T F O U M B C I M Y O R T 1 (12/8/04) I E S R C E O V U

Advanced VLSI Design Memory CMPE 640 Memory Can be categorized into: Read Write Memory (RWM) Random Access Memory (RAM): static SRAM (faster) verses dynamic DRAM (smaller) structures possible. Access time independent of physical

381 views • 21 slides
Architectural Principles for Database Systems on Storage-Class Memory Dr.-Ing. Ismail Oukid

Architectural Principles for Database Systems on Storage-Class Memory Dr.-Ing. Ismail Oukid

Architectural Principles for Database Systems on Storage-Class Memory Dr.-Ing. Ismail Oukid Dissertationspreis Vortrag BTW Rostock, March 8, 2019 Storage-Class Memory SCM is byte-addressable non-volatile memory Main Memory High Performance

600 views • 25 slides
Memory CoW in Xen Talk overview Why is CoW need? Memory CoW basics CoW mechanism:

Memory CoW in Xen Talk overview Why is CoW need? Memory CoW basics CoW mechanism:

Memory CoW in Xen Talk overview Why is CoW need? Memory CoW basics CoW mechanism: Establishing sharings No-Copy-on-Write Asynchronous memory reclamation Conclusions Why is memory CoW needed? In VMMs there is a

273 views • 15 slides
UMBC A B M A L T F O U M B C I M Y O R T 1 (Feb. 25, 2002) I E S R C E O

UMBC A B M A L T F O U M B C I M Y O R T 1 (Feb. 25, 2002) I E S R C E O

Systems Design & Programming Memory I CMPE 310 Memory Types Two basic types: ROM : Read-only memory RAM : Read-Write memory Four commonly used memories: ROM Flash (EEPROM) Static RAM (SRAM) Dynamic RAM (DRAM)

348 views • 12 slides
TOS Arno Puder 1 Objectives Learn how to read and write from memory using C-pointers

TOS Arno Puder 1 Objectives Learn how to read and write from memory using C-pointers

TOS Arno Puder 1 Objectives Learn how to read and write from memory using C-pointers Implement the very first TOS functions 2 Memory Access Memory often needs to be manipulated manually. Two operations: - Peek: peek (i.e.

380 views • 25 slides
Lect ure # 11 ADVANCED DATABASE SYSTEMS System Catalogs and Database Compression @

Lect ure # 11 ADVANCED DATABASE SYSTEMS System Catalogs and Database Compression @

Lect ure # 11 ADVANCED DATABASE SYSTEMS System Catalogs and Database Compression @ Andy_Pavlo // 15- 721 // Spring 2018 DATABASE TALK Oracle In-Memory Database Engine Monday February 26 th @ 4:30pm GHC 4401

709 views • 60 slides
Generalizing the Kawaguchi-Kyan Bound to Stochastic Parallel Machine Scheduling Sven Jger

Generalizing the Kawaguchi-Kyan Bound to Stochastic Parallel Machine Scheduling Sven Jger

Generalizing the Kawaguchi-Kyan Bound to Stochastic Parallel Machine Scheduling Sven Jger Martin Skutella Combinatorial Optimization and Graph Algorithms Technische Universitt Berlin 22 nd Combinatorial Optimization Workshop, Aussois w

525 views • 37 slides
Simple Graphs: 99 Isomorphism 67 306 99 145 306 67 isomorphism.1 isomorphism.2 Albert R

Simple Graphs: 99 Isomorphism 67 306 99 145 306 67 isomorphism.1 isomorphism.2 Albert R

The Graph Abstraction Mathematics for Computer Science MIT 6.042J/18.062J Same graph (different layouts) 122 257 122 145 257 Simple Graphs: 99 Isomorphism 67 306 99 145 306 67 isomorphism.1 isomorphism.2 Albert R Meyer April 1,

328 views • 5 slides
From Saving the Princess to From Saving the Princess to Saving the Cow Saving the Cow Content

From Saving the Princess to From Saving the Princess to Saving the Cow Saving the Cow Content

From Saving the Princess to From Saving the Princess to Saving the Cow Saving the Cow Content Lessons from MMOs to Facebook Games Content Lessons from MMOs to Facebook Games Anne Toole The Write Toole Writer/Narrative Designer Who am I?

1.52k views • 39 slides
Moving toward formalisation COMP60421 Sean Bechhofer sean.bechhofer@manchester.ac.uk

Moving toward formalisation COMP60421 Sean Bechhofer sean.bechhofer@manchester.ac.uk

Moving toward formalisation COMP60421 Sean Bechhofer sean.bechhofer@manchester.ac.uk (thanks to Bijan Parsia for slides) Previously... We started the acquisition process... We elicited tacit knowledge In a variety of ways

811 views • 52 slides
Searching with turn cost and related problems Spyros Angelopoulos CNRS-UPMC Joint work with

Searching with turn cost and related problems Spyros Angelopoulos CNRS-UPMC Joint work with

Searching with turn cost and related problems Spyros Angelopoulos CNRS-UPMC Joint work with Diogo Arsenio Christoph Drr Alex Lopez-Ortiz CNRS CNRS Univ. of Waterloo Monday, April 7, 14 Outline of the presentation Monday, April 7, 14

1.43k views • 109 slides
Parallelism vs. Concurrency Key concerns Concurrency: Parallelism: Correctly

Parallelism vs. Concurrency Key concerns Concurrency: Parallelism: Correctly

12/9/15 Parallelism vs. Concurrency Key concerns Concurrency: Parallelism: Correctly and efficiently manage Use extra resources to access to shared resources solve a problem

431 views • 3 slides
means of Natural Language Processing Viorel Morari Chair of Web Technology and Information

means of Natural Language Processing Viorel Morari Chair of Web Technology and Information

Mining Rhetorical Devices by means of Natural Language Processing Viorel Morari Chair of Web Technology and Information Systems Master Thesis Defense January 23 rd , 2018 viorel.morari@uni-weimar.de Prof. Dr. Benno Stein Advisor: Khalid

1.91k views • 151 slides
SANS ISC Free Software RMLLSEC16 Rump Session SANS Internet Storm Center Created in 2001 to

SANS ISC Free Software RMLLSEC16 Rump Session SANS Internet Storm Center Created in 2001 to

SANS ISC Free Software RMLLSEC16 Rump Session SANS Internet Storm Center Created in 2001 to track the Li0n worm Today, sensors covers 500K IPs from 50 countries Data collection, analysis and warning system (like weather forecasts)

509 views • 11 slides

More recommend