decentralized deduplication in san cluster file systems
play

Decentralized Deduplication in SAN Cluster File Systems Austin T. - PowerPoint PPT Presentation

Apr 04, 2023 •102 likes •1.28k views

Decentralized Deduplication in SAN Cluster File Systems Austin T. Clements Irfan Ahmad Murali Vilayannur Jinyuan Li VMware, Inc. MIT CSAIL Decentralized Deduplication in SAN Cluster File Systems Storage Area Networks Decentralized

  1. The Index 0e7a26.. Unique 15ba2b.. Shared 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Map from hashes to block locators, list sorted by hash Unique blocks are located in files and remain mutable Decentralized Deduplication in SAN Cluster File Systems

  2. The Index 0e7a26.. Unique 15ba2b.. Shared 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Map from hashes to block locators, list sorted by hash Unique blocks are located in files and remain mutable Decentralized Deduplication in SAN Cluster File Systems

  3. The Index 0e7a26.. Unique 15ba2b.. Shared 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Map from hashes to block locators, list sorted by hash Unique blocks are located in files and remain mutable A virtual arena stores COW references to all shared blocks Decentralized Deduplication in SAN Cluster File Systems

  4. The Index 0e7a26.. Unique 15ba2b.. Shared 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Map from hashes to block locators, list sorted by hash Unique blocks are located in files and remain mutable A virtual arena stores COW references to all shared blocks Decentralized Deduplication in SAN Cluster File Systems

  5. The Index 0e7a26.. Unique 15ba2b.. Shared 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Map from hashes to block locators, list sorted by hash Unique blocks are located in files and remain mutable A virtual arena stores COW references to all shared blocks Decentralized Deduplication in SAN Cluster File Systems

  6. Indexing and Duplicate Elimination 0e7a26.. Unique 15ba2b.. Shared 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  7. Indexing and Duplicate Elimination 0e7a26.. Unique 15ba2b.. Shared 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  8. Indexing and Duplicate Elimination 0e7a26.. Unique 15ba2b.. Shared 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  9. Indexing and Duplicate Elimination 6cd412.. 12067c.. c277d6.. 0e7a26.. Unique 15ba2b.. Shared 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  10. Indexing and Duplicate Elimination 12067c.. 0e7a26.. Unique 6cd412.. 15ba2b.. Shared c277d6.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  11. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. 15ba2b.. Shared 6cd412.. 6cd412.. c277d6.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  12. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 6cd412.. 15ba2b.. c277d6.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  13. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 6cd412.. 15ba2b.. c277d6.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  14. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 6cd412.. 15ba2b.. c277d6.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  15. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 15ba2b.. 6cd412.. 6cd412.. c277d6.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  16. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 15ba2b.. 6cd412.. c277d6.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  17. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 15ba2b.. 6cd412.. c277d6.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  18. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 15ba2b.. 6cd412.. c277d6.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  19. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 15ba2b.. 6cd412.. c277d6.. ab7373.. bc6887.. c277d6.. d5e341.. ? f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  20. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 15ba2b.. 6cd412.. c277d6.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  21. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  22. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  23. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  24. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  25. Indexing and Duplicate Elimination 0e7a26.. Unique 12067c.. Shared 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Decentralized Deduplication in SAN Cluster File Systems

  26. Phew. Decentralized Deduplication in SAN Cluster File Systems

  27. Evaluation Decentralized Deduplication in SAN Cluster File Systems

  28. Evaluation How much space does DeDe save? Decentralized Deduplication in SAN Cluster File Systems

  29. Evaluation How much space does DeDe save? How much overhead does DeDe introduce? Decentralized Deduplication in SAN Cluster File Systems

  30. Evaluation How much space does DeDe save? How much overhead does DeDe introduce? How fast can DeDe deduplicate? Decentralized Deduplication in SAN Cluster File Systems

  31. Space Savings: VDI Cluster Corporate Virtual Desktop Infrastructure cluster Desktop XP VM’s 6–12 months of active use ... Originally cloned from small number of base images Decentralized Deduplication in SAN Cluster File Systems

  32. Space Savings: VDI Cluster Corporate Virtual Desktop 113 VM’s Infrastructure cluster                            Desktop XP VM’s 6–12 months of active use ... Originally cloned from small number of base images   1.9 TB  Decentralized Deduplication in SAN Cluster File Systems

  33. Space Savings: VDI Cluster Corporate Virtual Desktop 113 VM’s Infrastructure cluster                            Desktop XP VM’s 6–12 months of active use ... Originally cloned from small number of base images   1.9 TB  1.3 TB Decentralized Deduplication in SAN Cluster File Systems

  34. Space Savings: VDI Cluster 1.3 TB Decentralized Deduplication in SAN Cluster File Systems

  35. Space Savings: VDI Cluster 237 GB 1.3 TB Decentralized Deduplication in SAN Cluster File Systems

  36. Space Savings: VDI Cluster 237 GB 1.3 TB 173 GB 61 GB unique shared Decentralized Deduplication in SAN Cluster File Systems

  37. Space Savings: VDI Cluster 237 GB 1.3 TB 173 GB 61 GB 2.7 GB unique shared Decentralized Deduplication in SAN Cluster File Systems

  38. Space Savings: VDI Cluster 237 GB 1.3 TB 173 GB 61 GB 2.7 GB unique shared 1.3 GB 194 MB 1.1 GB index file v. arena FS metadata Decentralized Deduplication in SAN Cluster File Systems

  39. Runtime Effects Write monitoring Disk array caching Decentralized Deduplication in SAN Cluster File Systems

  40. Runtime Effects Write monitoring Disk array caching Decentralized Deduplication in SAN Cluster File Systems

  41. Runtime Effects Write monitoring Disk array caching EMC CLARiiON CX3-40 Decentralized Deduplication in SAN Cluster File Systems

  42. Runtime Overhead: Write Monitoring Worst-case benchmark: 100% sequential write IO, No computation Decentralized Deduplication in SAN Cluster File Systems

  43. Runtime Overhead: Write Monitoring Worst-case benchmark: 100% sequential write IO, No computation Baseline Write Monitor CPU 33% 220% Decentralized Deduplication in SAN Cluster File Systems

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Decentralized Deduplication in SAN Cluster File Systems Austin T. Clements Irfan Ahmad Murali

Decentralized Deduplication in SAN Cluster File Systems Austin T. Clements Irfan Ahmad Murali

Decentralized Deduplication in SAN Cluster File Systems Austin T. Clements Irfan Ahmad Murali Vilayannur Jinyuan Li MIT CSAIL VMware, Inc. Abstract SANs provide multiple hosts with direct SCSI access to shared storage volumes. Regular

677 views • 14 slides
On the Effjciency of Decentralized File Storage for Personal Information Management Systems

On the Effjciency of Decentralized File Storage for Personal Information Management Systems

Gabriele DAngelo 2 1 Universidad Politcnica de Madrid 2 University of Bologna 3 University of Urbino Carlo Bo On the Effjciency of Decentralized File Storage for Personal Information Management Systems Mirko Zichichi 1 , 2 , Stefano

469 views • 22 slides
Profiling and diagnosing large-scale decentralized systems David Oppenheimer ROC Retreat

Profiling and diagnosing large-scale decentralized systems David Oppenheimer ROC Retreat

Profiling and diagnosing large-scale decentralized systems David Oppenheimer ROC Retreat Thursday, June 5, 2003 1 Why focus on P2P systems? There are a few real ones file trading, backup, IM Look a lot like other decentralized

685 views • 22 slides
Divide and conquer Shared disk cluster file systems shipped with the Linux kernel Udo Seidel

Divide and conquer Shared disk cluster file systems shipped with the Linux kernel Udo Seidel

Divide and conquer Shared disk cluster file systems shipped with the Linux kernel Udo Seidel Shared file systems Multiple server access same data Different approaches Network based, e.g. NFS, CIFS Clustered Shared disk,

768 views • 40 slides
Lustre at GSI - Evaluation of a cluster file system Walter Schn, GSI Walter Schn, GSI Topic

Lustre at GSI - Evaluation of a cluster file system Walter Schn, GSI Walter Schn, GSI Topic

Lustre at GSI - Evaluation of a cluster file system Walter Schn, GSI Walter Schn, GSI Topic Introduction motivation lustre test cluster Performance server controller, RAID level file systems parallel I/O

689 views • 22 slides
Evaluating selected cluster file systems with Parabench Internship report Authors: Marcel Krause;

Evaluating selected cluster file systems with Parabench Internship report Authors: Marcel Krause;

Evaluating selected cluster file systems with Parabench Internship report Authors: Marcel Krause; Jens Schlager, B.A. Tutors: Olga Mordvinova, Julian M. Kunkel Winter term 20092010 Overview Test Scenarios Test Patterns

542 views • 13 slides
Chapter 6: File Systems File systems n Files n Directories & naming n File system

Chapter 6: File Systems File systems n Files n Directories & naming n File system

Chapter 6: File Systems File systems n Files n Directories & naming n File system implementation n Example file systems Chapter 6 CS 1550, cs.pitt.edu 2 (originaly modified by Ethan Long-term information storage n Must store large amounts

1.01k views • 66 slides
Chapter 6: File Systems File systems Files Directories & naming File system

Chapter 6: File Systems File systems Files Directories & naming File system

Chapter 6: File Systems File systems Files Directories & naming File system implementation Example file systems Chapter 6 CMPS 111, UC Santa Cruz 2 Long-term information storage Must store large amounts of data

829 views • 57 slides
Efficient Locally Trackable from seed Deduplication in Replicated Systems Joo Barreto and

Efficient Locally Trackable from seed Deduplication in Replicated Systems Joo Barreto and

technology Efficient Locally Trackable from seed Deduplication in Replicated Systems Joo Barreto and Paulo Ferreira Distributed Systems Group INESC-ID/Technical University Lisbon, Portugal www.gsd.inesc-id.pt Instituto de Engenharia de

491 views • 24 slides
File Systems: All data structures are cached for better performance Create a new file

File Systems: All data structures are cached for better performance Create a new file

File Systems: Consistency Issues What about Multiple Updates? File systems maintain many data structures Several file system operations update multiple data structures Free list/bit vector Directories Examples: File headers and inode

257 views • 3 slides
an Object-Based File System for Large-Scale Federated IT Infrastructures Jan Stender, Zuse

an Object-Based File System for Large-Scale Federated IT Infrastructures Jan Stender, Zuse

an Object-Based File System for Large-Scale Federated IT Infrastructures Jan Stender, Zuse Institute Berlin HPC File Systems: From Cluster To Grid October 3-4, 2007 In this talk ... Introduction: Object-based File Systems Target Environment

525 views • 31 slides
Distributed File Systems Distributed File Systems A distributed file system (DFS) is a

Distributed File Systems Distributed File Systems A distributed file system (DFS) is a

Distributed File Systems Distributed File Systems A distributed file system (DFS) is a distributed Definitions implementation of a classical time-sharing model of a file Distributed system - A number of loosely coupled system. machines

997 views • 8 slides
File Systems: Semantics & Structure What is a File a file is a named collection of

File Systems: Semantics & Structure What is a File a file is a named collection of

5/16/2018 File Systems: Semantics & Structure What is a File a file is a named collection of information 11A. File Semantics primary roles of file system: 11B. Namespace Semantics to store and retrieve data 11C. File

373 views • 13 slides
File Systems: Semantics & Structure What is a File a file is a named collection of

File Systems: Semantics & Structure What is a File a file is a named collection of

5/15/2017 File Systems: Semantics & Structure What is a File a file is a named collection of information 11A. File Semantics primary roles of file system: 11B. Namespace Semantics to store and retrieve data 11C. File

352 views • 16 slides
Steganographic File Systems Steganographic File Systems 1 Conventional Protection Mechanisms in

Steganographic File Systems Steganographic File Systems 1 Conventional Protection Mechanisms in

Steganographic File Systems Steganographic File Systems 1 Conventional Protection Mechanisms in File S Systems t User Access Control The operating system is fully trusted to enforce the security policy. Is it good enough? Is it

888 views • 44 slides
CSN08101 Digital Forensics Lecture 8: File Systems Lecture 8: File Systems Module Leader: Dr

CSN08101 Digital Forensics Lecture 8: File Systems Lecture 8: File Systems Module Leader: Dr

CSN08101 Digital Forensics Lecture 8: File Systems Lecture 8: File Systems Module Leader: Dr Gordon Russell Lecturers: Robert Ludwiniak Objectives Investigative Process Analysis Framework last week File Systems File

795 views • 50 slides
in Group Messaging: Computational, Statistical, Optimal Lior Rotem Gil Segev Hebrew University

in Group Messaging: Computational, Statistical, Optimal Lior Rotem Gil Segev Hebrew University

Out-of-Band Authentication in Group Messaging: Computational, Statistical, Optimal Lior Rotem Gil Segev Hebrew University Messaging is Popular 2 Major Effort: E2E-Encrypted Messaging Government surveillance and/or coercion

585 views • 40 slides
Exploiting Latent I/O Asynchrony in Petascale Science Applications Patrick Widener, Mary Payne,

Exploiting Latent I/O Asynchrony in Petascale Science Applications Patrick Widener, Mary Payne,

Exploiting Latent I/O Asynchrony in Petascale Science Applications Patrick Widener, Mary Payne, Patrick Bridges University of New Mexico Matthew Wolf, Hasan Abbasi, Scott McManus, Karsten Schwan Georgia Institute of Technology The research

560 views • 17 slides
Web Annotations Building the Experience Annotation An annotation is something added. It is not

Web Annotations Building the Experience Annotation An annotation is something added. It is not

Web Annotations Building the Experience Annotation An annotation is something added. It is not the primary content, or even the secondary content. It is the association. Annotation (cont.) Annotations exist already - Links - Block quotes

452 views • 14 slides
Usability and Security of Out-Of-Band Channels in Secure Device Pairing Protocols Ronald Kainda,

Usability and Security of Out-Of-Band Channels in Secure Device Pairing Protocols Ronald Kainda,

Outline Introduction HISPs Proposed OOB Methods Experimental Design Results Analysis and Discussion Conclusion Usability and Security of Out-Of-Band Channels in Secure Device Pairing Protocols Ronald Kainda, Ivan Flechais, and A.W.

670 views • 41 slides
Instrumenting and Debugging FireSim-Simulated Designs https://fires.im @firesimproject MICRO

Instrumenting and Debugging FireSim-Simulated Designs https://fires.im @firesimproject MICRO

Instrumenting and Debugging FireSim-Simulated Designs https://fires.im @firesimproject MICRO 2019 Tutorial Speaker: Alon Amid Tutorial Roadmap Custom SoC Configuration FireMarshal RTL Generators Bare-metal & RISC-V Multi-level

413 views • 37 slides
Online Analysis and Telemetry WG Moderators: Michael Chynoweth & Ahmad Yasin Scalable Tools

Online Analysis and Telemetry WG Moderators: Michael Chynoweth & Ahmad Yasin Scalable Tools

Online Analysis and Telemetry WG Moderators: Michael Chynoweth & Ahmad Yasin Scalable Tools Workshop Solitude, Utah - July 11 th , 2018 End in Mind Optimization of one's system environment to safer, faster Online analysis of the

364 views • 4 slides
Out-of-band Flow Control for Reliable Multica st 00S-SIW-071 Harry Wolfson <

Out-of-band Flow Control for Reliable Multica st 00S-SIW-071 Harry Wolfson <

Out-of-band Flow Control for Reliable Multica st 00S-SIW-071 Harry Wolfson < HarryWolfson@LL.MI T.EDU > MIT Li ncoln Laboratory March 30, 2000 MIT L inco ln Lab oratory 1 hw 30-Mar-00 Outline Introd ucti on Ti me Management I

520 views • 26 slides
Search for tetrahedral states in Yb nuclei Search for tetrahedral states in Yb nuclei with N~90

Search for tetrahedral states in Yb nuclei Search for tetrahedral states in Yb nuclei with N~90

Search for tetrahedral states in Yb nuclei Search for tetrahedral states in Yb nuclei with N~90 through Coulomb excitation with N~90 through Coulomb excitation using HIE-ISOLDE and Miniball using HIE-ISOLDE and Miniball C.M. Petrache

283 views • 26 slides

More recommend