storage deduplication in cloud computing
play

Storage Deduplication in Cloud Computing Joo Paulo and Jos Pereira - PowerPoint PPT Presentation

Oct 15, 2023 •192 likes •618 views

Storage Deduplication in Cloud Computing Joo Paulo and Jos Pereira University of Minho July 2010 Joo Paulo and Jos Pereira Storage Deduplication in Cloud Computing Cloud Computing Overview Cloud Computing Cloud services allow clients

  1. Storage Deduplication in Cloud Computing João Paulo and José Pereira University of Minho July 2010 João Paulo and José Pereira Storage Deduplication in Cloud Computing

  2. Cloud Computing Overview Cloud Computing Cloud services allow clients to shift their data and applications into the “cloud“. These services run in a scalable and dependable infrastructure, which has a large server pool in several data centres. Virtualization Virtualization is a key aspect to achieve the Elasticity provided by cloud computing. Virtual Machines (VMs) can be deployed/migrated in few minutes. VMs Isolation allows a better management of resources and failures. João Paulo and José Pereira Storage Deduplication in Cloud Computing

  3. Cloud Computing Overview Cloud Computing Cloud services allow clients to shift their data and applications into the “cloud“. These services run in a scalable and dependable infrastructure, which has a large server pool in several data centres. Virtualization Virtualization is a key aspect to achieve the Elasticity provided by cloud computing. Virtual Machines (VMs) can be deployed/migrated in few minutes. VMs Isolation allows a better management of resources and failures. João Paulo and José Pereira Storage Deduplication in Cloud Computing

  4. Deduplication Cloud services store client’s data, applications and VMs images. Deduplication allows to: Decrease storage’s size. Optimize the management of storage’s data. Deduplication introduces overhead to the service. João Paulo and José Pereira Storage Deduplication in Cloud Computing

  5. Outline Shared Storage Deduplication 1 Experimental Evaluation - Preliminary Results 2 Conclusions 3 Future Work and Challenges 4 João Paulo and José Pereira Storage Deduplication in Cloud Computing

  6. Shared Storage Deduplication Scenario VM VM VM VM VM VM VM VM VM VM VM VM Groups of VMs run in different physical machines. Each VM has its own virtual disk. Virtual disks are kept in a shared storage. João Paulo and José Pereira Storage Deduplication in Cloud Computing

  7. Shared Storage Deduplication XEN Blktap mechanism Blktap Implemented within Xen. Allows to implement virtual block devices for Virtual Machines. User-level disk I/O interface (Tapdisk). Allows to have independent per-disk handler processes. Easy to implement Copy-on-Write. João Paulo and José Pereira Storage Deduplication in Cloud Computing

  8. Shared Storage Deduplication XEN Blktap mechanism Physical Machine 1 Physical Machine 2 VM1 VM2 VM3 Tap Tap Tap aio aio aio VM1 VM2 VM3 Disk Disk Disk João Paulo and José Pereira Storage Deduplication in Cloud Computing

  9. Shared Storage Deduplication XEN Blktap mechanism Physical Machine 1 Physical Machine 2 VM1 VM2 VM3 Read/ write Tap Tap Tap aio aio aio VM1 VM2 VM3 Disk Disk Disk João Paulo and José Pereira Storage Deduplication in Cloud Computing

  10. Shared Storage Deduplication XEN Blktap mechanism Physical Machine 1 Physical Machine 2 VM1 VM2 VM3 Read/ write Tap Tap Tap aio aio aio VM1 VM2 VM3 Disk Disk Disk João Paulo and José Pereira Storage Deduplication in Cloud Computing

  11. Shared Storage Deduplication XEN Blktap mechanism Physical Machine 1 Physical Machine 2 VM1 VM2 VM3 Read/ write Tap Tap Tap aio aio aio VM1 VM2 VM3 Disk Disk Disk João Paulo and José Pereira Storage Deduplication in Cloud Computing

  12. Shared Storage Deduplication Deduplication Challenges Deduplication is usually used for backup scenarios where data is practically immutable. In a virtualized scenario where stored data changes constantly, we must have in account: The overhead introduced by the deduplication algorithm. The best approach to find duplicated data, which must be transparent to the VMs. The metadata needed to share identical data. João Paulo and José Pereira Storage Deduplication in Cloud Computing

  13. Shared Storage Deduplication Deduplication Algorithm Physical Machine 1 DHT Physical Machine 2 VM1 VM2 VM3 Tap Extend Tap disk Server disk Shared Storage João Paulo and José Pereira Storage Deduplication in Cloud Computing

  14. Shared Storage Deduplication Deduplication Algorithm Physical Machine 1 DHT Physical Machine 2 VM1 VM2 VM3 Read/ write Tap Extend Tap disk Server disk Shared Storage João Paulo and José Pereira Storage Deduplication in Cloud Computing

  15. Shared Storage Deduplication Deduplication Algorithm Physical Machine 1 DHT Physical Machine 2 VM1 VM2 VM3 V‐>P … V‐>P … V‐>P Read/ … … write … … Tap Extend Tap disk Server disk Shared Storage João Paulo and José Pereira Storage Deduplication in Cloud Computing

  16. Shared Storage Deduplication Deduplication Algorithm Physical Machine 1 DHT Physical Machine 2 VM1 VM2 VM3 V‐>P … V‐>P … V‐>P Read/ … … write … … Tap Extend Tap disk Server disk Shared Storage João Paulo and José Pereira Storage Deduplication in Cloud Computing

  17. Shared Storage Deduplication Deduplication Algorithm Physical Machine 1 DHT Physical Machine 2 VM1 VM2 VM3 Write Dirty addresses Dirty Dirty addresses addresses Tap Extend Tap disk Server disk Shared Storage João Paulo and José Pereira Storage Deduplication in Cloud Computing

  18. Shared Storage Deduplication Deduplication Algorithm Physical Machine 1 DHT Physical Machine 2 VM1 VM2 VM3 V‐>P … … COW Share Share Tap Extend Tap disk Server disk Shared Storage João Paulo and José Pereira Storage Deduplication in Cloud Computing

  19. Shared Storage Deduplication Deduplication Algorithm Physical Machine 1 DHT Physical Machine 2 VM1 VM2 VM3 V‐>P … … COW Share Share Tap Extend Tap disk Server disk Shared Storage João Paulo and José Pereira Storage Deduplication in Cloud Computing

  20. Shared Storage Deduplication Deduplication Algorithm Physical Machine 1 DHT Physical Machine 2 VM1 VM2 VM3 V‐>P … Hash‐>(Padd,Cont) … COW Share Share Tap Extend Tap disk Server disk Shared Storage João Paulo and José Pereira Storage Deduplication in Cloud Computing

  21. Shared Storage Deduplication Deduplication Algorithm Physical Machine 1 DHT Physical Machine 2 VM1 VM2 VM3 V‐>P … … Free blocks update queue Share Share Tap Extend Tap disk Server disk Shared Storage João Paulo and José Pereira Storage Deduplication in Cloud Computing

  22. Shared Storage Deduplication Deduplication Algorithm Physical Machine 1 DHT Physical Machine 2 VM1 VM2 VM3 Write COW COW Dirty COW addresses Addresses Addresses Tap Extend Tap disk Server disk free free blocks blocks buffer buffer Shared Storage João Paulo and José Pereira Storage Deduplication in Cloud Computing

  23. Shared Storage Deduplication Deduplication Algorithm Physical Machine 1 DHT Physical Machine 2 VM1 VM2 VM3 GC GC Tap Extend Tap disk Server disk free free blocks blocks buffer buffer Shared Storage João Paulo and José Pereira Storage Deduplication in Cloud Computing

  24. Shared Storage Deduplication Deduplication Algorithm Physical Machine 1 DHT Physical Machine 2 VM1 VM2 VM3 Hash‐>(Padd,Cont) GC GC Tap Extend Tap disk Server disk free free blocks blocks buffer buffer Shared Storage João Paulo and José Pereira Storage Deduplication in Cloud Computing

  25. Shared Storage Deduplication Deduplication Algorithm Physical Machine 1 DHT Physical Machine 2 VM1 VM2 VM3 Free blocks queue GC GC Tap Extend Tap disk Server disk free free blocks blocks buffer buffer Shared Storage João Paulo and José Pereira Storage Deduplication in Cloud Computing

  26. Shared Storage Deduplication Deduplication Algorithm Physical Machine 1 DHT Physical Machine 2 VM1 VM2 VM3 Free blocks queue GC/ GC/ Share Share Tap Extend Tap disk Server disk free free blocks blocks buffer buffer Shared Storage João Paulo and José Pereira Storage Deduplication in Cloud Computing

  27. Shared Storage Deduplication Deduplication Algorithm Physical Machine 1 DHT Physical Machine 2 VM1 VM2 VM3 Free blocks queue GC/ Share Tap Extend Tap disk Server disk free free blocks blocks buffer buffer Shared Storage João Paulo and José Pereira Storage Deduplication in Cloud Computing

  28. Shared Storage Deduplication Deduplication Algorithm Physical Machine 1 DHT Physical Machine 2 VM1 VM2 VM3 Free blocks queue GC/ Share Tap Extend Tap disk Server disk free free blocks blocks buffer buffer Shared Storage João Paulo and José Pereira Storage Deduplication in Cloud Computing

  29. Experimental Evaluation - Preliminary Results Outline Shared Storage Deduplication 1 Experimental Evaluation - Preliminary Results 2 Conclusions 3 Future Work and Challenges 4 João Paulo and José Pereira Storage Deduplication in Cloud Computing

  30. Experimental Evaluation - Preliminary Results Evaluated Prototype Physical Machine 1 VM1 VM2 Free blocks Without Distribution and queue Fault Tolerant design. GC/ Two Optimizations: Tap Share disk Set of mutexes for each VM’s Translation table. free blocks VM’s free blocks buffer buffer refilling granularity. Shared Storage João Paulo and José Pereira Storage Deduplication in Cloud Computing

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Cloud Computing and Cloud Storage By: Maurice Kelly History of Internet and Cloud Computing

Cloud Computing and Cloud Storage By: Maurice Kelly History of Internet and Cloud Computing

Cloud Computing and Cloud Storage By: Maurice Kelly History of Internet and Cloud Computing Internet began in the 1950s Internet has been quite revolutoinary due to its lightning fast communication privelages and data sharing. Cloud

378 views • 6 slides
HPDedup: A Hybrid Prioritized Data Deduplication Mechanism for Primary Storage in the Cloud Huijun

HPDedup: A Hybrid Prioritized Data Deduplication Mechanism for Primary Storage in the Cloud Huijun

HPDedup: A Hybrid Prioritized Data Deduplication Mechanism for Primary Storage in the Cloud Huijun Wu 1,4 , Chen Wang 2 , Yinjin Fu 3 , Sherif Sakr 1 , Liming Zhu 1,2 and Kai Lu 4 The University of New South Wales 1 Data61, CSIRO 2 PLA University

652 views • 33 slides
What is the Cloud? Simply put, Cloud Computing is the delivery of computing services,

What is the Cloud? Simply put, Cloud Computing is the delivery of computing services,

What is the Cloud? Simply put, Cloud Computing is the delivery of computing services, including Servers, Storage, Databases, Networking, Software, Analytics and Intelligence over the Internet (The Cloud) to offer faster innovation,

338 views • 13 slides
ChunkStash: Speeding Up Storage Deduplication using Flash Memory Biplob Debnath + , Sudipta

ChunkStash: Speeding Up Storage Deduplication using Flash Memory Biplob Debnath + , Sudipta

ChunkStash: Speeding Up Storage Deduplication using Flash Memory Biplob Debnath + , Sudipta Sengupta * , Jin Li * * Microsoft Research, Redmond (USA) + Univ. of Minnesota, Twin Cities (USA) Deduplication of Storage Detect and remove duplicate

626 views • 38 slides
Security and Privacy for Cloud Computing Refik Molva Cloud Computing Outsourcing

Security and Privacy for Cloud Computing Refik Molva Cloud Computing Outsourcing

Security and Privacy for Cloud Computing Refik Molva Cloud Computing Outsourcing Storage Computation High availability No maintenance Decreased Costs Elasticity & Flexibility Security and Privacy for Cloud

529 views • 36 slides
Distributed and Cloud Storage Systems Corso di Sistemi Distribuiti e Cloud Computing A.A. 2018/19

Distributed and Cloud Storage Systems Corso di Sistemi Distribuiti e Cloud Computing A.A. 2018/19

Universit degli Studi di Roma Tor Vergata Dipartimento di Ingegneria Civile e Ingegneria Informatica Distributed and Cloud Storage Systems Corso di Sistemi Distribuiti e Cloud Computing A.A. 2018/19 Valeria Cardellini Why to scale

396 views • 37 slides
Distributed and Cloud Storage Systems Corso di Sistemi Distribuiti e Cloud Computing A.A. 2017/18

Distributed and Cloud Storage Systems Corso di Sistemi Distribuiti e Cloud Computing A.A. 2017/18

Universit degli Studi di Roma Tor Vergata Dipartimento di Ingegneria Civile e Ingegneria Informatica Distributed and Cloud Storage Systems Corso di Sistemi Distribuiti e Cloud Computing A.A. 2017/18 Valeria Cardellini Why to scale

1.15k views • 82 slides
Limitlessly Scalable Storage for Capacity-Intensive Computing Meet Cloudian S3-compatible

Limitlessly Scalable Storage for Capacity-Intensive Computing Meet Cloudian S3-compatible

Limitlessly Scalable Storage for Capacity-Intensive Computing Meet Cloudian S3-compatible object storage Appliances and Software-defined Storage Start small, scale to over an exabyte Hybrid cloud and multi-cloud ready 2 See

257 views • 7 slides
Cloud Computing & Cloud Models Cloud Models Topics Defining cloud computing

Cloud Computing & Cloud Models Cloud Models Topics Defining cloud computing

Cloud Computing & Cloud Models Cloud Models Topics Defining cloud computing Understanding: Distributed Application Design Resource Management Automation Virtualized Computing Environments High-performance Computing

1.02k views • 49 slides
Outline Get Off of My Cloud 8271 discussion of cloud computing security (combined)

Outline Get Off of My Cloud 8271 discussion of cloud computing security (combined)

Outline Get Off of My Cloud 8271 discussion of cloud computing security (combined) Administrative discussions Stephen McCamant University of Minnesota Multi-Cloud Oblivious Storage Old and new topics in security Cloud threats, old and new

645 views • 8 slides
CS2 : A Searchable Cryptographic Cloud Storage System Seny Kamara (MSR) Charalampos Papamanthou

CS2 : A Searchable Cryptographic Cloud Storage System Seny Kamara (MSR) Charalampos Papamanthou

CS2 : A Searchable Cryptographic Cloud Storage System Seny Kamara (MSR) Charalampos Papamanthou (UC Berkeley) Tom Roeder (MSR) Cloud Computing Cloud Computing o Main concern o will my data be safe? o will anyone see it? o can anyone modify it?

960 views • 42 slides
Research in an Open Cloud Exchange CLOUD COMPUTING IS HAVING A DRAMATIC IMPACT On-demand

Research in an Open Cloud Exchange CLOUD COMPUTING IS HAVING A DRAMATIC IMPACT On-demand

Research in an Open Cloud Exchange CLOUD COMPUTING IS HAVING A DRAMATIC IMPACT On-demand access Economies of scale All compute/storage will move to the cloud? Todays IaaS clouds One company responsible for implementing and

1.11k views • 65 slides
Cloud Storage Nabil Abdennadher nabil.abdennadher@hesge.ch 1 Cloud storage Objective

Cloud Storage Nabil Abdennadher nabil.abdennadher@hesge.ch 1 Cloud storage Objective

Cloud Storage Nabil Abdennadher nabil.abdennadher@hesge.ch 1 Cloud storage Objective Provide logical storage pools that abstract a more complex distributed infrastructure made of commodity hardware Separate storage service

627 views • 15 slides
The MNM-CloudLab -- Ideas, Concepts & Implemenation 17.7. 22.7. 2011 Nils gentschen Felde

The MNM-CloudLab -- Ideas, Concepts & Implemenation 17.7. 22.7. 2011 Nils gentschen Felde

DEUTSCH-FRANZSISCHE SOMMERUNIVERSITT UNIVERSIT DT FRANCO-ALLEMANDE FR NACHWUCHSWISSENSCHAFTLER 2011 POUR JEUNES CHERCHEURS 2011 CLOUD COMPUTING : CLOUD COMPUTING : CLOUD COMPUTING : CLOUD COMPUTING : DFIS ET OPPORTUNITS

965 views • 50 slides
Data Centers and Cloud Computing Data Centers Virtualization Cloud Computing

Data Centers and Cloud Computing Data Centers Virtualization Cloud Computing

Data Centers and Cloud Computing Data Centers Virtualization Cloud Computing Computer Science Computer Science Lecture 24, page 1 Data Centers Large server and storage farms 1000s of servers Many TBs or PBs of data

700 views • 14 slides
SADedupe: Skew Area Inline Deduplication for Distributed Storage Binqi Zhang , Bing Bing Zhou,

SADedupe: Skew Area Inline Deduplication for Distributed Storage Binqi Zhang , Bing Bing Zhou,

SADedupe: Skew Area Inline Deduplication for Distributed Storage Binqi Zhang , Bing Bing Zhou, Chen Wang * , Dong Yuan, Albert Y. Zomaya The University of Sydney, Sydney, Australia * CSIRO, Sydney, Australia 1 Introduction Deduplication

107 views • 7 slides
P2P Audio Streaming for the iPod Touch P2P Audio Streaming for the iPod Touch Student: Tran,

P2P Audio Streaming for the iPod Touch P2P Audio Streaming for the iPod Touch Student: Tran,

P2P Audio Streaming for the iPod Touch P2P Audio Streaming for the iPod Touch Student: Tran, Thanh Minh Tu Student: Tran, Thanh Minh Tu Advisor: Prof. Dr. Thomas Schmidt Advisor: Prof. Dr. Thomas Schmidt Information Engineering Information

718 views • 51 slides
15 Application Layer Application Layer Overall Architecture Overall Architecture Tracker

15 Application Layer Application Layer Overall Architecture Overall Architecture Tracker

Application Layer Application Layer File Distribution: Server-Client vs P2P File distribution time: server-client Question : How much time to distribute file Server server sequentially from one server to N peers ? u 2 F u 1 d 1 sends N

423 views • 5 slides
Models and Tools for the High-Level Simulation of a Name-Based Interdomain Routing Architecture

Models and Tools for the High-Level Simulation of a Name-Based Interdomain Routing Architecture

I NTRODUCTION P ROBLEM A PPLICATION -D RIVEN H IGH -L EVEL S IMULATION S IMULATION C OMPONENTS R ESULTS Models and Tools for the High-Level Simulation of a Name-Based Interdomain Routing Architecture Kari Visala, Andrew Keating Helsinki

236 views • 20 slides
DSF: A Common Platform For Distributed Systems Research and Development Chunqiang (CQ) Tang IBM

DSF: A Common Platform For Distributed Systems Research and Development Chunqiang (CQ) Tang IBM

IBM Research DSF: A Common Platform For Distributed Systems Research and Development Chunqiang (CQ) Tang IBM Research December 4, 2009 IBM Research IBM Research Motivation: a Personal but Common Experience I develop production-quality

480 views • 16 slides
P2P Conversational Services Sipping Peer-to-Peer Ad-Hoc , IETF #64 11-11-2005 Marco Tomsu,

P2P Conversational Services Sipping Peer-to-Peer Ad-Hoc , IETF #64 11-11-2005 Marco Tomsu,

P2P Conversational Services Sipping Peer-to-Peer Ad-Hoc , IETF #64 11-11-2005 Marco Tomsu, Alcatel Research & Innovation marco.tomsu@alcatel.de Outline P2P Services Dilemma Conversational Services Digest P2P SIP Basic Service

512 views • 10 slides
Mike Blows (UK) mike blows@hotmail.c om 20 long ye ar s! 5000 dya ds/ tria ds/ fa milie s the

Mike Blows (UK) mike blows@hotmail.c om 20 long ye ar s! 5000 dya ds/ tria ds/ fa milie s the

T e a c hing me nta l he a lth c olle a g ue s how the ir own a tta c hme nt stra te g ie s ma y distort the ir c linic a l re sponse s a nd judg e me nts, by using vide o c lips portra ying c hild a buse . Mike Blows (UK) mike

248 views • 13 slides
Comparative Advantage and Optimal Trade Taxes Arnaud Costinot (MIT), Dave Donaldson (MIT),

Comparative Advantage and Optimal Trade Taxes Arnaud Costinot (MIT), Dave Donaldson (MIT),

Comparative Advantage and Optimal Trade Taxes Arnaud Costinot (MIT), Dave Donaldson (MIT), Jonathan Vogel (Columbia) and Ivn Werning (MIT) June 2014 Motivation Two central questions... 1. Why do nations trade? 2. How should they

966 views • 81 slides
The Multilingual Language Library @ LREC 2012 Lets build it together! Nicoletta Calzolari w ith

The Multilingual Language Library @ LREC 2012 Lets build it together! Nicoletta Calzolari w ith

The Multilingual Language Library @ LREC 2012 Lets build it together! Nicoletta Calzolari w ith Riccardo Del Gratta, Francesca Frontini, Francesco Rubino, Irene Russo Istituto di Linguistica Computazionale - CNR - Pisa glottolo@ilc.cnr.it N.

454 views • 12 slides

More recommend