distributed storage allocation for high reliability
play

Distributed Storage Allocation for High Reliability Derek Leong 1 - PowerPoint PPT Presentation

Feb 20, 2024 •141 likes •757 views

Distributed Storage Allocation for High Reliability Derek Leong 1 Alex Dimakis 2 Tracey Ho 1 1 Department of Electrical Engineering California Institute of Technology Pasadena, California, USA 2 Department of Electrical Engineering University of

  1. Distributed Storage Allocation for High Reliability Derek Leong 1 Alex Dimakis 2 Tracey Ho 1 1 Department of Electrical Engineering California Institute of Technology Pasadena, California, USA 2 Department of Electrical Engineering University of Southern California Los Angeles, California, USA ICC 2010 2010-05-26

  2. Introduction Motivating Example A Water Analogy Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  3. Introduction Motivating Example A thirsty lion wanders across the savanna in search of water ... Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  4. Introduction Motivating Example ... he needs at least 1 ℓ of water to survive ... ≥ 1 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  5. Introduction Motivating Example ... suddenly, he finds five abandoned jerrycans ... ≥ 1 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  6. Introduction Motivating Example ... he chooses three jerrycans at random and starts chewing them open ... ≥ 1 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  7. Introduction Motivating Example Budget of 1.5 ℓ Survival Allocation Probability 1 A 1 0 0 0 ? 2 1 1 1 1 1 B ? 3 3 3 3 6 1 1 1 C ? 0 0 2 2 2 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  8. Introduction Motivating Example Allocation A 1 � 3-Subset 1 0 0 0 ≥ 1? 2 1 ✧ (i) 1 0 0 0 2 1 ✧ (ii) 1 0 0 0 2 1 ✧ (iii) 1 0 0 0 2 1 ✧ (iv) 1 0 0 0 2 1 ✧ (v) 1 0 0 0 2 1 ✧ (vi) 1 0 0 0 2 1 ✪ (vii) 1 0 0 0 2 1 ✪ (viii) 1 0 0 0 2 1 ✪ (ix) 1 0 0 0 2 1 ✪ (x) 1 0 0 0 2 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  9. Introduction Motivating Example Allocation A 1 � 3-Subset 1 0 0 0 ≥ 1? 2 1 ✧ (i) 1 0 0 0 2 1 ✧ (ii) 1 0 0 0 2 1 ✧ (iii) 1 0 0 0 2 60% 1 ✧ (iv) 1 0 0 0 2 1 ✧ (v) 1 0 0 0 Survival 2 1 ✧ (vi) 1 0 0 0 Probability 2 1 ✪ (vii) 1 0 0 0 2 1 ✪ (viii) 1 0 0 0 2 1 ✪ (ix) 1 0 0 0 2 1 ✪ (x) 1 0 0 0 2 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  10. Introduction Motivating Example Allocation B 1 1 1 1 1 � 3-Subset ≥ 1? 3 3 3 3 6 1 1 1 1 1 ✧ (i) 3 3 3 3 6 1 1 1 1 1 ✧ (ii) 3 3 3 3 6 1 1 1 1 1 ✪ (iii) 3 3 3 3 6 1 1 1 1 1 ✧ (iv) 3 3 3 3 6 1 1 1 1 1 ✪ (v) 3 3 3 3 6 1 1 1 1 1 ✪ (vi) 3 3 3 3 6 1 1 1 1 1 ✧ (vii) 3 3 3 3 6 1 1 1 1 1 ✪ (viii) 3 3 3 3 6 1 1 1 1 1 ✪ (ix) 3 3 3 3 6 1 1 1 1 1 ✪ (x) 3 3 3 3 6 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  11. Introduction Motivating Example Allocation B 1 1 1 1 1 � 3-Subset ≥ 1? 3 3 3 3 6 1 1 1 1 1 ✧ (i) 3 3 3 3 6 1 1 1 1 1 ✧ (ii) 3 3 3 3 6 1 1 1 1 1 ✪ (iii) 3 3 3 3 6 40% 1 1 1 1 1 ✧ (iv) 3 3 3 3 6 1 1 1 1 1 ✪ (v) Survival 3 3 3 3 6 1 1 1 1 1 ✪ (vi) Probability 3 3 3 3 6 1 1 1 1 1 ✧ (vii) 3 3 3 3 6 1 1 1 1 1 ✪ (viii) 3 3 3 3 6 1 1 1 1 1 ✪ (ix) 3 3 3 3 6 1 1 1 1 1 ✪ (x) 3 3 3 3 6 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  12. Introduction Motivating Example Allocation C 1 1 1 � ≥ 1? 3-Subset 0 0 2 2 2 1 1 1 ✧ (i) 0 0 2 2 2 1 1 1 ✧ (ii) 0 0 2 2 2 1 1 1 ✧ (iii) 0 0 2 2 2 1 1 1 ✧ (iv) 0 0 2 2 2 1 1 1 ✧ (v) 0 0 2 2 2 1 1 1 ✪ (vi) 0 0 2 2 2 1 1 1 ✧ (vii) 0 0 2 2 2 1 1 1 ✧ (viii) 0 0 2 2 2 1 1 1 ✪ (ix) 0 0 2 2 2 1 1 1 ✪ (x) 0 0 2 2 2 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  13. Introduction Motivating Example Allocation C 1 1 1 � ≥ 1? 3-Subset 0 0 2 2 2 1 1 1 ✧ (i) 0 0 2 2 2 1 1 1 ✧ (ii) 0 0 2 2 2 1 1 1 ✧ (iii) 0 0 2 2 2 70% 1 1 1 ✧ (iv) 0 0 2 2 2 1 1 1 ✧ (v) 0 0 Survival 2 2 2 1 1 1 ✪ (vi) 0 0 Probability 2 2 2 1 1 1 ✧ (vii) 0 0 2 2 2 1 1 1 ✧ (viii) 0 0 2 2 2 1 1 1 ✪ (ix) 0 0 2 2 2 1 1 1 ✪ (x) 0 0 2 2 2 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  14. Introduction Motivating Example Budget of 1.5 ℓ Survival Allocation Probability 1 A 1 0 0 0 60% 2 1 1 1 1 1 B 40% 3 3 3 3 6 1 1 1 C 70% 0 0 2 2 2 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  15. Introduction Motivating Example Water ≈ Coded Data Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  16. Introduction Motivating Example Water ≈ Coded Data Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  17. Introduction Motivating Example Water ≈ Coded Data storage node 1 x 1 t 1 storage node 2 x 2 r 1 s ... ... storage node n t 2 x n r 2 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  18. Introduction Key Question storage node 1 x 1 Given a limited storage budget, how should we storage node 2 store a data object over a set of nodes so that it x 2 s t can be recovered with maximum reliability? r ... ... storage node n x n Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  19. Introduction Key Question storage node 1 x 1 Given a limited storage budget, how should we storage node 2 store a data object over a set of nodes so that it x 2 s t can be recovered with maximum reliability? r ... ... storage node n x n Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  20. Introduction Key Question storage node 1 x 1 Given a limited storage budget, how should we storage node 2 store a data object over a set of nodes so that it x 2 s t can be recovered with maximum reliability? r ... ... storage node n x n Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  21. Introduction Key Question storage node 1 x 1 Given a limited storage budget, how should we storage node 2 store a data object over a set of nodes so that it x 2 s t can be recovered with maximum reliability? r ... ... storage node n x n Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  22. General Problem Description Storage Allocation A source has a data object of unit size which it can code and store over a set of n storage nodes Let  1 ,  2 , . . . ,  n be the amount of coded storage node 1 x 1 data stored in node 1 , 2 , . . . , n storage node 2 x 2 Although any amount of data can be stored s t r ... ... in each node, the total amount of storage storage node n x n used must not exceed a given budget T , i.e. n �   ≤ T  = 1 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  23. General Problem Description Storage Allocation A source has a data object of unit size which it can code and store over a set of n storage nodes Let  1 ,  2 , . . . ,  n be the amount of coded storage node 1 x 1 data stored in node 1 , 2 , . . . , n storage node 2 x 2 Although any amount of data can be stored s t r ... ... in each node, the total amount of storage storage node n x n used must not exceed a given budget T , i.e. n �   ≤ T  = 1 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  24. General Problem Description Storage Allocation A source has a data object of unit size which it can code and store over a set of n storage nodes Let  1 ,  2 , . . . ,  n be the amount of coded storage node 1 x 1 data stored in node 1 , 2 , . . . , n storage node 2 x 2 Although any amount of data can be stored s t r ... ... in each node, the total amount of storage storage node n x n used must not exceed a given budget T , i.e. n �   ≤ T  = 1 Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  25. General Problem Description Access by a Data Collector A data collector subsequently attempts to recover the original data object by accessing only a random subset r of the nodes, where r is to be specified by the storage node 1 x 1 assumed access model or failure model storage node 2 By using an appropriate code, successful x 2 s t r recovery occurs when the total amount of ... ... storage node n data in the accessed nodes is at least the x n size of the original data object, i.e. �   ≥ 1  ∈ r Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

  26. General Problem Description Access by a Data Collector A data collector subsequently attempts to recover the original data object by accessing only a random subset r of the nodes, where r is to be specified by the storage node 1 x 1 assumed access model or failure model storage node 2 By using an appropriate code, successful x 2 s t r recovery occurs when the total amount of ... ... storage node n data in the accessed nodes is at least the x n size of the original data object, i.e. �   ≥ 1  ∈ r Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

ZEPPELIN High Available KV Storage Service OVERVIEW CAP High Available Distributed KV

ZEPPELIN High Available KV Storage Service OVERVIEW CAP High Available Distributed KV

ZEPPELIN High Available KV Storage Service OVERVIEW CAP High Available Distributed KV storage Online Searching 600,000 QPS 24 Physical Machine 3 meta server 96 node server 40+ tables Highest Table Total query

683 views • 35 slides
Scalable QoS for Distributed Storage Clusters using Dynamic Token Allocation Yuhan Peng 1 ,

Scalable QoS for Distributed Storage Clusters using Dynamic Token Allocation Yuhan Peng 1 ,

Scalable QoS for Distributed Storage Clusters using Dynamic Token Allocation Yuhan Peng 1 , Qingyue Liu 2 , Peter Varman 2 Department of Computer Science 1 Department of Electrical and Computer Engineering 2 Rice University 35th International

761 views • 44 slides
Lecture 9 MIT OpenCourseWare Dynamic Storage Allocation Stack allocation: LIFO (last-in-first-out)

Lecture 9 MIT OpenCourseWare Dynamic Storage Allocation Stack allocation: LIFO (last-in-first-out)

Spring 2009 6.172 Performance Engineering of Software Systems Lecture 9 MIT OpenCourseWare Dynamic Storage Allocation Stack allocation: LIFO (last-in-first-out) Array and pointer A unused used P after P before Figure 1: A

280 views • 8 slides
Impacts from Distributed Energy Resources 2019 ESS Safety & Reliability Forum Albuquerque,

Impacts from Distributed Energy Resources 2019 ESS Safety & Reliability Forum Albuquerque,

Energy Storage Systems for Addressing Bulk Power System Impacts from Distributed Energy Resources 2019 ESS Safety & Reliability Forum Albuquerque, NM March 7, 2019 Charlie Vartanian, P.E. Pacific Northwest National Laboratory Expected

219 views • 18 slides
Pufferbench: Evaluating and Optimizing Malleability of Distributed Storage Nathanal Cheriere,

Pufferbench: Evaluating and Optimizing Malleability of Distributed Storage Nathanal Cheriere,

Pufferbench: Evaluating and Optimizing Malleability of Distributed Storage Nathanal Cheriere, Matthieu Dorier, Gabriel Antoniu PDSW-DISCS 2018, Dallas Data is everywhere High variety of applications High variety of needs Resource

832 views • 25 slides
Automatic Memory Management Storage Allocation Static Allocation Bind names at compile

Automatic Memory Management Storage Allocation Static Allocation Bind names at compile

Automatic Memory Management Storage Allocation Static Allocation Bind names at compile time Pros: Fast (no run time allocation, no indirection) Safety : memory requirements known in advance Cons: Sizes must be

417 views • 14 slides
Storage and reliability Computer Architecture J. Daniel Garca Snchez (coordinator) David

Storage and reliability Computer Architecture J. Daniel Garca Snchez (coordinator) David

Storage and reliability Storage and reliability Computer Architecture J. Daniel Garca Snchez (coordinator) David Expsito Singh Francisco Javier Garca Blas ARCOS Group Computer Science and Engineering Department University Carlos III

808 views • 41 slides
Locality and Availability ! in Distributed Storage Dimitris Papailiopoulos Dimacs Workshop on

Locality and Availability ! in Distributed Storage Dimitris Papailiopoulos Dimacs Workshop on

! Locality and Availability ! in Distributed Storage Dimitris Papailiopoulos Dimacs Workshop on Algorithms for Green Data Storage joint work with Ankit Rawat Alex Dimakis Sriram Vishwanath Coding for Distributed Storage Current state

561 views • 22 slides
Efficiency and Fairness in Distributed Resource Allocation Ulle Endriss Institute for Logic,

Efficiency and Fairness in Distributed Resource Allocation Ulle Endriss Institute for Logic,

Distributed Resource Allocation TU Delft Agent Colloquium: Nov 2006 Efficiency and Fairness in Distributed Resource Allocation Ulle Endriss Institute for Logic, Language and Computation University of Amsterdam joint work with

405 views • 23 slides
Algorithms and Methods for Distributed Storage Networks 8 Storage Virtualization and DHT

Algorithms and Methods for Distributed Storage Networks 8 Storage Virtualization and DHT

Algorithms and Methods for Distributed Storage Networks 8 Storage Virtualization and DHT Christian Schindelhauer Albert-Ludwigs-Universitt Freiburg Institut fr Informatik Rechnernetze und Telematik Wintersemester 2007/08 Overview

634 views • 34 slides
Balancing Fairness and Efficiency in Tiered Storage Systems with Bottleneck-Aware Allocation Hui

Balancing Fairness and Efficiency in Tiered Storage Systems with Bottleneck-Aware Allocation Hui

Balancing Fairness and Efficiency in Tiered Storage Systems with Bottleneck-Aware Allocation Hui Wang, Peter Varman Rice University FAST14, Feb 2014 Tiered Storage v Tiered storage: HDs and SSDs q Advantages: } Performance }

832 views • 36 slides
Distributed Storage Systems part 1 Marko Vukoli Distributed Systems and Cloud Computing This

Distributed Storage Systems part 1 Marko Vukoli Distributed Systems and Cloud Computing This

Distributed Storage Systems part 1 Marko Vukoli Distributed Systems and Cloud Computing This part of the course (5 slots) Distributed Storage Systems CAP theorem and Amazon Dynamo Apache Cassandra Distributed Systems

767 views • 52 slides
Spark RDD 1 Where are we? Distributed storage in HDFS MapReduce query execution in Hadoop

Spark RDD 1 Where are we? Distributed storage in HDFS MapReduce query execution in Hadoop

Spark RDD 1 Where are we? Distributed storage in HDFS MapReduce query execution in Hadoop High-level data manipulation using Pig 2 A Step Back to MapReduce Designed in the early 2000s Machines were unreliable Focused on fault-tolerance

562 views • 28 slides
Outline Distributed storage system 1 Regenerating code 2 Our works 3 2 The age of big data

Outline Distributed storage system 1 Regenerating code 2 Our works 3 2 The age of big data

MMC, Sep 7,2017 MDS Codes for Distributed Storage System Xi Xiaohu Tang Speaker xh xhutang@sw swjtu.edu.cn cn Email Joint works with J. Li, P. Udaya, and C. Tian Outline Distributed storage system 1 Regenerating code 2 Our works 3 2

844 views • 52 slides
Reliability: Serial interconnections: Thus: Reliability of a generic component i :

Reliability: Serial interconnections: Thus: Reliability of a generic component i :

Evaluate the reliability and the availability of a system composed of three CPUs (working in parallel and whose output is given by a voter), a RAID 1 storage system with 4+4 disks three buses (working in parallel and whose outputs are given by

190 views • 3 slides
SAFETY CULTURES 1 HIGH RELIABILITY AT TEAM LEVEL High reliability The unusual capacity to

SAFETY CULTURES 1 HIGH RELIABILITY AT TEAM LEVEL High reliability The unusual capacity to

Improving joy and meaning in work COLLECTIVE LEADERSHIP AND (COLLECTIVE LEADERSHIP FOR TEAM PERFORMANCE) SAFETY CULTURES 1 HIGH RELIABILITY AT TEAM LEVEL High reliability The unusual capacity to produce collective outcomes of a certain

282 views • 12 slides
1 Homology: similarity among two or more individuals or lineages in a feature/character, or

1 Homology: similarity among two or more individuals or lineages in a feature/character, or

Phylogenetics 2: Phylogenetic and genealogical homology Phylogenies distinguish homology from similarity Analogy Homology Polarity Ancestral character Derived character 1 Homology: similarity among two or more individuals or

451 views • 18 slides
Extended BPF A New Type of Software Brendan Gregg UbuntuMasters Oct 2019 BPF 50 Years, one

Extended BPF A New Type of Software Brendan Gregg UbuntuMasters Oct 2019 BPF 50 Years, one

Extended BPF A New Type of Software Brendan Gregg UbuntuMasters Oct 2019 BPF 50 Years, one (dominant) OS model Applications System Calls Kernel Hardware Origins: Multics, 1960s Applications Supervisor Hardware Privilege Ring 0 Ring

979 views • 48 slides
Welcome! The Webinar will Begin Shortly Technical Assistance FAQs 1. Why cant I hear anything?

Welcome! The Webinar will Begin Shortly Technical Assistance FAQs 1. Why cant I hear anything?

Welcome! The Webinar will Begin Shortly Technical Assistance FAQs 1. Why cant I hear anything? Choose your audio connection by clicking on the phone icon at the bottom of the screen. There is NO hold music , so you will not hear anything

702 views • 53 slides
Introduction to Childrens System of Care Rebekah Behan, BS Tim Marshall, LCSW Project

Introduction to Childrens System of Care Rebekah Behan, BS Tim Marshall, LCSW Project

Introduction to Childrens System of Care Rebekah Behan, BS Tim Marshall, LCSW Project Coordinator, Child Health and Director of the Office of Community Development Institute of Connecticut, Mental Health, Connecticut Department of Children

587 views • 20 slides
BPF in-kernel virtual machine 1 BPF is Berkeley Packet Filter low level

BPF in-kernel virtual machine 1 BPF is Berkeley Packet Filter low level

BPF in-kernel virtual machine 1 BPF is Berkeley Packet Filter low level instruction set kernel infrastructure around it interpreter JITs maps helper functions Agenda status and new use cases architecture

680 views • 41 slides
Public WH Design (Problem 24) K $/yr = ( ) b AC $/slot-yr M slot Demand assumed

Public WH Design (Problem 24) K $/yr = ( ) b AC $/slot-yr M slot Demand assumed

Public WH Design (Problem 24) K $/yr = ( ) b AC $/slot-yr M slot Demand assumed uncorrelated since it belongs to different customers N M SS 1 i i = + + M SS i 2

102 views • 6 slides
Analyzing DPDK applications with eBPF Sharpening the toolset Stephen Hemminger Fosdem, February

Analyzing DPDK applications with eBPF Sharpening the toolset Stephen Hemminger Fosdem, February

Analyzing DPDK applications with eBPF Sharpening the toolset Stephen Hemminger Fosdem, February 1, 2020 Microsoft 1 Table of Contents Introduction Packet Capture Tracing Lttng Bpftrace Performance Conclusion 2 Introduction Ancient

875 views • 60 slides
DMSP Nighttime Lights of China: D i Driven by Growth b G th Christopher D. Elvidge (NOAA) Ch i

DMSP Nighttime Lights of China: D i Driven by Growth b G th Christopher D. Elvidge (NOAA) Ch i

DMSP Nighttime Lights of China: D i Driven by Growth b G th Christopher D. Elvidge (NOAA) Ch i h D El id (NOAA) Kimberly Baugh (CIRES), Daniel Ziskin (CIRES), Sharolyn J. Anderson (CIRES) h l d ( ) and Tilottama Ghosh (CIRES) February

280 views • 12 slides

More recommend