Distributed Storage Allocation for High Reliability Derek Leong 1 - - PowerPoint PPT Presentation
Distributed Storage Allocation for High Reliability Derek Leong 1 Alex Dimakis 2 Tracey Ho 1 1 Department of Electrical Engineering California Institute of Technology Pasadena, California, USA 2 Department of Electrical Engineering University of
Distributed Storage Allocation for High Reliability
Derek Leong1 Alex Dimakis2 Tracey Ho1
1Department of Electrical Engineering
California Institute of Technology Pasadena, California, USA
2Department of Electrical Engineering
University of Southern California Los Angeles, California, USA
ICC 2010 2010-05-26
A Water Analogy
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
Introduction
Motivating Example
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
A thirsty lion wanders across the savanna in search of water ...
Introduction
Motivating Example
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
... he needs at least 1ℓ of water to survive ...
Introduction
Motivating Example
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
... suddenly, he finds five abandoned jerrycans ...
Introduction
Motivating Example
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
... he chooses three jerrycans at random and starts chewing them open ...
Introduction
Motivating Example
Budget of 1.5ℓ Allocation Survival Probability A 1
1 2
? B
1 3 1 3 1 3 1 3 1 6
? C
1 2 1 2 1 2
?
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
Introduction
Motivating Example
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
Allocation A
3-Subset 1
1 2
(i) 1
1 2
✧ (ii) 1
1 2
✧ (iii) 1
1 2
✧ (iv) 1
1 2
✧ (v) 1
1 2
✧ (vi) 1
1 2
✧ (vii) 1
1 2
✪ (viii) 1
1 2
✪ (ix) 1
1 2
✪ (x) 1
1 2
✪
Introduction
Motivating Example
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
Allocation A
3-Subset 1
1 2
(i) 1
1 2
✧ (ii) 1
1 2
✧ (iii) 1
1 2
✧ (iv) 1
1 2
✧ (v) 1
1 2
✧ (vi) 1
1 2
✧ (vii) 1
1 2
✪ (viii) 1
1 2
✪ (ix) 1
1 2
✪ (x) 1
1 2
✪
60% Survival Probability Introduction
Motivating Example
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
Allocation B
3-Subset
1 3 1 3 1 3 1 3 1 6
(i)
1 3 1 3 1 3 1 3 1 6
✧ (ii)
1 3 1 3 1 3 1 3 1 6
✧ (iii)
1 3 1 3 1 3 1 3 1 6
✪ (iv)
1 3 1 3 1 3 1 3 1 6
✧ (v)
1 3 1 3 1 3 1 3 1 6
✪ (vi)
1 3 1 3 1 3 1 3 1 6
✪ (vii)
1 3 1 3 1 3 1 3 1 6
✧ (viii)
1 3 1 3 1 3 1 3 1 6
✪ (ix)
1 3 1 3 1 3 1 3 1 6
✪ (x)
1 3 1 3 1 3 1 3 1 6
✪
Introduction
Motivating Example
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
Allocation B
3-Subset
1 3 1 3 1 3 1 3 1 6
(i)
1 3 1 3 1 3 1 3 1 6
✧ (ii)
1 3 1 3 1 3 1 3 1 6
✧ (iii)
1 3 1 3 1 3 1 3 1 6
✪ (iv)
1 3 1 3 1 3 1 3 1 6
✧ (v)
1 3 1 3 1 3 1 3 1 6
✪ (vi)
1 3 1 3 1 3 1 3 1 6
✪ (vii)
1 3 1 3 1 3 1 3 1 6
✧ (viii)
1 3 1 3 1 3 1 3 1 6
✪ (ix)
1 3 1 3 1 3 1 3 1 6
✪ (x)
1 3 1 3 1 3 1 3 1 6
✪
40% Survival Probability Introduction
Motivating Example
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
Allocation C
3-Subset
1 2 1 2 1 2
(i)
1 2 1 2 1 2
✧ (ii)
1 2 1 2 1 2
✧ (iii)
1 2 1 2 1 2
✧ (iv)
1 2 1 2 1 2
✧ (v)
1 2 1 2 1 2
✧ (vi)
1 2 1 2 1 2
✪ (vii)
1 2 1 2 1 2
✧ (viii)
1 2 1 2 1 2
✧ (ix)
1 2 1 2 1 2
✪ (x)
1 2 1 2 1 2
✪
Introduction
Motivating Example
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
Allocation C
3-Subset
1 2 1 2 1 2
(i)
1 2 1 2 1 2
✧ (ii)
1 2 1 2 1 2
✧ (iii)
1 2 1 2 1 2
✧ (iv)
1 2 1 2 1 2
✧ (v)
1 2 1 2 1 2
✧ (vi)
1 2 1 2 1 2
✪ (vii)
1 2 1 2 1 2
✧ (viii)
1 2 1 2 1 2
✧ (ix)
1 2 1 2 1 2
✪ (x)
1 2 1 2 1 2
✪
70% Survival Probability Introduction
Motivating Example
Budget of 1.5ℓ Allocation Survival Probability A 1
1 2
60% B
1 3 1 3 1 3 1 3 1 6
40% C
1 2 1 2 1 2
70%
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
Introduction
Motivating Example
Water ≈ Coded Data
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
Introduction
Motivating Example
Water ≈ Coded Data
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
Introduction
Motivating Example
Water ≈ Coded Data
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
storage node 1 storage node 2 storage node n
Introduction
Motivating Example
Given a limited storage budget, how should we store a data object over a set of nodes so that it can be recovered with maximum reliability?
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t r ...
storage node 1
x2
storage node 2
xn
storage node n
...
Introduction
Key Question
Given a limited storage budget, how should we store a data object over a set of nodes so that it can be recovered with maximum reliability?
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t r ...
storage node 1
x2
storage node 2
xn
storage node n
...
Introduction
Key Question
Given a limited storage budget, how should we store a data object over a set of nodes so that it can be recovered with maximum reliability?
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t r ...
storage node 1
x2
storage node 2
xn
storage node n
...
Introduction
Key Question
Given a limited storage budget, how should we store a data object over a set of nodes so that it can be recovered with maximum reliability?
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t r ...
storage node 1
x2
storage node 2
xn
storage node n
...
Introduction
Key Question
A source has a data object of unit size which it can code and store over a set of n storage nodes Let 1, 2, . . . , n be the amount of coded data stored in node 1, 2, . . . , n Although any amount of data can be stored in each node, the total amount of storage used must not exceed a given budget T, i.e.
n
≤ T
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t r ...
storage node 1
x2
storage node 2
xn
storage node n
...
General Problem Description
Storage Allocation
A source has a data object of unit size which it can code and store over a set of n storage nodes Let 1, 2, . . . , n be the amount of coded data stored in node 1, 2, . . . , n Although any amount of data can be stored in each node, the total amount of storage used must not exceed a given budget T, i.e.
n
≤ T
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t r ...
storage node 1
x2
storage node 2
xn
storage node n
...
General Problem Description
Storage Allocation
A source has a data object of unit size which it can code and store over a set of n storage nodes Let 1, 2, . . . , n be the amount of coded data stored in node 1, 2, . . . , n Although any amount of data can be stored in each node, the total amount of storage used must not exceed a given budget T, i.e.
n
≤ T
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t r ...
storage node 1
x2
storage node 2
xn
storage node n
...
General Problem Description
Storage Allocation
A data collector subsequently attempts to recover the original data object by accessing only a random subset r of the nodes, where r is to be specified by the assumed access model or failure model By using an appropriate code, successful recovery occurs when the total amount of data in the accessed nodes is at least the size of the original data object, i.e.
≥ 1
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t r ...
storage node 1
x2
storage node 2
xn
storage node n
...
General Problem Description
Access by a Data Collector
A data collector subsequently attempts to recover the original data object by accessing only a random subset r of the nodes, where r is to be specified by the assumed access model or failure model By using an appropriate code, successful recovery occurs when the total amount of data in the accessed nodes is at least the size of the original data object, i.e.
≥ 1
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t r ...
storage node 1
x2
storage node 2
xn
storage node n
...
General Problem Description
Access by a Data Collector
For a given budget T, we seek the optimal allocation {1, 2, . . . , n} that maximizes the probability of successful recovery
P
≥ 1
This optimization problem is difficult in general because the objective function is discrete and nonconvex, and there is a large space of feasible allocations to consider
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t r ...
storage node 1
x2
storage node 2
xn
storage node n
...
General Problem Description
Objective
For a given budget T, we seek the optimal allocation {1, 2, . . . , n} that maximizes the probability of successful recovery
P
≥ 1
This optimization problem is difficult in general because the objective function is discrete and nonconvex, and there is a large space of feasible allocations to consider
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t r ...
storage node 1
x2
storage node 2
xn
storage node n
...
General Problem Description
Objective
Discussion between R. Karp, R. Kleinberg,
at UC Berkeley, 2005
“Using redundancy to cope with failures in a delay tolerant network,” SIGCOMM 2005 LDH, “Distributed storage allocation problems,” NetCod 2009
distributed storage networks,” ISIT 2010
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t r ...
storage node 1
x2
storage node 2
xn
storage node n
...
General Problem Description
Related Work
Discussion between R. Karp, R. Kleinberg,
at UC Berkeley, 2005
“Using redundancy to cope with failures in a delay tolerant network,” SIGCOMM 2005 LDH, “Distributed storage allocation problems,” NetCod 2009
distributed storage networks,” ISIT 2010
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t r ...
storage node 1
x2
storage node 2
xn
storage node n
...
General Problem Description
Related Work
Discussion between R. Karp, R. Kleinberg,
at UC Berkeley, 2005
“Using redundancy to cope with failures in a delay tolerant network,” SIGCOMM 2005 LDH, “Distributed storage allocation problems,” NetCod 2009
distributed storage networks,” ISIT 2010
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t r ...
storage node 1
x2
storage node 2
xn
storage node n
...
General Problem Description
Related Work
Discussion between R. Karp, R. Kleinberg,
at UC Berkeley, 2005
“Using redundancy to cope with failures in a delay tolerant network,” SIGCOMM 2005 LDH, “Distributed storage allocation problems,” NetCod 2009
distributed storage networks,” ISIT 2010
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t r ...
storage node 1
x2
storage node 2
xn
storage node n
...
General Problem Description
Related Work
Data collector accesses an r-subset of storage nodes, selected uniformly at random from the collection
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
Access to a Fixed-Size Subset of Nodes
Problem Description
Maximize recovery probability for a given budget T
(n, r, T) : maximize
1,...,n
|r|=r
1 n
r
·
≥ 1 subject to
n
≤ T ≥ 0 ∀ ∈ {1, . . . , n}
For the trivial budget T = 1, the optimal allocation is {1, 0, . . . , 0}
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t ...
storage node 1
x2
storage node 2
xn
storage node n
... r
Data collector accesses an r-subset
selected uniformly at random from the collection of all possible r-subsets, where r is a constant
Access to a Fixed-Size Subset of Nodes
Problem Description
Minimize budget required to achieve a given recovery probability PS
′(n, r, PS) : minimize
1,...,n
T subject to
|r|=r
≥ 1 ≥ PS n r
≤ T ≥ 0 ∀ ∈ {1, . . . , n}
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t ...
storage node 1
x2
storage node 2
xn
storage node n
... r
Data collector accesses an r-subset
selected uniformly at random from the collection of all possible r-subsets, where r is a constant
Access to a Fixed-Size Subset of Nodes
Problem Description
(n, r) = (5, 3)
0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 budget T
maximum recovery probability
max Ps
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
Access to a Fixed-Size Subset of Nodes
Some Numerical Results
(n, r) = (5, 3)
0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 budget T
maximum recovery probability
max Ps
{0,0,0,0,0}
1 2 1 2 1 2
{0,0,0,0,0}
1 3 1 3 1 3 1 3 1 3
{1,0,0,0,0}
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
Access to a Fixed-Size Subset of Nodes
Some Numerical Results
(n, r) = (6, 2)
0.5 1.0 1.5 2.0 2.5 3.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 budget T
maximum recovery probability
max Ps
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
Access to a Fixed-Size Subset of Nodes
Some Numerical Results
(n, r) = (6, 2)
0.5 1.0 1.5 2.0 2.5 3.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 budget T
maximum recovery probability
max Ps
{0,0,0,0,0,0}
1 2 1 2 1 2 1 2 1 2
{0,0,0,0,0,0}
1 2 1 2 1 2 1 2 1 2 1 2
{1,1,0,0,0,0} {1,0,0,0,0,0}
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
Access to a Fixed-Size Subset of Nodes
Some Numerical Results
Theorem: Probability-1 Recovery The allocation
= 1 r , = 1, . . . , n,
minimizes the budget if probability-1 recovery is required.
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t ...
storage node 1
x2
storage node 2
xn
storage node n
... r
Data collector accesses an r-subset
selected uniformly at random from the collection of all possible r-subsets, where r is a constant
Access to a Fixed-Size Subset of Nodes
Special Case of Probability-1 Recovery
Theorem: Case of r | n Suppose that n is a multiple of r. The allocation
= 1 r , = 1, . . . , n,
minimizes the budget if and only if the required recovery probability exceeds
1 − r n .
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t ...
storage node 1
x2
storage node 2
xn
storage node n
... r
Data collector accesses an r-subset
selected uniformly at random from the collection of all possible r-subsets, where r is a constant
Access to a Fixed-Size Subset of Nodes
Regime of High Recovery Probability
Theorem: Case of r ∤ n Suppose that n is not a multiple of r. The allocation
= 1 r , = 1, . . . , n,
minimizes the budget if the required recovery probability exceeds
1 − gcd(r, r′) α gcd(r, r′) + r′ ,
where n = α r + r′, α ∈ Z+
0 ,
and r′ ∈ {r + 1, . . . , 2r − 1}.
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t ...
storage node 1
x2
storage node 2
xn
storage node n
... r
Data collector accesses an r-subset
selected uniformly at random from the collection of all possible r-subsets, where r is a constant
Access to a Fixed-Size Subset of Nodes
Regime of High Recovery Probability
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t ...
storage node 1
x2
storage node 2
xn
storage node n
... r
Data collector accesses an r-subset
selected uniformly at random from the collection of all possible r-subsets, where r is a constant
Proof Idea: Consider (n, r) = (4, 2). Begin with the case of probability-1 recovery...
′(n = 4, r = 2, PS = 1) : minimize
1,2,3,4
T subject to 1 + 2 ≥ 1 2 + 3 ≥ 1 1 + 3 ≥ 1 2 + 4 ≥ 1 1 + 4 ≥ 1 3 + 4 ≥ 1 T ≥ 1 + 2 + 3 + 4 1 ≥ 0 2 ≥ 0 3 ≥ 0 4 ≥ 0
Access to a Fixed-Size Subset of Nodes
Regime of High Recovery Probability
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t ...
storage node 1
x2
storage node 2
xn
storage node n
... r
Data collector accesses an r-subset
selected uniformly at random from the collection of all possible r-subsets, where r is a constant
Proof Idea: Consider (n, r) = (4, 2). Optimal allocation is 1 =2 =3 =4 = 1
r = 1 2,
which gives T = n
r = 2.
′(n = 4, r = 2, PS = 1) : minimize
1,2,3,4
T subject to 1 + 2 ≥ 1 2 + 3 ≥ 1 1 + 3 ≥ 1 2 + 4 ≥ 1 1 + 4 ≥ 1 3 + 4 ≥ 1 T ≥ 1 + 2 + 3 + 4 1 ≥ 0 2 ≥ 0 3 ≥ 0 4 ≥ 0
Access to a Fixed-Size Subset of Nodes
Regime of High Recovery Probability
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t ...
storage node 1
x2
storage node 2
xn
storage node n
... r
Data collector accesses an r-subset
selected uniformly at random from the collection of all possible r-subsets, where r is a constant
Proof Idea: Consider (n, r) = (4, 2). This allocation remains optimal even after dropping some r-subset constraints...
′(n = 4, r = 2, PS = 1) : minimize
1,2,3,4
T subject to 1 + 2 ≥ 1 2 + 3 ≥ 1 1 + 3 ≥ 1 2 + 4 ≥ 1 1 + 4 ≥ 1 3 + 4 ≥ 1 T ≥ 1 + 2 + 3 + 4 1 ≥ 0 2 ≥ 0 3 ≥ 0 4 ≥ 0
Access to a Fixed-Size Subset of Nodes
Regime of High Recovery Probability
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t ...
storage node 1
x2
storage node 2
xn
storage node n
... r
Data collector accesses an r-subset
selected uniformly at random from the collection of all possible r-subsets, where r is a constant
Proof Idea: Consider (n, r) = (4, 2). We still need T ≥ 2 = n
r in order to satisfy the
highlighted r-subset constraints...
′(n = 4, r = 2, PS = 1) : minimize
1,2,3,4
T subject to 1 + 2 ≥ 1 2 + 3 ≥ 1 1 + 3 ≥ 1 2 + 4 ≥ 1 1 + 4 ≥ 1 3 + 4 ≥ 1 T ≥ 1 + 2 + 3 + 4 1 ≥ 0 2 ≥ 0 3 ≥ 0 4 ≥ 0
Access to a Fixed-Size Subset of Nodes
Regime of High Recovery Probability
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t ...
storage node 1
x2
storage node 2
xn
storage node n
... r
Data collector accesses an r-subset
selected uniformly at random from the collection of all possible r-subsets, where r is a constant
Proof Idea: Consider (n, r) = (4, 2). We still need T ≥ 2 = n
r in order to satisfy the
highlighted r-subset constraints...
′(n = 4, r = 2, PS = 1) : minimize
1,2,3,4
T subject to 1 + 2 ≥ 1 2 + 3 ≥ 1 1 + 3 ≥ 1 2 + 4 ≥ 1 1 + 4 ≥ 1 3 + 4 ≥ 1 T ≥ 1 + 2 + 3 + 4 1 ≥ 0 2 ≥ 0 3 ≥ 0 4 ≥ 0
Access to a Fixed-Size Subset of Nodes
Regime of High Recovery Probability
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t ...
storage node 1
x2
storage node 2
xn
storage node n
... r
Data collector accesses an r-subset
selected uniformly at random from the collection of all possible r-subsets, where r is a constant
Proof Idea: Consider (n, r) = (4, 2). We still need T ≥ 2 = n
r in order to satisfy the
highlighted r-subset constraints...
′(n = 4, r = 2, PS = 1) : minimize
1,2,3,4
T subject to 1 + 2 ≥ 1 2 + 3 ≥ 1 1 + 3 ≥ 1 2 + 4 ≥ 1 1 + 4 ≥ 1 3 + 4 ≥ 1 T ≥ 1 + 2 + 3 + 4 1 ≥ 0 2 ≥ 0 3 ≥ 0 4 ≥ 0
Access to a Fixed-Size Subset of Nodes
Regime of High Recovery Probability
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t ...
storage node 1
x2
storage node 2
xn
storage node n
... r
Data collector accesses an r-subset
selected uniformly at random from the collection of all possible r-subsets, where r is a constant
Proof Idea: Consider (n, r) = (4, 2). Each r-subset constraint appears in exactly
r-subset Constraints 1 + 2 ≥ 1 1 + 3 ≥ 1 1 + 4 ≥ 1 2 + 3 ≥ 1 2 + 4 ≥ 1 3 + 4 ≥ 1 “Partitions” {1 + 2 ≥ 1, 3 + 4 ≥ 1} {1 + 3 ≥ 1, 2 + 4 ≥ 1} {1 + 4 ≥ 1, 2 + 3 ≥ 1}
Access to a Fixed-Size Subset of Nodes
Regime of High Recovery Probability
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t ...
storage node 1
x2
storage node 2
xn
storage node n
... r
Data collector accesses an r-subset
selected uniformly at random from the collection of all possible r-subsets, where r is a constant
Proof Idea: Consider (n, r) = (4, 2). For each r-subset constraint removed, we deactivate at most one “partition”...
r-subset Constraints 1 + 2 ≥ 1 1 + 3 ≥ 1 1 + 4 ≥ 1 2 + 3 ≥ 1 2 + 4 ≥ 1 3 + 4 ≥ 1 “Partitions” {1 + 2 ≥ 1, 3 + 4 ≥ 1} {1 + 3 ≥ 1, 2 + 4 ≥ 1} {1 + 4 ≥ 1, 2 + 3 ≥ 1}
Access to a Fixed-Size Subset of Nodes
Regime of High Recovery Probability
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t ...
storage node 1
x2
storage node 2
xn
storage node n
... r
Data collector accesses an r-subset
selected uniformly at random from the collection of all possible r-subsets, where r is a constant
Proof Idea: Consider (n, r) = (4, 2). To deactivate all 3 “partitions”, we need to remove at least 3 r-subset constraints...
r-subset Constraints 1 + 2 ≥ 1 1 + 3 ≥ 1 1 + 4 ≥ 1 2 + 3 ≥ 1 2 + 4 ≥ 1 3 + 4 ≥ 1 “Partitions” {1 + 2 ≥ 1, 3 + 4 ≥ 1} {1 + 3 ≥ 1, 2 + 4 ≥ 1} {1 + 4 ≥ 1, 2 + 3 ≥ 1}
Access to a Fixed-Size Subset of Nodes
Regime of High Recovery Probability
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t ...
storage node 1
x2
storage node 2
xn
storage node n
... r
Data collector accesses an r-subset
selected uniformly at random from the collection of all possible r-subsets, where r is a constant
Proof Idea: Consider (n, r) = (4, 2). To deactivate all 3 “partitions”, we need to remove at least 3 r-subset constraints In other words, we will have at least one active “partition” if fewer than 3 r-subset constraints are removed Therefore, if the required recovery probability exceeds 1 − 3
6 = 1 2, then we will
need T ≥ 2 = n
r , that is, the allocation
1 =2 =3 =4 = 1
r = 1 2 is optimal
Access to a Fixed-Size Subset of Nodes
Regime of High Recovery Probability
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t ...
storage node 1
x2
storage node 2
xn
storage node n
... r
Data collector accesses an r-subset
selected uniformly at random from the collection of all possible r-subsets, where r is a constant
Proof Idea: Consider (n, r) = (4, 2). To deactivate all 3 “partitions”, we need to remove at least 3 r-subset constraints In other words, we will have at least one active “partition” if fewer than 3 r-subset constraints are removed Therefore, if the required recovery probability exceeds 1 − 3
6 = 1 2, then we will
need T ≥ 2 = n
r , that is, the allocation
1 =2 =3 =4 = 1
r = 1 2 is optimal
Access to a Fixed-Size Subset of Nodes
Regime of High Recovery Probability
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t ...
storage node 1
x2
storage node 2
xn
storage node n
... r
Data collector accesses an r-subset
selected uniformly at random from the collection of all possible r-subsets, where r is a constant
Proof Idea: Consider (n, r) = (4, 2). To deactivate all 3 “partitions”, we need to remove at least 3 r-subset constraints In other words, we will have at least one active “partition” if fewer than 3 r-subset constraints are removed Therefore, if the required recovery probability exceeds 1 − 3
6 = 1 2, then we will
need T ≥ 2 = n
r , that is, the allocation
1 =2 =3 =4 = 1
r = 1 2 is optimal
Access to a Fixed-Size Subset of Nodes
Regime of High Recovery Probability
Intervals of recovery probability PS over which the allocation = 1
r , = 1, . . . , n, is optimal, for n = 40 5 10 15 20 25 30 35 40 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
subset size r recovery probability
Ps
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
Access to a Fixed-Size Subset of Nodes
Regime of High Recovery Probability
1
We are able to describe the optimal allocation in the regime of high recovery probability
2
Question: For the general problem, how much will we lose if we were to consider
3
Question: What is the optimal symmetric allocation?
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t ...
storage node 1
x2
storage node 2
xn
storage node n
... r
Data collector accesses an r-subset
selected uniformly at random from the collection of all possible r-subsets, where r is a constant
Access to a Fixed-Size Subset of Nodes
Summary
1
We are able to describe the optimal allocation in the regime of high recovery probability
2
Question: For the general problem, how much will we lose if we were to consider
3
Question: What is the optimal symmetric allocation?
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t ...
storage node 1
x2
storage node 2
xn
storage node n
... r
Data collector accesses an r-subset
selected uniformly at random from the collection of all possible r-subsets, where r is a constant
Access to a Fixed-Size Subset of Nodes
Summary
1
We are able to describe the optimal allocation in the regime of high recovery probability
2
Question: For the general problem, how much will we lose if we were to consider
3
Question: What is the optimal symmetric allocation?
Distributed Storage Allocation for High Reliability ICC 2010 2010-05-26
x1 s t ...
storage node 1
x2
storage node 2
xn
storage node n
... r
Data collector accesses an r-subset
selected uniformly at random from the collection of all possible r-subsets, where r is a constant
Access to a Fixed-Size Subset of Nodes
Summary
Thank you!
Table of Contents
1
Introduction Motivating Example Key Question
2
General Problem Description Storage Allocation Access by a Data Collector Objective Related Work
3
Access to a Fixed-Size Subset of Nodes Problem Description Some Numerical Results Special Case of Probability-1 Recovery Regime of High Recovery Probability Summary