Heterogeneity and Load Balance in Distributed Hash Tables Brighten - - PowerPoint PPT Presentation
Heterogeneity and Load Balance in Distributed Hash Tables Brighten Godfrey Joint Work with Ion Stoica Computer Science Division, UC Berkeley IEEE INFOCOM March 15, 2005 The goals Distributed Hash Tables partition an ID space among n nodes
Multiple disjoint segments
High capacity node Low capacity node
n
n (with random
n
n (with random
n
n (with random
n
n (with random
n
n (with random
n
n (with random
n
Simulated Real
n
Simulated Real
Message
n
Simulated Real
Message
n
Simulated Real
Message
Lemma 1 If node v has at least one ID in the ring and α = Θ(log n), then (1) v has between αcv/(γcγu) − O(1) and αcvγcγu + O(1) IDs w.h.p., and (2) v has at least γdα(n) − O(1) IDs w.h.p. Proof: (1) Note that, due to the estimaton error parameters, the factor γc lazy update
cv, and the factor 2 lazy update of ˜ n, we always have ˜ cv within a factor γcγu
n within a factor 2γn of n w.h.p. Thus, for some constant k, the num- ber of IDs that v chooses is at most ⌊0.5 + ˜ cvα(˜ n)⌋ ≤ ˜ cvα(˜ n) + O(1) ≤ γcγucvk log(2γnn) + O(1) ≤ γcγuα(n) + O(1). The lower bound follows similarly, noting that we are not concerned with nodes that have been discarded. (2) Simi- larly, if v has decided to stay in the ring, we must have ˜ cv ≥ γd and the bound follows by the above technique. We now break the ring into frames of length equal to the smallest spacing parameter smin used by any node. The following lemma implies that smin ≥ 1/(2γnn) w.h.p. Lemma 2 Let β = (1 − γcγuγd)/(γcγu). When α ≥
8γn βε2 ln n, each frame
contains at least (1 − ε)βαnsmin − O(1) IDs w.h.p. for any ε > 0. Proof: Assume that no node has more than one ID in any frame; if this is not the case, we can break the high-capacity nodes for which it is false into multiple “virtual nodes” without disturbing the rest of the proof. Consider any particular frame f. Let Xv be the indicator variable for the event that node v chooses an ID in f and let X =
v X. We wish to lower-bound X. Sup-
pose v chooses mv points. Since f covers a fraction smin of the ID space, we have E[Xv] = mvsmin. By Lemma 1, mv ≥ αcv/(γcγu) − O(1) for nodes R in the ring. Thus, E[X] =
E[Xv] ≥
smin (αcv/(γcγu) − O(1)) (Lemma 1) ≥ −O(1) +
sminαcv γcγu = −O(1) + sminα γcγu
cv ≥ −O(1) + sminα γcγu · (1 − γcγuγd)n (Claim ??) = βαnsmin − O(1), with β defined as in the lemma statement. (Note that although Claim ?? was stated in the context of Chord, it applies to our partitioning scheme without modification.) A Chernoff bound tells us that Pr[X < (1 − ε)E[X]] < e−(βαnsmin−O(1))ε2/2 = O(e−βαnsminε2/2) < e−βαε2/(4γn) (Lemma ??) = O(n−2) when α ≥ 8γn
βε2 ln n. Again by Lemma ??, there are at ≤ 2γdn frames, so the lemma
follows from a union bound over them. Proof: (Of Theorem ??) If node v is discarded, its share is 0, so we need only con- sider nodes in the ring. Such a node v chooses one ID in each of m ≤ αcvγcγu +O(1) frames (Lemma 1). We first fix the nodes’ choices of the frames in which they place their IDs. Let X1, . . . , Xm be the fraction of the ID space owned by each of node v’s IDs. The ran- domness in the Xis is over the intra-frame positions of the nodes’ IDs, which are chosen independently and uniformly at random. By Lemma 2, we may assume that each frame has at least one ID. Thus, the interval assigned to the ith ID may span at most one frame bound- ary, so Xi depends only on the locations of the IDs in its frame and in the counterclockwise preceding frame. Thus, the odd-indexed Xis are mutually independent, as are the even- indexed Xis. We will bound the share of these two groups in the same way, one at a time. Consider first the odd-indexed Xis. Break each frame into d buckets of equal size; we’ll pick d later. A bucket is occupied when some node other than v chooses an ID inside it, and is empty otherwise. To analyze the node v’s share of the ID space, we’ll count the number of empty buckets counterclockwise- following v’s chosen IDs. Define an infinite sequence of random variables Yj, each of which will be the indicator variable for the event that a particular bucket is occupied. Y1 will correspond to the bucket counterclockwise-following v’s first odd-indexed ID. Suppose Yj corresponds to the kth bucket following v’s ℓth ID. Then we have two cases. (1) If Yj = 0, Yj+1 corresponds to the next bucket following the same ID. (2) Otherwise, Yj+1 corre- sponds to the first bucket following the next odd-indexed ID, i.e. the (ℓ + 2)th one. If m/2 < ℓ + 2 then we simply set Yj+1 = 1. Thus, the number of zeros in the sequence
With the goal of upper-bounding the number of zeros, we first deal with depen- dence among the Yjs. By Lemma 2 we may assume that each frame has at least r = (1 − ε)βsminnα(n) − O(1) IDs for sufficiently large α. View Y1, Y2, . . . as a
may occupy Yj’s bucket. If we are in Case (1) then Yj’s bucket is in the same frame as that of Yj−1, which implies that some of the buckets in that frame are empty, in which case there are at least r IDs distributed u.a.r. in a subset of the frame including Yj’s bucket. This discussion implies that, regardless of the history of the Yjs, the probability that Yj = 1 is at least 1 − (1 − 1/d)r. Formally, we define another sequence of variables Zj which are independent Poisson trials with success probability p to be picked below. For any indeces j1, . . . , jk, we have Pr[Yj1 = · · · = Yjk = 1] =
k
Pr[Yjℓ = 1|Yj1 = · · · = Yjℓ−1 = 1] ≥
k
1 d
r
≥
= Pr[Zj1 = · · · = Zjk = 1] where we have chosen the success probability for the Zjs to be p = 1 − e−r/d. This implies that an upper bound the number of 0’s in the independent Zj sequence is also an upper bound the number of 0’s in the dependent Yk sequence, a fact which we use next. If we see m/2 ones in the first x Yjs, then by the definition of the sequence, we have seen all the zeros, of which there are at most x − m/2. Thus node v will own at most x − m/2 complete buckets, plus 2 · m/2 partial buckets (one at each end of the m/2 contiguous sequences of complete buckets), for a total of at most x + m/2 buckets due to its m/2 odd-numbered IDs. We now show that we see the required m/2 succeses w.h.p. when x =
m 2p(1−δ) . Let P be the number of 1’s in the first x Yjs,
and let P ′ be the corresponding value for the Zjs. By the above discussion we have Pr[P < m/2] ≤ Pr[P ′ < m/2], and E[P ′] = xp =
m 2(1−δ) , so
Pr[P < m/2] ≤ Pr[P ′ < m/2] = Pr[P ′ < (1 − δ) · m 2(1 − δ) ] ≤ e
− mδ2 4(1−δ)
(Chernoff bound) ≤ O(e
− γdαδ2 4(1−δ) )
(Lemma 1 part (2)) = O(n−2) when α ≥ 8(1−δ)
γdδ2
ln n. In this case, counting now both odd- and even-indexed points, node v owns at most m +
m p(1−δ) buckets, each of size smin/d. Normalizing by v’s
fair share cv/n, we have share(v) ≤ 1 cv/n ·
d + msmin dp(1 − δ)
Recall that d is arbitrary. Taking the limit as d → ∞, we have dp → r = (1 − ε)βsminnα(n) − O(1) so share(v) ≤ 1 cv/n · msmin (1 − δ)((1 − ε)βsminnα(n) − O(1)) ≤ 1 cv · m (1 − δ)(1 − ε)(1 − ε′)βα(n) ≤ 1 cv · α(n)cvγcγu + O(1) (1 − δ)(1 − ε)(1 − ε′)βα(n) (Lemma 1 part (1)) ≤ (1 + ε′′)(γcγu)2 (1 − δ)(1 − ε)(1 − γcγuγd) with probability 1 − O(n−2) for any ε′, ε′′ > 0 and sufficiently large n, so by a union bound, this is true of all nodes w.h.p. Finally, we require that α is the maximum of the requirement given above and that of Lemma 2; setting δ = ε for convenience of presen- tation, we have max{ 8(1−ε) ln n
γdε2
,
8γnγcγu ln n (1−γcγuγd)ε2 } ≤ 8γnγcγu ln n (1−γcγuγd)γdε2 , as
required by the theorem.
1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 3 10 100 1000 10000 100000 Maximum share Number of nodes Chord, alpha = log(n) Y_0, alpha = 2 log n Y_0, alpha = 3 log n
1 2 3 4 5 6 7 8 9 10 32 64 128 256 512 1024 Maximum share Average normalized degree Chord, homogeneous Chord, SGG Y0, homogeneous Y0, SGG
1 2 3 4 5 6 7 1.5 2 2.5 3 3.5 Average route length Degree of power law capacity distribution Chord, alpha = 1 Y_0, alpha = 2 log n
1 2 3 4 5 6 7 8 10 100 1000 10000 Average route length Number of nodes Y_0, homogeneous Chord, homogeneous Chord, SGG Y_0, SGG
10 20 30 40 50 60 70 80 10 100 1000 10000 Maximum congestion Number of nodes Chord, homogeneous Chord, SGG Y_0, homogeneous Y_0, SGG
50 100 150 200 250 300 350 400 450 500 5 10 15 20 25 30 Average normalized degree Number of virtual servers (alpha) Chord Y_0
1 2 3 4 5 6 7 8 1.5 2 2.5 3 3.5 Maximum share Exponent of power law capacity distribution Chord, alpha = log n Y_0, alpha = log n Y_0, alpha = 2 log n Y_0, alpha = 3 log n