On Random Sampling, its Applications, and Efficient Distributed - - PowerPoint PPT Presentation

On Random Sampling, its Applications, and Efficient Distributed Algorithms

Asad Awan, Ronaldo Ferreira, Ramanathan Muralikrishna, Suresh Jagannathan, and Ananth Grama. Parallel and Distributed Systems Lab Department of Computer Sciences Purdue University

Acknowledgements: National Science Foundation

Overview of P2P Networks

• Structured:

– Napster: Star topology. – Chord, Pastry: O(log n)-regular graphs. – CAN: Mesh structure.

• Unstructured:

– Gnutella, Freenet, Morpheus, ...

• New Generation:

– BitTorrent.

Research Problems in P2P Networks

• File sharing is the predominant application of p2p networks.
• Structured:

– Guaranteed resource lookup using hashes, lacks keyword search.

• Unstructured:

– Search ∗ Flooding used by real-world software – high overhead. ∗ Recent techniques rely on replication [Cohen et. al, SIGCOMM

’02], random walks [Gkantisidis et. al, INFOCOM ’04].

∗ Trend towards distributed randomized algorithms. – Distributed Randomized Algorithms: Applications ∗ (1) Search, (2) duplicate elimination & controlled replication, (3) leader election, (4) routing, (5) group communication. ∗ Underlying substrate: uniform sampling in Real-World distributed settings.

Duplicate Elimination

• Issues:

– Reliability - At least one copy of the data needs to be present. – Scalability to large unstructured networks.

• Techniques:

– Delta encoding, duplicate block suppression, Rabin fingerprints for chunking, REBL [USENIX ’04] for local sites. – Self Arranging Lossy Associative Database (SALAD) - [ICDCS

’02], aggregate file content and local information.

– Probabilistic approaches: ∗ Elimination by estimation. ∗ Elimination by election.

Elimination by Estimation

• Two step approach:

– Estimate the number of copies. – Based

• n

the above estimate and the required replication factor of the system, each site deletes the copy probabilistically.

• Estimation:

– Abstracted using a balls-and-bins model. – Each peer with a duplicate ∗ selects γ√n peers u.a.r and send an estimate message. ∗ selects another γ√n peers u.a.r and request the number of unique estimate message received by each peer. ∗ estimates the number of copies as

1 γ2 of the total

received in the previous step. – If K is the number of copies, the standard deviation of the above estimate is

√ K γ .

Elimination by Estimation - continued

• Replication factor - ρ.
• Current number of copies - K (K >> ρ).
• Each site keeps the copy with probability ρ

K.

• Message complexity - O(n√n).
• Issues:

– the number of sites should be known. – methods for uniform sampling.

Elimination by Election

• For each file, elect a leader.
• Leader is responsible for the file.
• Traditional Leader Election:

– Unique leader is a requirement. – Identity of the leader is known by all the participants. – Message complexity - O(n log n).

• Randomized Leader Election:

– Uniqueness is desired but absolutely required. – Participants need not know the identity of the elected leader. – Message complexity - O(n).

Definitions and Features

• Contender: Participating site in the protocol that holds a

copy of the file.

• Mediator: Site that receives a message from a contender

and arbitrates whether the contender participates in subsequent steps of the protocol.

• Round:

Composed of communication between a contender and a set of mediators.

• Features:

– A unique leader elected w.h.p (1 −

1 nΩ(1)).

– Message Complexity - O(n). – Round Complexity - O(log n).

Randomized Leader Election

• Performed in two phases.
• First Phase:

– Goal: Reduce the number of contenders to a desired level. – Each contender C sends a message to √ 2i ln 2 mediators selected u.a.r in round i. – Each mediator M that received a message from C sends back a response indicating that C can proceed to round i + 1 iff it did not receive message from another contender. – C proceeds to the round i + 1 if it receives positive responses from all the mediators to which it sent a

• message. Otherwise C deletes the local copy of the file.

– After each round, the expected number of mediators reduces by half.

An Example

6

3 2 5 6 7 8 Round 3 (log n): Each 6 contender casts 5 balls.

6

5 4

5

First Phase 7 8 1 Contenders Mediators Leader Second Phase 6 5 5 6

5

1 2 6 3

4

5

6 2

6 7

8 7

8 6 Round 1: Each contender

5

2 3

1

3

4

4 5 casts one ball. 4

6

5 4 2 5 1 1 2 3 4 5 6 7 8 Round 2: Each contender casts two balls 2

Randomized Leader Election - continued

• Second Phase:

– Goal: Elect a unique leader. – Based on the Probabilistic Quorum Systems of Malkhi et. al (PODC ’97). – Each contender sends a message with a random number to √ n ln n mediators selected u.a.r. – A mediator responds positively to the contender with the largest random number. – A contender is the leader iff it receives positive responses from all its mediators.

Distributed Uniform Sampling in Real-World Networks

• Random walk of a prescribed minimum length gives

random sampling – independent of origin of walk.

• Random walks are conveniently analyzed using Markov

model: – Represent the network as a graph G(V, E), and define transition probability matrix P, where pij = 1/di, (i, j) ∈ E.

∗ Walks are memoryless: P r(Xt = j|Xt−1 = i, Xt−2 = i1, . . .) = pij. ∗ G is connected & aperiodic ⇒ MC is irreducible and aperiodic.

– A long walk reaches stationary distribution π, which is irrespective of origin and path.

→ This gives random sampling with distribution πi = di/2|E|. → Is this a uniformly random sample?

Not if the network nodes have non-uniform degree distribution!

Random Walks in Real-World Networks

• Real-World networks have non-uniform degree distribution.

e.g., Power-law topology of Gnutella.

• Stationary sampling distribution for random walks on a

50,000 node power-law random graph:

1e-05 2e-05 3e-05 4e-05 5e-05 6e-05 7e-05 8e-05 9e-05 10000 20000 30000 40000 50000 Probability of Selection Node (Sorted by degree) Simple RW, Length=30logn

Node selection probability has almost an order of magnitude variation!

Uniform Sampling via Random Walks

• To get uniform sampling we need to change P, over which

the random walk is defined.

What class of transition matrices will yield random walks with πi = 1/n?

• Recall that stationary distribution satisfies πT = πTP.

πuniform = (1/n)1T = (1/n)1TP . ⇒ The sum of entries in each column of P is 1. Thus, P should be doubly stochastic.

• Observation:

Symmetric transition probability matrix are doubly stochastic.

Row-stochastic because sum of probabilities = 1, and column-stochastic by virtue of symmetry.

• A key issue remains:

What should be the length of walk to reach stationarity?

Length of Random Walks

• Eigen-structure of P: 1 = λ1 > |λ2| ≥ |λ3| ≥ . . . ≥ |λn|
• Convergence to stationarity:

P Tπ = P π = π Thus, π is an eigenvector with eigenvalue 1, i.e., largest eigenvalue. ∴ P ∞ = 1πT (infinite step walk)

• Perron-Frobenius theorem:

P t = P ∞ + O(|λ2|ttm2−1) |λ2| < 1 ⇒ |λ2|t ≈ 0, for a shorter length walk if |λ2| is small

• Mixing time or length of walk = O(log n/(1 − SLEM)) for
• expanders. (Lovasz ’96).

The required length of the walk to reach stationary distribution is small, if SLEM is small.

Enabling Efficient Uniform Sampling

Aim: Design distributed algorithms to locally compute transitions between neighboring nodes; resulting in a global transition matrix with stationary distribution πuniform.

• Known algorithms:

– Maximum-Degree Algorithm: P md is symmetric ⇒ πuniform pmd

ij

= 8 < : 1/dmax if i = j and j ∈ Γ(i) 1 − di/dmax if i = j

• therwise.

– Metropolis-Hastings Algorithm: P mh is symmetric ⇒ πuniform

(Adapted for uniform stationary distribution.)

pmh

ij

= 8 > < > : 1/max(di, dj) if i = j and j ∈ Γ(i) 1 − P

j∈Γ(i)(pmh ij )

if i = j

• therwise.
• Issues:

– MD requires knowledge of a global dynamic variable (dmax) – Both algorithms have a high self-transition probability, which intuitively implies longer mixing times.

Enabling Efficient Uniform Sampling

• Random Weight Distribution (RWD)

– Initialization: Assign a small constant transition probability, 1/ρ (system parameter ρ ≥ dmax) to each edge. This leaves a high self- transition probability (called weight). – Iteration: Each node i, randomly distributes its weight to neighbors, by incrementing transition probability with them symmetrically – using ACKs and NACKs. – Termination: For node i, either pii = 0 or pjj = 0, where j is a neighbor of i. – Each increment is done by a quantum value (system parameter). – ρ is a static system parameter – overestimate of dmax. It is often easy to estimate, e.g., supernodes have a max connection limit in P2P

• networks. As we shall demonstrate, a high overestimate is adequate.

– If quantum (δ) is small, number of increment messages is higher (msgs < (1 − (di/ρ))/δ + di < 1/δ + di). These msgs can be piggybacked on routine communication between neighbors (ping- pong, queries, etc.).

Random Weight Distribution

• Algorithm is decentralized.
• Resulting P rwd is symmetric and allows uniform sampling.
• Low self-transition probabilities imply fast mixing, i.e., shorter

walks.

• We

use extensive simulation results using large non- uniform network topologies to quantify the performance advantage of RWD.

Performance Evaluation

Simulation setup:

• Power-law random topology with α = 0.8 (Gnutella P2P).
• Power-law distribution: If nodes are sorted by degree, then

the ith node has degree = D/iα, where D is a constant.

• Constructing a power-law random topology: first degrees

are assigned to nodes, using power-law distribution, and then they are randomly connected.

• Motivated by real-world systems max degree capped at

100.

Convergence to Stationarity and SLEM

• We find SLEM and walk length, t, which satisfies:

|P t − (1/n)11T| ≤ ε.

• Since we explicitly compute SLEM we limit the topology size

to n = 5000.

• P rwd, P mh, and P md

are compared. Evaluation using different values for quantum size and ρ, for RWD are presented.

Convergence to Stationarity and SLEM - continued

• Varying quantum size between 0.001 and 0.1, (ρ = 200):

SLEM Length

0.8 0.85 0.9 0.95 1 1.05 1.1 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1 SLEM Quantum Size Quantum Size vs SLEM (alpha = 0.8, n = 5000, rho = 200) RWD MH MD 100 200 300 400 500 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1 Length of RW Quantum Size Quantum Size vs Length (alpha = 0.8, n = 5000, rho = 200) RWD MH MD

• Large value of quantum leaves less freedom for weight distribution and

performs worse.

• MH takes almost twice the number of steps taken by RWD to converge.

MD is even worse. (1 − SLEM) of RWD is almost 1.5 times that of MH.

• Choosing a quantum value of 0.025 requires at most 30 more messages

per node as compared to choosing 0.1, while performance benefit in terms of length of walk is more than two fold.

Convergence to Stationarity and SLEM - continued

• Varying ρ between 150 and 300, (quantum = 0.025):

SLEM Length

0.8 0.85 0.9 0.95 1 1.05 1.1 140 160 180 200 220 240 260 280 300 SLEM rho rho vs SLEM (alpha = 0.8, n = 5000, quantum = 0.025) RWD MH MD 100 200 300 400 500 140 160 180 200 220 240 260 280 300 Length of RW rho rho vs Length (alpha = 0.8, n = 5000, quantum = 0.025) RWD MH MD

• Overestimation of dmax by more than a factor of 1.5 results in consistent

good performance of RWD.

Characterizing Uniformity vs Length of Walk

• Topology of size 50, 000 is used.
• Random walks of different lengths are executed.

Uniformity is characterized by variance from mean sampling probability of a node i.e., 1/n.

• Lengths used: 3 log n, 5 log n, 7 log n, 10 log n, 15 log n, and 20 log n.
• Variance from mean vs.

length of walk for P rwd, P mh, and P md are shown in the plot.

• Variance

in samples from C function drand48() is shown for comparison.

1e-06 1e-05 0.0001 5 10 15 20 Standard Deviation Length of RW uniform RWD MH MD

• Length of walk required by RWD to achieve sufficient uniformity is half

that of MH and quarter that of MD.

Characterizing Uniformity vs Length of Walk - continued

• Node sampling probability at 3 log n for RWD, MH, and MD.

1e-05 2e-05 3e-05 4e-05 5e-05 6e-05 7e-05 8e-05 10000 20000 30000 40000 50000 Probability of Selection Node RWD, Length=3logn 0.0005 0.001 0.0015 0.002 0.0025 0.003 10000 20000 30000 40000 50000 Probability of Selection Node MH, Length=3logn 0.001 0.002 0.003 0.004 0.005 0.006 0.007 0.008 0.009 10000 20000 30000 40000 50000 Probability of Selection Node MD, Length=3logn

• MH and MD get stuck in self-transitions for low degree nodes, with

variations up to 3 orders of magnitude.

Characterizing Uniformity vs Length of Walk - continued

• Node sampling probability at 5 log n for RWD, MH, and MD.

5e-06 1e-05 1.5e-05 2e-05 2.5e-05 3e-05 3.5e-05 10000 20000 30000 40000 50000 Probability of Selection Node RWD, Length=5logn 0.0001 0.0002 0.0003 0.0004 0.0005 0.0006 10000 20000 30000 40000 50000 Probability of Selection Node MH, Length=5logn 0.0005 0.001 0.0015 0.002 0.0025 0.003 0.0035 0.004 0.0045 10000 20000 30000 40000 50000 Probability of Selection Node MD, Length=5logn

• RWD reaches non-biased uniform sampling with only 5 log n = 54 steps.

Network Messages

• Length of walks includes self-transitions and network messages.
• To evaluate network message overhead,

we use the length of walk for each of RWD, MH, and MD, which corresponds to the length required to get uniform sample for that algorithm. (i.e., 5 log n, 10 log n, and 20 log n respectively). Algorithm Network Msgs Percentage of total walk RWD 38 70.4 MH 51 46.8 MD 57 21.6

• Although, most of the walk using RWD comprises of network messages,

it still out performs MH and MD significantly.

Concluding Remarks and Ongoing Work

• Random walk with uniform sampling is one of the most ubiquitous

substrates of distributed probabilistic algorithms.

• We presented here an algorithm, Random Weight Distribution, which

yields excellent convergence (short walks), has low overhead (that can be piggybacked on to data messages), and is highly scalable.

• Current work in our group is focusing on a precise quantification of the

length of the walk and its application to a variety of problems in routing and resource management.