and Size of Social Networks via Random Walk Stephen J. Hardiman* - - PowerPoint PPT Presentation
Estimating Clustering Coefficients and Size of Social Networks via Random Walk Stephen J. Hardiman* Liran Katzir Capital Fund Management Advanced Technology Labs France Microsoft Research, Israel *Research was conducted while the author was
Estimating Clustering Coefficients and Size of Social Networks via Random Walk
Stephen J. Hardiman*
Capital Fund Management France
Liran Katzir
Advanced Technology Labs Microsoft Research, Israel
*Research was conducted while the author was unaffiliated
Motivation: Social Networks
Renren LinkedIn Vkontakte
Bebo
Tagged Orkut
Netlog
Friendster hi5
Flixster
MyLife
Classmates.com Sonico.com Plaxo
Motivation: External access
v1 v2 v3 v5 v6 v7 v4 v8 v9 Social Analytics
The online social network Disk Space Communication Privacy
Task: Estimate parameters
Business development/ advertisement/ market size. Predicting Social Products’ Potential. Global Clustering Coefficient Network Average CC Number of Registered Users
Global CC = 3 x number of triangles number of connected triplet
Global Clustering Coefficient
v1 v2 v3 v5 v6 v7 v4 v8 v9
Triangle Connected Triplet
Global Clustering Coefficient
Exact: [Alon et al, 1997] Estimation – input is read at least once:
Estimation – sampling:
Ci = #connections between vi′s neighbors di (di−1)/2
Local Clustering Coefficient
v1 v2 v3 v5 v6 v7 v4 v8 v9
di – degree of node i d1 = 1 d9 = 2 d2 = 3 C2 =1/3
Network Average CC = average local CC
Network Average CC
Exact: Naïve. Estimation – input is read at least once:
Estimation – sampling:
[Gjoka et al, 2010], This work – Improved accuracy.
Number of Registered Users
Exact: trivial Estimation – sampling:
[Katzir et al, 2011], This work – Improved accuracy.
Random Walk
v1 v2 v3 v5 v6 v7 v4 v8 v9 Sampled Nodes: v1 v2 v3 v4 1 22 3 22 2 22 2 22 Stationary Distribution =
𝑒𝑗 𝑒𝑗3 22 2 22 3 22 4 22 2 22 v5
Random Walk - Summary
v1 v2 v3 v5 v6 v7 v4 v8 v9 Visible Nodes Invisible Nodes Sampled Nodes Visible Edges Invisible Edges
Global CC Algorithm
– Sampled nodes average degree - 1.
𝜚𝑙 = 1 if there is an edge 𝑤𝑙−1 − 𝑤𝑙+1, 0 Otherwise.
The estimated global clustering coefficient: 𝑑 =
Φ Ψ
𝜚𝑙 = 1 iff 𝑤𝑙−1, 𝑤𝑙, 𝑤𝑙+1 is a triangle
Global CC Example
v1 v2 v3 v5 v4 𝜚2 = 0 𝜚3 = 1 Φ = 1 3 0 + 2 + 0 = 2 3 Ψ
= 15 0 + 2 + 1 + 3 + 1 = 7 5 𝑑 = 2 3 5 7 ≈ 0.47 𝑑 = 9 23 ≈ 0.39 𝜚4 = 0 v6 v7
Expectation of 𝝔𝒍
𝐹 𝜚𝑙𝑒𝑙 = 𝑒𝑗 𝐸 𝐹 𝜚𝑙𝑒𝑙|𝑦𝑙 = 𝑤𝑗
𝑜 𝑗=1= 𝑒𝑗 𝐸
𝑜 𝑗=12𝑚𝑗 𝑒𝑗𝑒𝑗 𝑒𝑗 = 2𝑚𝑗 𝐸
𝑜 𝑗=1Total expectation 𝑒𝑗𝑒𝑗 combinations. 2𝑚𝑗 yield 𝜚𝑙=1
𝑚𝑗 – The number of triangles contain vi. 𝑒𝑗 – The degree of node vi. 𝑜 – The number of nodes. 𝐸 = 𝑒𝑗
𝑜 𝑗=1 Global CC Proof
𝐸 = 𝑒𝑗
𝑜 𝑗=1𝑚𝑗 – The number of triangles contain vi. 𝑒𝑗 – The degree of node vi. 𝑜 – The number of nodes. 𝐹 Φ = 𝐹 𝜚𝑙𝑒𝑙 = 2 𝐸 𝑚𝑗
𝑜 𝑗=1𝐹 Ψ
= 1𝐸 𝑒𝑗 𝑒𝑗 − 1
𝑜 𝑗=1𝑑 = Φ
concentration bounds 𝐹 ΦΨ
concentration bounds 𝐹 Ψ ≅ 2 𝑚𝑗
𝑜 𝑗=1𝑒𝑗 𝑒𝑗 − 1
𝑜 𝑗=1= 𝑑
Guarantees
For any 𝜗 ≤ 1
8 and 𝜀 ≤ 1, we have
Prob 1 − 𝜁 𝑑 ≤ 𝑑 ≤ 1 + 𝜁 𝑑 ≥ 1 − 𝜀 when the number of samples, r, satisfies 𝑠 ≥ 𝑠
= 𝑃 mixing time(𝜁)
Network Average CC Algorithm
𝜚𝑙 = 1 if there is an edge 𝑤𝑙−1 − 𝑤𝑙+1, 0 Otherwise.
1 𝑒𝑙−1 .
The estimated network average CC: 𝑑𝑚 = Φ𝑚
Ψ𝑚
Evaluations
Network n (size) D/n cl cg DBLP 977,987 8.457 0.7231 0.1868 Orkut 3,072,448 76.28 0.1704 0.0413 Flickr 2,173,370 20.92 0.3616 0.1076 Live Journal 4,843,953 17.69 0.3508 0.1179 DBLP facts: Paper with most co-authors: has 119 listed authors. Most prolific author: Vincent Poor with 798 entries.
Global CC
Relative improvement ranges between 300% and 500% depending
0.5 1 1.5 2 2.5 3 3.5 0.5 1 1.5 2 Relative estimation value Percentage of mined nodes DBLP Network Gjoka et al* Ribeiro et al* This work
Network Average CC
Relative improvement ranges between 50% and 400% depending
0.5 1 1.5 2 2.5 0.5 1 1.5 2 Relative estimation value Percentage of mined nodes Orkut Network Ribeiro et al Gjoka et al Random walk
Conclusions
Clustering Coefficient.
Clustering Coefficient.
users.
Estimating Sizes of Social Networks via Biased Sampling
Liran Katzir
Yahoo! Labs, Haifa, Israel
Edo Liberty
Yahoo! Labs, Haifa, Israel
Oren Somekh
Yahoo! Labs, Haifa, Israel
The expected number of collisions in a list of r i.i.d. samples from a set of n elements is 𝑠 𝑠−1
2𝑜
.
The Birthday “Paradox”
A collision is a pair of identical samples. Example: Samples: X = (d, b, b, a, b, e). Total 3 collisions, (x2, x3), (x2, x5), and (x3, x5)
Cardinality estimation uniform
Needs 𝑠 = 𝑃 𝑜 samples to converge. Used by [Ye et al, 2010] to estimate the size. When C collisions are observed n ≅ 𝑠 𝑠 − 1 2𝐷
Stationary distribution sampling
v1 v2 v3 v5 v6 v7 v4 v8 v9 Sampled Nodes: v5 1 22 3 22 2 22 2 22 Stationary Distribution =
𝑒𝑗 𝑒𝑗3 22 2 22 3 22 4 22 2 22 v2 v5 v4 v2
Cardinality estimation stationary
Needs 𝑠 = 𝑃 𝑜
4log 𝑜 samples to converge when 𝑒𝑗~𝑨𝑗𝑞𝑔( 𝑜, 2). When C collisions are observed n ≅ 𝑒𝑦 1 𝑒𝑦 2𝐷
Example:
v1 v2 v3 v5 v6 v7 v4 v8 v9 v5 v2 v5 v4 v2
𝑒𝑦 = 2 + 3 + 2 + 4 + 3 1 𝑒𝑦 = 1 2 + 1 3 + 1 2 + 1 4 + 1 3 𝑜 =
1423
122∙2 ≈ 6.7
Global CC Proof
𝐸 = 𝑒𝑗
𝑜 𝑗=1𝑒𝑗 – The degree of node vi. 𝑜 – The number of nodes. 𝐹 𝑒𝑦 = 𝑒𝑗 𝐸 𝑒𝑗
𝑜 𝑗=1𝐹 1 𝑒𝑦 = 𝑒𝑗 𝐸 1 𝑒𝑗
𝑜 𝑗=1= 𝑜 𝐸 𝑜 = 𝑒𝑦 1 𝑒𝑦
concentration bounds 𝐹 𝑒𝑦 𝐹1 𝑒𝑦 2𝐷
concentration bounds 2𝐹 𝐷≅ 𝑒𝑗 𝐸 𝑒𝑗 𝑜 𝐸 𝑒𝑗 𝐸 𝑒𝑗 𝐸 = 𝑜 𝐹 𝐷 = 𝑒𝑗 𝐸 𝑒𝑗 𝐸
𝑜 𝑗=1 Improvements
All Samples
Restrict computation to indexes m steps apart, 𝐽 = 𝑙, 𝑚 | 𝑙 − 𝑚 ≥ 𝑛 A collision is only be considered within 𝐽. Φ = 𝑦𝑙 = 𝑦𝑚 | 𝑙, 𝑚 ∈ 𝐽 Ratio of degrees is similarly defined Ψ = 𝑒𝑦𝑙 𝑒𝑦𝑚
𝑙,𝑚 ∈𝐽
Conditional Monte Carlo
A collision between 𝑦𝑙 and 𝑦𝑚, is replaced by the conditional collision is steps k+1 and l+1 respectively. 𝐹 1𝑦𝑙+1=𝑦𝑚+1|𝑦𝑙, 𝑦𝑚 = Common Neighbors 𝑒𝑦𝑙𝑒𝑦𝑚
Conditional Monte Carlo
contributes 1
12 to the collision counter.
v1 v2 v3 v5 v6 v7 v4 v8 v9
Size Estimation
0.5 1 1.5 2 2.5 0.5 1 1.5 2 2.5 Relative estimation value Percentage of mined nodes DBLP Network Priot art This work
Thanks