Gossip algorithms for solving Laplacian systems Anastasios Zouzias - - PowerPoint PPT Presentation
Gossip algorithms for solving Laplacian systems Anastasios Zouzias University of Toronto joint work with Nikolaos Freris (EPFL) Based on : 1.Fast Distributed Smoothing for Clock Synchronization (CDC 12) 2.Randomized Extended Kaczmarz for
Anastasios Zouzias University of Toronto
joint work with Nikolaos Freris (EPFL)
Based on : 1.Fast Distributed Smoothing for Clock Synchronization (CDC ‘12) 2.Randomized Extended Kaczmarz for Solving Least-Squares (Arxiv, May ‘12)
III.Randomized Gossip Model & Averaging Problem IV.Gossip Solvers via Randomized (Extended) Kaczmarz
III.Randomized Gossip Model & Averaging Problem IV.Gossip Solvers via Randomized (Extended) Kaczmarz
exchanges packets with them only
errors; ignore numerical issues
Model of computation G=(V,E): n nodes, m edges
b1 b4 b2 b3 b5 b7 b8 b9 b6 b13 b12 b10 b11 x2 x1 x4 x5 x6 x13 x12 x9 x8 x11 x10 x3 x7
Problem I Input: Each node i gets bi Goal: Each node i computes
ith coordinate of xLS := L†b
b x =
Laplacian System
G=(V,E): n nodes, m edges
2 2
7
1
3 6
7 3 9
8 4 1
x2 x1 x4 x5 x6 x13 x12 x9 x8 x11 x10 x3 x7
Edge-vertex System
= B y
m × n
x
Problem II Input: Each edge (i,j) gets y(i,j) Goal: Each node i computes ith
coordinate of xLS = B†y
Bek := −1, if k = i; 1, if k = j; 0,
Normal equation of Bx=y is Laplacian system (L = B⊤B)
III.Randomized Gossip Model & Averaging Problem IV.Gossip Solvers via Randomized (Extended) Kaczmarz
WSNs
v Assumptions
Nodes can approx. relative
for every
u ∈ Neigh(v)
Neigh(v)
Clock Synchronization Problem Input: Estimates for all
(u, v) ∈ E
Goal: Compute offsets that min.
(˜
max
u,v E |˜
ˆ
G=(V,E): n nodes, m edges
Idea: Build a spanning tree In general, no hope for better accuracy ...but wireless networks are ``well-connected’’
u v Sync error between u & v grows like ≈ O(
Every edge: normal error
ˆ
ˆ
Path u~v : diam(G)
Q: Can we do better on Random Geometric Graphs?
Yes! Spatial Smoothing [KEES03,GK06] ...as Random Geometric Graphs Square: unit area
O
log n
r = O
n
Observation: Every loop in G: sum of relative offsets equals zero
Idea: Incorporate the loop constraints Bx = o
Relative offset
Encode constraints in linear system:
How?
= B
m × n
x
Thm[KEES03] Replace each edge by unit resistor. Then error variance between
any pair of nodes u and v is :
E |˜
Effective resistances of RGG bounded by O(1) [GK06]
Q: How to compute the LS solution?
Gaussian error: compute LS solution of Bx = ˆ
O(n1/4) O(1)
vs
Asynchronous Jacobi
Each node regularly:
v ∈ V
˜
dv
(˜
ˆ
Thm[GK06]: After rounds, it holds that where is the min-cut value
k ≥ 4m2 β2 ln(x∗ /ε)
β
It converges[BT89]
Synchronous Jacobi
For k = 1, 2, . . .
˜
v
← 1 dv
u
+ ˆ
∀v ∈ V
Use coordinate descent:
∂ ∂xu Bx − o2 = 0
Thm[GK06]: After rounds, it holds that where is the min-cut value
k ≥ 4m2 β2 ln(x∗ /ε)
β
Synchronous Model
For k = 1, 2, . . .
˜
v
← 1 dv
u
+ ˆ
∀v ∈ V
all
Asynchronous Model
Each node regularly:
v ∈ V
˜
dv
(˜
ˆ
It converges[BT89]
nothing
Randomized Gossip Model
(a.k.a. asynchronous time model) [BTA86,BGPS06]
Each node u (randomly) activates itself w.p. pu & performs local computation
III.Randomized Gossip Model & Averaging Problem IV.Gossip Solvers via Randomized (Extended) Kaczmarz
7 1 3 2
Distributed Averaging:
Input: Every node u gets Goal: Every node want access
to global average
wu
5 4 4 How many rounds required to approx. withinε?
Basic primitive for
Averaging can solve Problems I and II
[BDFSV10,XBL05,XBL06]
Gossip averaging algorithm
1.Every node u activates uniformly at random 2.Picks random neighbor v and averages their current values wu,wv
2.5 2.5
[BGPS06] proved that rounds are sufficient whp Special cases, complete graph [KSSV00,KDG03,KDN+06]
O( n λ2(G) log(n/ε))
Claim: Non-uniform sampling of nodes is feasible with zero communication under gossip model (given ‘s)
γu
Assumptions
γu
γu
*minimum of ind. Poisson is equivalent to single Poisson with sum of their rates
Goal: Design and analyze gossip algorithms for Problem I and II
III.Randomized Gossip Model & Averaging Problem IV.Gossip Solvers via Randomized (Extended) Kaczmarz
H2 Hm
x
x(0)
x(1)
x(2)
x(m)
y
Repeat:
Set ik = k mod m + 1
k = k + 1
x(0) = 0
x(k+1) = x(k) + yik −
2 A(ik)
Initialize: Kaczmarz Method (K) (Assumption: Ax=y has solution) It convergences [K37] Huge literature; many extensions; rediscovered many times
H1 =
H1 =
ik ∈ [m]
pi ∝
Repeat:
k = k + 1
x(0) = 0
x(k+1) = x(k) + yik −
2 A(ik)
Initialize:
Randomized Kaczmarz(RK) [SV06]
Pick rows randomly
H2 Hm
x(0)
x(1)
x(2)
x(m)
x
Exponential convergence
E
≤
1 κ2
F(A)
k x∗2
κ2
F(A) :=
A2
F
σ2
min(A)
where
(Assumption: Ax=y has solution)
b1 b2 b3 b5 b7 b8 b9 b6 b13 b12 b10 b11 x2 x1 x4 x5 x6 x13 x12 x9 x8 x11 x10 x3 x7
b4
b x =
Laplacian System
(Assumption: Lx=b has solution) Pick w.p.
Repeat:
k = k + 1
x(0) = 0
Initialize:
Randomized Kaczmarz(RK) [SV06]
pi ∝
ik ∈ [n]
x(k+1) = x(k) + bik −
2 L(ik)
Node u activates w.p.
Repeat:
Each node u: Gossip Laplacian Solver
d2
u + du
θ = xu − 1 du
xℓ
Every : xv ← xv + θ − bu/du
1 + du
xu ← xu + bu − duθ 1 + du v ∈ Neigh(u)
xu = 0
sparsity
RK analysis & diag. preconditioning : rounds whp
2(G))
2 2
7
1
3 6
7 3 9
8 4 1
x2 x1 x4 x5 x6 x13 x12 x9 x8 x11 x10 x3 x7
=
B y
m × n
x
Randomized Kaczmarz(RK) [SV06]
Pick uniformly
Repeat:
k = k + 1 x(0) = 0
Initialize:
e = (i, j) ∈ E
x(k+1) = x(k) + ye −
2 B(e)
(Assumption: Bx=y has solution) Node u activates w.p. du & selects random neighbor v
Repeat:
Every node u: xu=0 Gossip Edge-Vertex Solver
Similarly for node v
xu ← (xu + xv + y(u,v))/2 sparsity
RK analysis & diag. preconditioning : rounds whp
Consistency assumption (limitation of RK) How to handle the general case?
H1 =
H2 Hm
xLS
Converges to ball centered at LS solution
x
w +
E
≤
1 κ2
F(A)
k xLS2 + w2 σ2
min(A)
RK is robust to noise [Needell09]
Randomized Kaczmarz(RK) [SV06]
Pick w.p.
ik ∈ [m]
pi ∝
Repeat:
k = k + 1
x(0) = 0
Initialize:
x(k+1) = x(k+1) + yik + wik −
2 A(ik)
(Assumption: Ax=y has solution)
`Modify’ original system Ax=y s.t. 1) xLS is preserved 2) Modified system has LS error ≈ 0 Idea:
H2
xLS
H3
H1
✓ ✓ Problem
Given subspace as colspan(A), vector y and
Goal: Find z s.t.
ε > 0
Robustness of RK: convergence into fixed ball
Ax = y Ax = yR(A)
How?
E
1 κ2
F(A)
k y2
Exponential convergence[Z.Freris12]:
Orthogonality gives yR(A)
Problem
Given subspace as colspan(A), vector y and
Goal: Find z s.t.
ε > 0
jk ∈ [n]
z(k+1) =
A(jk)A⊤
(jk)
Pick w.p.
Repeat:
k = k + 1
Initialize: randOP
pj ∝
z(0) = y
x(k+1) = x(k) + yik − z(k)
ik −
2 A(ik)
y - z(k)→yR(A)
Pick w.p.
ik ∈ [m]
pi ∝
Repeat:
k = k + 1
x(0) = 0
Initialize: ;
Randomized Extended Kaczmarz
Pick w.p.
jk ∈ [n]
p(j) ∝
z(k+1) = (I − Pjk)z(k)
RK applied on Ax=yR(A)
E
≤
1 κ2
F(A)
⌊ k
2 ⌋xLS2 + 2 b2 σmin(A)
*Inspired by Extended Kaczmarz method [Pop99]
RK + randOP = Randomized Extended Kaczmarz
z(0) = y
2 2
7
1
3 6
7 3 9
8 4 1
x2 x1 x4 x5 x6 x13 x12 x9 x8 x11 x10 x3 x7
=
B y
m × n
x
Repeat:
k = k + 1
x(0) = 0
Initialize: ;
Randomized Extended Kaczmarz
z(k+1) = (I − Pjk)z(k)
Node u activates w.p. du & selects random neighbor v
Repeat:
Every node u: xu=0, z(u,v)=y(u,v) v in Nu Gossip Edge-Vertex Solver
Similarly node v
adjacent to u;broadcast to nghbrs
xu ← (xu + xv + y(u,v) − z(u,v))/2
sparsity
z(0) = y
Pick node w.p. Pick uniformly
e = (i, j) ∈ E
x(k+1) = x(k) + ye − z(k)
e
−
2 B(e)
Same rate of convergence
djk
jk
b1 b2 b3 b5 b7 b8 b9 b6 b13 b12 b10 b11 x2 x1 x4 x5 x6 x13 x12 x9 x8 x11 x10 x3 x7
b4
b x =
Laplacian System
Two solutions: 1.Use REK as before 2.Use RK & gossip averaging to project b
1⊥ b′ = b − bavg1
Topics not covered: