Growth Rate of Spatially Coupled LDPC codes Workshop on Spatially - - PowerPoint PPT Presentation

Growth Rate of Spatially Coupled LDPC codes

森 立平

Workshop on Spatially Coupled Codes and Related Topics at Tokyo Institute of Technology 2011/2/19

Contents

2 / 34

• 1. Factor graph, Bethe approximation and belief propagation
• 2. Relation between annealed free energy and belief propagation
• 3. Growth rate of spatially coupled LDPC codes and threshold

saturation phenomenon Here, growth rate is G(ω) = lim

N→∞

1 N log E[Z(ω)] Z(ω): the number of codewords of relative weight ω ∈ [0, 1].

Factor graph, Bethe approximation and belief propagation

3 / 34

Factor graph

4 / 34

Factor graph: bipartite graph which defines probability measure p(x x x) = 1 Z

• a

fa(x x x∂a) Z :=

• x

x x∈X n

• a

fa(x x x∂a), (partition function) fa(x x x∂a) : X ra → R≥0

Gibbs free energy

5 / 34

p(x x x) = 1 Z

• a

fa(x x x∂a) Approximation by simple distribution q of p which is defined by factor graph D(qp) =

• x

x x

q(x x x) log q(x x x) p(x x x) = log Z−

• x

x x

q(x x x) log

• a

fa(x x x∂a)

• +
• x

x x

q(x x x) log q(x x x) =: log Z + U(q)−H(q) =: log Z + FGibbs(q) U(p): internal energy H(p): entropy FGibbs(p): Gibbs free energy

Mean field approximation and Bethe approximation

6 / 34

Mean field approximation q(x x x) =

• i

bi(xi) Degree of freedom is reduced from qn to nq Bethe approximation q(x x x) =

• a ba(x

x x∂a)

• i bi(xi)di−1

di: degree of variable node i When factor graph is tree, Bethe approximation can be exact

Bethe free energy

7 / 34

U(q) = −

• x

x x

q(x x x) log

• a

fa(x x x∂a)

• ≈−
• a
• x

x x∂a

ba(x x x∂a) log fa(x x x∂a) =: UBethe({ba}) b(x x x)≈

• a ba(x

x x∂a)

• i bi(x

x x)di−1 H(b) = −

• x

x x

b(x) log b(x) ≈−

• x

x x

b(x) log

• a ba(x

x x∂a)

• i bi(x

x x)di−1 = −

• a
• x

x x∂a

ba(x x x∂a) log ba(x x x∂a) +

• i

(di−1)

• i

bi(xi) log bi(xi) =: HBethe({bi}, {ba})

Minimization of Bethe free energy

8 / 34

FBethe({bi}, {ba}) := UBethe({ba})−HBethe({bi}, {ba}) minimize : FBethe({bi}, {ba}) subject to : bi(xi)≥0, ∀i ba(x x x∂a)≥0, ∀a

• i

bi(xi) = 1

• a

ba(x x x∂a) = 1

• x

x x∂a\xi

ba(x x x∂a) = bi(xi), ∀a, ∀i ∈ ∂a

Stationary point of Lagrangian of Bethe free energy

9 / 34

[Yedidia, Freeman, and Weiss 2005]

L := FBethe({bi}, {ba}) +

• a

γa

• x

x x∂a

ba(x x x∂a)−1

• +
• i

γi

• x

bi(x)−1

• +
• a
• i∈∂a
• xi

λai(xi)  bi(xi) −

• x

x x∂a\xi

ba(x x x∂a)   Stationary points of Lagrangian is fixed points of BP ba(x x x∂a) ∝ fa(x x x∂a)

• i∈∂a

mi→a(xi) bi(xi) ∝

• i∈∂a

ma→i(xi) where mi→a(xi) ∝

• c∈∂i\a

mc→i(xi) ma→i(xi) ∝

• x

x x∂a\xi

fa(x x x∂a)

• j∈∂a\i

mj→a(xj)

Relation between annealed free energy and belief propagation

10 / 34

Random regular factor graph ensemble

11 / 34

Factor graph: bipartite graph which defines probability measure µ(x x x) = 1 Z

• a

fa(x x x∂a) Z :=

• x

x x∈X n

• a

fa(x x x∂a), (partition function) Random (l, r)-regular factor graph ensemble: l: degree of variable nodes, r: degree of factor nodes Random ensemble of factor graphs

Annealed free energy

12 / 34

Factor graph: bipartite graph which defines probability measure µ(x x x) = 1 Z

• a

fa(x x x∂a) Z :=

• x

x x∈X n

• a

fa(x x x∂a), (partition function) Random (l, r)-regular factor graph ensemble: l: degree of variable nodes, r: degree of factor nodes Random ensemble of factor graphs (Quenched) free energy: lim

N→∞

1 N E[log Z] Annealed free energy: lim

N→∞

1 N log E[Z]

Contribution to partition function of particular types

13 / 34

{vx}x∈X: the number of variable nodes of value x ∈ X is vx {ux

x x}x x x∈X r: the number of factor nodes of value x

x x ∈ X r is ux

x x

Z =

• x

x x∈X N

• a

f (x x x∂a) =

• {v},{u}

N({v}, {u})

• x

x x∈X r

f (x x x)ux

x x.

E[N({v}, {u})] =

• N

{vx}x∈X

• l

r N

{ux

x x}x x x∈X r

• x∈X(vxl)!

(Nl)! . lim

N→∞

1 N log E[Z({ν}, {µ})] = l r H({µ})−(l−1)H({ν}) + l r

• x

x x∈X r

µ(x x x) log f (x x x).

Annealed free energy of fixed type and Bethe free energy

14 / 34

FBethe({bi}, {ba}) = −

• a
• x

x x∂a

ba(x x x∂a) log fa(x x x∂a) +

• a
• x

x x∂a

ba(x x x∂a) log ba(x x x∂a)−

• i

(di−1)

• i

bi(xi) log bi(xi) lim

N→∞

1 N log E[Z({ν}, {µ})] = l r

• x

x x∈X r

µ(x x x) log f (x x x) + l r H({µ})−(l−1)H({ν}).

Maximization of the exponents of contributions

15 / 34

maximize : l r H({µ})−(l−1)H({ν}) + l r

• x

x x∈X r

µ(x x x) log f (x x x) subject to : ν(x)≥0, ∀x ∈ X µ(x x x)≥0, ∀x x x ∈ X r

• x∈X

ν(x) = 1

• x

x x∈X r

µ(x x x) = 1 1 r

r

• k=1
• x

x x\xk xk=z

µ(x x x) = ν(z), ∀z ∈ X

The stationary condition

16 / 34

The stationary condition is ν(x) ∝ mf →v(x)l µ(x x x) ∝ f (x x x)

r

• i=1

mv→f (xi) where mv→f (x) ∝ mf →v(x)l−1 mf →v(x) ∝

r

• k=1
• x

x x\xk xk=x

f (x x x)

• j=k

mv→f (xj). If f (x x x) is invariant under any permutation of x x x ∈ X r mf →v(x) ∝

• x

x x\x1 x1=x

f (x x x)

• j=1

mv→f (xj).

Annealed free energy

17 / 34

Theorem 1. lim

N→∞

1 N log E[Z] = max

(mf →v,mv→f )∈S

l r log Zf + log Zv−l log Zfv

• .

where S denotes the set of saddle points, and where Zv :=

• x

mf →v(x)l Zf :=

• x

x x

f (x x x)

r

• i=1

mv→f (xi) Zfv :=

• x

mf →v(x)mv→f (x).

Number of solutions

18 / 34

If

r

• k=1
• x

x x\xk xk=x

f (x x x) is constant among all x ∈ X, the uniform message mf →v(x)、mv→f (x) is a saddle point. The contribution of the uniform message is lim

N→∞

1 N log E[Z(ν, µ)] = log q + l r log Nf qr

• (design rate)

where q := |X|、Nf :=

x x x f (x

x x). For CSP i.e., f (x x x) ∈ {0, 1}, the expected number of solutions is about qN Nf qr l

r N

. This intuitively means all constraints are independent.

Contribution to partition function of fixed variable type

19 / 34

Z({ν}) :=

• {µ}

Z({ν}, {µ}) lim

N→∞

1 N log E[Z({ν})] = sup

{µ}

• l

r H({µ})−(l−1)H({ν}) + l r

• x

x x∈X r

µ(x x x) log f (x x x)

• where {µ} satisfies

µ(x x x)≥0, ∀x x x ∈ X r

• x

x x∈X r

µ(x x x) = 1 1 r

r

• k=1
• x

x x\xk xk=z

µ(x x x) = ν(z), ∀z ∈ X Convex optimization problem with linear constraints.

The stationary condition

20 / 34

The stationary condition is µ(x x x) ∝ f (x x x)

r

• i=1

mv→f (xi) where ν(x) ∝ h(x)mf →v(x)l mv→f (x) ∝ h(x)mf →v(x)l−1 mf →v(x) ∝

r

• k=1
• x

x x\xk xk=x

f (x x x)

• j=k

mv→f (xj). If f (x x x) is invariant under any permutation of x x x ∈ X r mf →v(x) ∝

• x

x x\x1 x1=x

f (x x x)

• j=1

mv→f (xj).

Growth rate of contribution to partition function of fixed variable type

21 / 34

Theorem 2. lim

N→∞

1 N log E[Z({ν})] = max

(mf →v,mv→f )∈S

• l

r log Zf + log Zv−l log Zfv−

• x

ν(x) log h(x)

• where S denotes the set of saddle points, and where

Zv :=

• x

h(x)mf →v(x)l Zf :=

• x

x x

f (x x x)

r

• i=1

mv→f (xi) Zfv :=

• x

mf →v(x)mv→f (x).

Growth rate of regular LDPC codes

22 / 34

G(ω) = l r log 1 + zr 2 + log

• eh

1 + y 2 l + e−h 1−y 2 l −l log 1 + yz 2 −ω′h where ω′ := 1−2ω and ω′ = tanh(h + l tanh−1(y)) y = zr−1 z = tanh(h + (l−1) tanh−1(y)). This result can be easily understood from correspondings ω′ = ν(0)−ν(1) y = mf →v(0)−mf →v(1) z = mv→f (0)−mv→f (1) h(x) = e(−1)xh

Growth rate of regular LDPC codes

23 / 34

• 0.4
• 0.3
• 0.2
• 0.1

0.1 0.2 0.3 0.4 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 G w (5,10) (6,10) (7,10) (8,10) (9,10) (10,10) (11,10)

Growth rate of binary CSP

24 / 34

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 G w k=1 k=2 k=3

Growth rate of (3,2)-regular-3-coloring

25 / 34

0.1 0.2 0.3 0.4 0.5 0.1 0.2 0.3 0.4 0.5

• 1
• 0.8
• 0.6
• 0.4
• 0.2

0.2 0.4 0.6

• 1
• 0.8
• 0.6
• 0.4
• 0.2

0.2 0.4 0.6

Replica theory

26 / 34

This story continues into the replica theory (see the paper in arXiv). But, we don’t deal with it here. E[log Z] = ∂ log E[Z n] ∂n

• n=0

lim

N→∞

1 N E[log Z] = lim

N→∞

1 N lim

n→0

log E[Z n] n

?

= lim

n→0

1 n lim

N→∞

1 N log E[Z n] The replica method is methematically not rigorous e.g., exchange of limits, analytic continuation of n.

Growth rate of spatially coupled LDPC codes and threshold saturation phenomenon

27 / 34

Protograph ensemble

28 / 34

The similar results also holds for protograph ensemble [Vontobel 2010] In this morning, Kenta has explained

■ Definition of protograph ensemble ■ Definition of spatially coupled LDPC codes ■ Threshold saturation phenomenon of EXIT curve

Growth rate of spatially coupled LDPC codes

29 / 34

G(ω) = 1 2L + 1

• l

r

L+l−1

• j=−L

log

• log 1 + l−1

k=0 zj,k

r l

2

• +

L

• i=−L

log

• eh

l−1

• k=0

1 + yi,k 2

• + e−h

l−1

• k=0

1−yi,k 2

• −

L

• i=−L

l−1

• k=0

log 1 + yi,kzi+k,k 2 −ω′h. ω′ = 1 2L + 1

L

• i=−L

tanh

• h +

l−1

• k=0

tanh−1 (yi,k)

• zj,k = tanh

 h +

l−1

• k′=0,k′=k

tanh−1 yj−k,k′   yi,k = zi+k,k

r l −1

l−1

• k′=0,k′=k

zi+k,k′

r l

ω′ versus h

30 / 34

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.2 0.4 0.6 0.8 1 0.4408159 L=2,4,8,16,32,64,128,256 (5,10)

h ω′

ω versus h: Derivative of growth rate

31 / 34

• 1
• 0.5

0.5 1 0.2 0.4 0.6 0.8 1 L=2,4,8,16,32,64,128,256 (5,10)

h ω

Growth rate

32 / 34

• 0.15
• 0.1
• 0.05

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.2 0.4 0.6 0.8 1 L=2,4,8,16,2048 (5,10)

G(ω) ω

Conclusion

33 / 34

■ Contribution to annealed free energy of particular type has similar

form of Bethe free energy.

■ The stationary condition of maximization problem for annealed free

energy is similar to equation of belief propagation.

■ There exists threshold saturation phenomenon in the calculation of

growth rate of spatially coupled LDPC codes.

■ We now can calculate annealed free energy of any coupled factor

• graphs. Effect of boundary condition is not obvious. BP iterations

does not necessarily converge (even for uncoupled cases).

Acknowledgment

34 / 34

The basic idea that the growth rate approaches to the concave hull is given by Nicolas Macris. I acknowledges Hamed Hassani and Toshiyuki Tanaka for encouragement and discussion.

■ arXiv:1102.3132, paper about connection between Bethe and

annealed free energies (submitted to ISIT 2011).

■ Joint paper with Hamed and Nicolas about growth rate of coupled

LDPC codes was submitted to ISIT 2011.