Replica method: a statistical mechanics approach to - - PowerPoint PPT Presentation

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography

Replica method: a statistical mechanics approach to probability-based information processing

Toshiyuki Tanaka tt@i.kyoto-u.ac.jp

Graduate School of Informatics, Kyoto University, Kyoto, Japan

17th International Symposium on Mathematical Theory of Networks and Systems, Kyoto, Japan, July 25, 2006

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography

Introduction

Replica method Developed in studies of spin glasses (=magnetic materials with random spin-spin interactions) Recently applied to problems in information sciences:

Neural networks Statistical learning theory Combinatorial optimization problems Error-correcting codes CDMA (digital wireless communication) Eigenvalue distribution of random matrices

Still lacks rigorous mathematical justification

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography

Introduction

Objectives To give a review of the replica method, as well as its mathematically questionable point. To demonstrate its applications.

Eigenvalue distribution of random matrices. Analysis of digital communication systems.

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic results Approaches Analysis via replica method

Problem

Basic defs. A: N × N real symmetric random matrix λ1, . . . , λN: Eigenvalues of A Empirical eigenvalue distribution ρA(x) = 1 N

N

• i=1

δ(x − λi) Problem To evaluate ρ(x) = lim

N→∞ EA

• ρA(x)
• Toshiyuki Tanaka

MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic results Approaches Analysis via replica method

Basic results

Wigner’s semicircle law (Wigner, 1951) A = (aij): N × N matrix, aij (i ≤ j): i.i.d., mean 0, variance 1/N. Mar˘ cenko-Pastur law (Mar˘ cenko & Pastur, 1967) A = ΞTΞ, Ξ = (ξµi): p × N matrix; ξµi i.i.d., mean 0, variance 1/N. (Girko’s) full-circle law (Girko, 1985) A = (aij), aij: i.i.d., mean 0, variance 1/N. (A not symmetric)

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic results Approaches Analysis via replica method

Wigner’s semicircle law

Wigner’s semicircle law A = (aij), aij (i ≤ j): i.i.d, mean 0, variance 1/N. ⇒ ρ(x) = ⎧ ⎨ ⎩ 1 2π

• 4 − x2

(|x| < 2) (|x| > 2)

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic results Approaches Analysis via replica method

Wigner’s semicircle law

• 2
• 1

1 2

Histogram of eigenvalues of a 6000 × 6000 random symmetric matrix with entries following Gaussian distribution.

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic results Approaches Analysis via replica method

Mar˘ cenko-Pastur law

Mar˘ cenko-Pastur law A = ΞTΞ, Ξ = (ξµi): p × N matrix; ξµi: i.i.d., mean 0, variance 1/N. ρ(x) = ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩

• 4α − (x − 1 − α)2

2πx χα(x) (α ≥ 1) (1 − α)δ(x) +

• 4α − (x − 1 − α)2

2πx χα(x) (0 < α < 1) α ≡ p/N χα(x): Characteristic function of interval [(1 − √α)2, (1 + √α)2].

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic results Approaches Analysis via replica method

Mar˘ cenko-Pastur law

0.1 0.2 0.3 0.4 0.5 0.6 0.7 1 2 3 4 5 6 ρ(x) x

α = 0.3, 0.6, 2; Terms proportional to δ(x) not shown.

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic results Approaches Analysis via replica method

Full-circle

• 1
• 0.5

0.5 1

• 1
• 0.5

0.5 1 Im(λ) Re(λ)

Eigenvalue distribution of a real 6000 × 6000 random matrix.

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic results Approaches Analysis via replica method

Eigenvalue distribution of random matrices

Useful for what? Wide applications in mathematical physics Applications in Information Processing

Statistical learning theory Digital communication (kernel) PCA (bioinformatics, mathematical finance, etc.)

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic results Approaches Analysis via replica method

Eigenvalue distribution of random matrix

Approaches Marginalization of joint eigenvalue distribution (ex. Mehta,

1967)

Evaluation of moments (ex. Brody et al., 1981) “Locator” expansion (ex. Bray & Moore, 1979; Hertz et al., 1989) Cavity method Free probability theory (ex. Voiculescu, 1985; Hiai & Petz, 2000) Replica method (ex. Edwards & Jones, 1976)

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic results Approaches Analysis via replica method

Reformulation

ρA(x) = 1 N

N

• i=1

δ(x − λi) mA(z) = 1 N tr(A − zI)−1 = 2 N d dz log ZA(z) ZA(z) = (−2πi)N/2|A − zI|−1/2 =

• RN exp
• − i

2uT(A − zI)u

• du

mA(z) =

• R

ρA(x) x − z dx ρA(x) = lim

ǫ→+0

1 πℑ

• mA(x + iǫ)
• Stieltjes trans.

tr∗−1 = (log det ∗)′ Gaussian integ.

• rep. of det.

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic results Approaches Analysis via replica method

Averaging over A

ρ(x) = EA 1 N

N

• i=1

δ(x − λi)

• m(z) = EA

1 N tr(A − zI)−1

• = 2 d

dz EA 1 N log ZA(z)

• ZA(z) =
• RN exp
• − i

2uT(A − zI)u

• du

mA(z) =

• R

ρA(x) x − z dx ρA(x) = lim

ǫ→+0

1 πℑ

• mA(x + iǫ)
• Stieltjes trans.

tr∗−1 = (log det ∗)′

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic results Approaches Analysis via replica method

Outline of the approach

1 Evaluate

f (z) = lim

N→∞ EA

1 N log ZA(z)

• where
• ZA(z) =
• RN exp
• − i

2uT(A − zI)u

• du
• 2 Calculate Stieltjes transform m(z) of ρ(x) with

m(z) = 2 d dz f (z).

3 Evaluate the inverse Stieltjes transform to obtain ρ(x):

ρ(x) = lim

ǫ→+0

1 πℑ

• m(x + iǫ)
• Toshiyuki Tanaka

MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic results Approaches Analysis via replica method

Replica method

Rewriting of formulas f (z) = lim

N→∞ EA

1 N log ZA(z)

• ∂

∂n log EA

• Z n

= EA(Z n log Z) EA(Z n)

• =

lim

N→∞

1 N lim

n→0

∂ ∂n log EA

• ZA(z)

n

• Exchange order of limn→0 ∂/∂n

and limN→∞.

• = lim

n→0

∂ ∂n lim

N→∞

1 N log EA

• ZA(z)

n

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic results Approaches Analysis via replica method

Replica method

Goal To evaluate lim

N→∞

1 N log EA

• ZA(z)

n . The replica “trick” Evaluate it by assuming n to be a positive integer. Believe the result to be valid for real n. No mathematically rigorous justification.

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic results Approaches Analysis via replica method

Random matrix ensemble

Covariance matrix of random samples A = ΞTΞ, Ξ = (ξµi) {ξµi; µ = 1, . . . , p; i = 1, . . . , N} i.i.d., E(ξ) = 0, E(ξ2) = O(1/N), E(ξm) = o(1/N) (m ≥ 3) ⇒ Mar˘ cenko-Pastur

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic results Approaches Analysis via replica method

Replica method

Introduction of “Replicated” systems ZA(z) =

• RN exp
• − i

2uT(A − zI)u

• du

⇒ For n = 1, 2, . . .,

• ZA(z)

n =

• RNn exp
• − i

2

n

• a=1

uT

a (A − zI)ua

• n
• a=1

dua EA

• ZA(z)

n =

• RNn EA
• exp
• − i

2

n

• a=1

uT

a Aua

• × exp

iz 2

n

• a=1
• ua
• 2

n

• a=1

dua

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic results Approaches Analysis via replica method

Calculation of EA(· · · )

Factor to be evaluated EA

• exp
• − i

2

n

• a=1

uT

a Aua

• Assumptions on ramdom mtx. ensemble

A = ΞTΞ, Ξ = (ξµi) (size: p × N) {ξµi; µ = 1, . . . , p; i = 1, . . . , N}: i.i.d. vµa ≡

N

• i=1

ξµiuai ⇒ uT

a Aua = p

• µ=1
• vµa

2

n

• a=1

uT

a Aua = p

• µ=1

n

• a=1
• vµa

2 =

p

• µ=1
• vµ
• 2,
• vµ = (vµ1, . . . , vµn)T

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic results Approaches Analysis via replica method

Calculation of EA(· · · )

Average over A = Average over v vµa ≡

N

• i=1

ξµiuai ⇒ uT

a Aua = p

• µ=1
• vµa

2

n

• a=1

uT

a Aua = p

• µ=1

n

• a=1
• vµa

2 =

p

• µ=1
• vµ
• 2
• vµ = (vµ1, . . . , vµn)T

⇒ EA

• exp
• − i

2

n

• a=1

uT

a Aua

• =
• Ev
• exp
• − i

2

• v
• 2p
• v = (ξTu1, . . . , ξTun)T

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic results Approaches Analysis via replica method

Assumptions on ramdom mtx. ensemble E(ξ) = 0, E(ξ2) = 1/N, E(ξm) = o(1/N) (m ≥ 3) Statistical properties of v = (ξT u1, . . . , ξTun)T v ∼ N(0, Q) for fixed {ua} (⇐ Central limit theorem) Order parameters Q = (qab), qab ≡ Eξ(vvT) = N−1

N

• i=1

uaiubi ⇒ Ev

• exp
• − i

2

• v
• 2

=

• I + iQ
• −1/2

(Gaussian integral) exp iz 2

n

• a=1
• ua
• 2

= exp Niz 2 trQ

• Toshiyuki Tanaka

MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic results Approaches Analysis via replica method

Integral with {ua} = Integral with Q Ev

• exp
• − i

2

• v
• 2

=

• I + iQ
• −1/2

(Gaussian integral) exp iz 2

n

• a=1
• ua
• 2

= exp Niz 2 trQ

• ⇒

EA

• ZA(z)

n =

• eNG(Q) µ(Q) dQ

G(Q) ≡ −α 2 log

• I + iQ
• + iz

2 trQ, α ≡ p/N µ(Q) ≡

a≤b

δ

• qab − 1

N

N

• i=1

uaiubi

• n
• a=1

dua (Subshell volume)

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic results Approaches Analysis via replica method

Varadhan’s theorem (Large deviation theory) lim

N→∞

1 N log

• eNG(Q) µ(Q) dQ = sup

Q

• G(Q) − I(Q)
• Rate function I(Q)

The heuristic formula µ(Q) = e−NI(Q) holds for large N with I(Q) = −1 2 log |Q| + n 2

• 1 + log(−2π)
• .

Stationary condition (saddle-point equation) ∂ ∂Q

• G(Q) − I(Q)
• = O

⇒ izI + Q−1 − iα(I + iQ)−1 = O

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic results Approaches Analysis via replica method

m(z) = lim

n→0

∂ ∂nitrQ, izI + Q−1 − iα(I + iQ)−1 = O Q uniquely determined by requiring integrals not to diverge. ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ Q = qI iz + 1 q − iα 1 + iq = 0 m(z) = iq ⇒ m(z) = −

• z −

α 1 + m(z) −1 ρ(x) = ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩

• 4α − (x − 1 − α)2

2πx χα(x) (α ≥ 1) (1 − α)δ(x) +

• 4α − (x − 1 − α)2

2πx χα(x) (0 < α < 1) (Mar˘ cenko-Pastur)

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic setting Use of randomness in communication Replica method

Communication

Basic setting

X Input Channel Y Output p(x) p(y|x)

Output Y · · · What we observe. Input X · · · What we want to know! Problem How much the output Y conveys information about the input X?

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic setting Use of randomness in communication Replica method

Communication

Problem How much the output Y conveys information about the input X? Mutual information I(X; Y ) =

• p(x, y) log p(x, y)

p(x)p(y) dx dy = H(Y ) − H(Y |X) H(Y |X) = − p(y|x) log p(y|x) dy

• p(x) dx

H(Y ) = −

• p(y) log p(y) dy

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic setting Use of randomness in communication Replica method

Use of randomness in communication

Basic diagram

X Input Channel ? Y Output

Error-correcting code: ? =Encoder Modulation: ? =Modulator

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic setting Use of randomness in communication Replica method

Use of randomness in communication

Examples “Turbo” codes (Berrou et al., 1993): Two convolutional codes interlinked with a random interleaver. Low-density parity-check codes (Gallager, 1962): Random ensemble of low-density parity-check matrices. Code-division multiple-access (CDMA): Spreading modulation with random spreading sequences.

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic setting Use of randomness in communication Replica method

Mobile communication

• Toshiyuki Tanaka

MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic setting Use of randomness in communication Replica method

Multiple access and CDMA

Multiple-access: Multiple users simultaneously commu- nicate with the same base station.

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic setting Use of randomness in communication Replica method

2-user CDMA system

× ⇑ × ⇑ Noise × ⇓ ⇓ Alice Information x1 Spreading code {sµ1} Bob Information x2 Spreading code {sµ2} Base st. {yµ} Rcvd. signal {sµ1} h1 ⇒ ˆ x1

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic setting Use of randomness in communication Replica method

K-user CDMA system

x1 x2 xK

• Information

{sµ1}{sµ2} {sµK}

• Spreading codes

× × ×

• Channel

+ Noise {nµ}

• Rcvd. signal

{yµ} yµ = 1 √ N

K

• k=1

sµkxk + nµ (µ = 1, . . . , N)

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic setting Use of randomness in communication Replica method

Use of randomness in communication

Basic diagram

X Input Channel S Y Output p(x) p(y|x, s)

Modeling of randomness p(y|x, s): Channel input-output characteristics depending on auxiliary random variable S (e.g., spreading codes).

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic setting Use of randomness in communication Replica method

Use of randomness in communication

Randomness-averaged mutual information One wants to evaluate: ES

• I(X; Y |S)
• = ES
• H(Y |S)
• − ES
• H(Y |X, S)
• Difficulty

ES

• H(Y |S)
• = −ES
• p(y|S) log p(y|S) dy
• p(y|S) =
• p(y|x, S) p(x) dx

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic setting Use of randomness in communication Replica method

Replica method

Evaluation of randomness-averaged entropy ES

• H(Y |S)
• = −ES
• p(y|S) log p(y|S) dy
• = − lim

n→0

∂ ∂n log Ξn Ξn ≡ ES p(y|S) n+1 dy

• =

p(y|s) n+1 p(s) dy ds Replica method provides a powerful approach to evaluate randomness-averaged mutual information.

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic setting Use of randomness in communication Replica method

Example of results

10-7 10-6 10-5 10-4 10-3 10-2 10-1 100 2 4 6 8 10 12 Bit-Error Rate Pb Signal-to-Noise Ratio Eb /N0 [dB]

System load β = 1, 1.2, 1.4, 1.6, 1.8, 2 Single-user limit

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic setting Use of randomness in communication Replica method

Discussion

S-shaped performance curve Multiple solutions for performance. Essentially the same as magnetization curve of ferromagnets: {Stable, Metastable, Unstable} solutions

• 1
• 0.5

0.5 1

• 0.4
• 0.2

0.2 0.4

Magnetization External magnetic field

“Hysteresis” · · · Affects behavior of estimation algorithms.

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic setting Use of randomness in communication Replica method

Replica method: Applications

Digital communication Turbo codes (Montanari & Sourlas, 2000) Low-density parity-check codes (Murayama et al., 2000) Code-division multiple-access (Tanaka, 2002) · · · Other fields Associative memory of neural network, Hopfield model (Amit et al., 1985) Perceptron learning (Gardner & Derrida, 1989) Random K-SAT problem (Monasson & Zecchina, 1997) · · ·

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography Basic setting Use of randomness in communication Replica method

Replica method

Mathematics Validity of replica solutions (not replica method) (Talagrand, 2003) “It is difficult to see (in the replica method) more than a way to guess the correct formula.” — Talagrand, 2003. Current status Empirically gives the correct results to various problems. Heuristics: Validity unknown. Justification (or counterexample) needed.

Toshiyuki Tanaka MTNS2006: Replica method

bg=black!2

Introduction Limiting eigenvalue distribution of random matrices Digital communication Bibliography

Bibliography

• D. J. Amit et al., Phys. Rev. Lett., 55(14), 1530–1533, 1985.

Berrou et al., Proc. IEEE Int. Conf. Commun., 1064–1070, 1993.

• A. Bray & Moore, J. Phys. C: Solid State Phys., 12(11), L441–L448, 1979.
• T. A. Brody et al., Rev. Mod. Phys., 53(3), 385–479, 1981.
• S. F. Edwards & R. C. Jones, J. Phys. A: Math. Gen., 9(10), 1595–1603, 1976.
• R. G. Gallager, Trans. IRE Info. Theory, 8, 21–28, 1962.
• E. Gardner & B. Derrida, J. Phys. A: Math. Gen., 22(12), 1983–1994, 1989.
• V. L. Girko, Theory of Prob. Its Appl. (USSR), 29(4), 694–706, 1985.
• J. A. Hertz et al., J. Phys. A: Math. Gen., 22(12), 2133–2150, 1989.
• F. Hiai & D. Petz, The Semicircle Law, Free Random Variables, and Entropy, Amer. Math. Soc., 2000.
• V. A. Mar˘

cenko & L. A. Pastur, Math. USSR-Sb., 1, 457–483, 1967.

• M. L. Mehta, Random Matrices, Academic Press, 1967.
• R. Monasson & R. Zecchina, Phys. Rev. E, 56(2), 1357–1370, 1997.
• A. Montanari & N. Sourlas, Eur. Phys. J. B, 18(1) 107–119, 2000.
• T. Murayama et al., Phys. Rev. E, 62(2), 1577–1591, 2000.
• M. Talagrand, Spin Glasses: A Challenge for Mathematicians, Springer, 2003.
• T. Tanaka, IEEE Trans. Info. Theory, 48(11), 2888–2910, 2002.
• D. V. Voiculescu, in Operator Algebras and Their Connection with Topology and Ergodic Theory,

Lecture Notes in Math., 1131, Springer, 1985, 556–588.

• E. P. Wigner, Proc Cambridge Phil. Soc., 47, 790–798, 1951.

Toshiyuki Tanaka MTNS2006: Replica method