STATISTICAL PHYSICS OF LEARNING (REVISITED) Florent Krzakala Ecole - - PowerPoint PPT Presentation

Florent Krzakala

Ecole Normale Supérieure, Paris

STATISTICAL PHYSICS OF LEARNING (REVISITED)

Léo Miolane (ENS) Jean Barbier

(EPFL→London)

Lenka Zdeborova

(CNRS, Saclay)

Nicolas Macris (EPFL)

LEARNING A RULE

STATISTICAL PHYSICS

?

N=2, M=22, α=M/N=11

x∗

i ∼ PX

y = sign(Fx∗)

coordinates

• f points:

hyper-plane parameters: labels

Fµi ∼ N(0, 1/N)

α = O(1), N, M → ∞

Goals: Estimation: Infer x* from M samples/examples of (Fµ,yµ). Generalization: Predict labels of new points.

EX:TEACHER-STUDENT PERCEPTRON

STATISTICAL PHYSICS

P(X|Y ) = P(Y |X)P(X) Z = elog (P (Y |X)P (X)) Z

X Y a probabilistic process

P(Y |X)

P(X)

Generative model Observation Posterior probability Hamiltonian Partition sum

TWO QUESTIONS

Many predictions from the heuristic replica method:

Asymptotic free energy/mutual information Derrida & Gardner 89’ Optimal overlap & optimal generalization Gyorgyi - Sompolinski & al 90’s can we prove them mathematically rigorously?

Both answered in this talk. What about practical algorithmic performance?

Not the focus of the physics literature until the early 00’s Can we reach optimal performances with efficient algorithms?

GENERALIZED LINEAR REGRESSION

labels: data matrix: regression parameters: noise: component-wise function e.g.:

y ∈ RM F ∈ RM×N ξ ∈ RM

y = ϕξ(Fx)

x ∈ RN

xi ∈ {±1}

ϕξ(x) = sign(z)

Goal: Find good fitting parameters x, from examples (Fµ,yµ).

Special cases: Compressed sensing, signal reconstruction in

computed tomography, magnetic resonance imaging, phase retrieval. LASSO, superposition error correcting codes, code-division multiple- access problem, group testing, logistic regression, …

Theorem 1 (informally): The replica free entropy is correct. ρ = EPX(x2) where

Barbier, FK, Macris, Miolane, Zdeborova, arXiv:1708.03395, COLT 2018

• Def. free entropy:

ΦPX( ˆ m) ≡ Ez,x0 h ln Ex h e ˆ

mxx0+ √ ˆ mxz− ˆ mx2/2ii

ΦPout(m; ρ) ≡ Ev,z h Z dy Pout(y|√m v + √ρ − m z) ln Ew ⇥ Pout(y|√m v + √ρ − m w) ⇤ i

x, x0 ∼ PX z, v, w ∼ N(0, 1)

α = M N

f ≡ lim

N→∞

1 N Ey,F log Z(y, F)

MAIN RESULTS

f = sup

m inf ˆ m fRS(m, ˆ

m) fRS(m, ˆ m) = ΦPX( ˆ m) + αΦPout(m; ρ) − m ˆ m 2

Theorem 1 (informally): The replica free entropy is correct. Theorem 2 (informally): Optimal error on estimation of x* is: ρ = EPX(x2)

MMSE = ρ − m∗

where m* is the extremizer of fRS.

α = M N

f = sup

m inf ˆ m fRS(m, ˆ

m) fRS(m, ˆ m) = ΦPX( ˆ m) + αΦPout(m; ρ) − m ˆ m 2

Barbier, FK, Macris, Miolane, Zdeborova, arXiv:1708.03395, COLT 2018

MAIN RESULTS

f ≡ lim

N→∞

1 N Ey,F log Z(y, F)

• Def. free entropy:

Theorem 1 (informally): The replica free entropy is correct.

f = sup

m inf ˆ m fRS(m, ˆ

m) fRS(m, ˆ m) = ΦPX( ˆ m) + αΦPout(m; ρ) − m ˆ m 2

Theorem 3 (informally): Optimal generalisation error is ρ = EPX(x2) where m* is the extremizer of fRS.

Egen = E

v,ξ

⇥ fξ(√ρ v)2⇤ −E

v E w,ξ

⇥ fξ( √ m∗ v+√ρ − m∗ w) ⇤2

v, w ∼ N(0, 1)

ξ ∼ Pξ

α = M N

Barbier, FK, Macris, Miolane, Zdeborova, arXiv:1708.03395, COLT 2018

MAIN RESULTS

f ≡ lim

N→∞

1 N Ey,F log Z(y, F)

• Def. free entropy:

PROOF IDEA

ξ ∼ N(0, 1)

x∗ ∼ PX

ΦPX( ˆ m) ΦPout(m; ρ) fRS(m, ˆ m) = ΦPX( ˆ m) + αΦPout(m; ρ) − m ˆ m 2 ˜ y ∼ Pout(˜ y|√m v + √ρ − m z∗)

v, z∗ ∼ N(0, 1)

ρ = EPX(x2) is the free entropy of a scalar denoising problem Notice is the free entropy of a scalar denoising problem

y0 = √ ˆ mx⇤ + ξ

Guerra-Toninelli-like interpolation between the original posterior and N + M independent scalar denoising problems. Interpolating Hamiltonian:

st,µ = √ 1 − t[Fx]µ + sZ t m(t0)dt0vµ + sZ t (ρ − m(t0))dt0zµ

Ht = −

M

X

µ=1

ln Pout(yµ|st,µ) + 1 2

N

X

i=1

(y0

i −

√ t ˆ mxi)2

fN = fN(t = 1) − Z 1 f 0

N(t)dt

PROOF IDEA

Adaptive interpolation: Choose interpolation path m(t) to cancel the integral

(Crucial contribution by Barbier & Macris, Probab. Theory Relat. Fields ’08)

Key Ingredient: (Nishimori/Bayes theorem): Under expectations ground truth x* is exchangeable for a sample from P(x|y,F).

Ex∗,yEP (x|y)[g(y, x(1), x∗)] = EyEP (x|y)[g(y, x(1), x(2))]

fN = freplica(m, ˆ m) + Z 1 R(m(t), ˆ m, t)dt

R(m(t), ˆ m, t)

with a complicated function

PROOF IDEA

PX(x) = 1 2[δ(x − 1) + δ(x + 1)]

EX: BINARY PERCEPTRON TRANSITION TO PERFECT LEARNING

y = sign(Fx∗)

Generalization error

α αIT = 1.249

PX(x) = 1 2[δ(x − 1) + δ(x + 1)]

y = sign(Fx∗)

Generalization error

α αIT = 1.249

Gyorgyi’90

CLOSING A 28 YEARS OLD CONJECTURE…

EX: BINARY PERCEPTRON TRANSITION TO PERFECT LEARNING

… AND SOLVING MANY NEW PROBLEMS:

COMPRESSED SENSING, PHASE RETRIEVAL, SUPERPOSITION CODES, LOGISTIC REGRESSION…

WHAT ABOUT EFFICIENT ALGORITHMS ?

A GENERIC APPROACH: THE TAP EQUATIONS

“Improved” mean-field equations for spin glasses

t+1 t t t

mi = tanh 2 4h + X

j

βJijmj − β2 X

j

J2

ij(1 − m2 j)mi

3 5

THOULESS-ANDERSON-PALMER ACCORDING TO BOLTHAUSEN

t+1 t t t-1

mi = tanh 2 4h + X

j

βJijmj − β2 X

j

J2

ij(1 − m2 j)mi

3 5

Convergence of the equations is now proven!

TAP EQUATIONS FOR PERCEPTRON

• M. Mézard (’89)

APPROXIMATE MESSAGE PASSING

THE MODERN AVATAR OF TAP

at+1

i

= f(At, Bt

i)

vt+1

i

= ✓ ∂f ∂B ◆ (At, Bt

i)

Mean and variance

• f the marginals:

η(Σ, R)

RIGOROUS STATE EVOLUTION

Define: mt in the AMP algorithm ( ) evolves as: then

MSE(t) = ρ − mt mt+1 = 2∂ ˆ

mΦPX( ˆ

mt) ˆ mt = 2α∂mΦPout(mt; ρ) mt ≡ 1 p

p

X

i=1

x∗

i at i

n, p → ∞, α = Θ(1)

Bolthausen ’06, Bayati-Montanari ’10, Rangan ’10, Donoho-Javamart-Montanari ‘12

fRS(m, ˆ m) = ΦPX( ˆ m) + αΦPout(m; ρ) − m ˆ m 2

Recall the RS free entropy? AMP is simply trying to extremize the replica function!

AMP-MSE given by the local maximum of the free entropy reached starting from small m/large MSE. Optimal MSE (MMSE) given by the global maximum of the free entropy.

WHEN DOES AMP WORK?

m free entropy mAMP mAMP mAMP

fRS(m, ˆ m) = ΦPX( ˆ m) + αΦPout(m; ρ) − m ˆ m 2

MMSE = ρ − argmaxfRS(m)

MSEAMP = ρ − mAMP

argmax fRS(m)

fRS(m) = inf ˆ

mfRS(m, ˆ

m)

Barbier, FK, Macris, Miolane, Zdeborova, arXiv:1708.03395, COLT 2018

PX(x) = 1 2[δ(x − 1) + δ(x + 1)]

EX: BINARY PERCEPTRON

Generalization error

y = sign(Fx∗) α αIT = 1.249 αAlg = 1.493

Gyorgyi’90

GAMP is optimal, out of the hard phase. Redemption of the “un-physical” branch.

hard

THIS HAPPENS ALL THE TIME!

PX(x) = 1 2[δ(x − 1) + δ(x + 1)]

SYMMETRIC BINARY PERCEPTRON

Generalization error K chosen so that P(y=1)=0.5 from: Barbier, FK, Macris, Miolane, Zdeborova arXiv:1708.03395

y = sign(|Fx∗| − K) αAlg = 1.566 αIT = 1

hard

data

F y

labels

x w1 w2

weights

p input units K hidden units

• utput unit

L=3 layers n training samples x learned, w fixed Limit:

Committee machine

Quenched teacher-student model studied in Schwarze’92. Proof and approximate message passing (Aubin, Maillard, Barbier, Macris, FK, Zdeborova ’18)

α = M N = O(1)

M, N → ∞

K = O(1)

MULTI-LAYER NETWORK

Large algorithmic gap: IT threshold: Algorithmic threshold

yµ = sign h K X

l=1

sign(

p

X

i=1

Fµix∗

il)

i

2 4 6 8 10 12

• α ≡ α/K

0.0 0.1 0.2 0.3 0.4 0.5

Generalization error g(˜ α)

0.0 0.2 0.4 0.6 0.8 1.0

Overlap q

SE q00 SE q01 SE g Spinodal 1st order transition

K 1

MULTI-LAYER NETWORK

αIT = 7.65K αAlgo = O(K2)

hard

HARD PHASE EVERYWHERE

Hard phase identified in:

• stochastic block model
• dense planted sub-matrix;
• low-rank tensor completion;
• compressed sensing;
• planted constraint satisfaction;
• Gaussian mixture clustering;
• low-density parity check error

correcting codes;

• sparse principal component analysis;
• generalised linear regression;
• dictionary learning;
• blind source separation;
• learning in binary perceptron;
• phase retrieval; …

Hard phase = spinodal region

• f first order phase transitions.

Fact : So far all known polynomial algorithms fail in the hard phase! Conjecture: AMP/TAP achieves the lowest error among all polynomial algorithms (asymptotically)

CONLUSIONS

Proof of the replica formulas and prediction for many learning/ inference problems (here perceptron, but similar results holds for generalised linear models, matrix and tensor factorisation, simple neural nets …) A mean-field type algorithm acheive optimal prediction in polytime… … unless metastability appears, in which case we believe nothing works! Many more challenging mathematical physics problems in learning and computer science (see Lenka Zdeborova tomorrow at 15.45)

Shameless advertising:

Many Postdoc positions opened in Ecole Normale in Paris

References

The committee machine: Computational to statistical gaps in learning a two-layers neural network

• B. Aubin, A. Maillard, J. Barbier, FK, N. Macris and L. Zdeborová arXiv:1806.05451

Fundamental limits of detection in the spiked Wigner model

• A. El Alaoui, FK, M. I. Jordan arXiv:1806.09588

Phase Transitions, Optimal Errors and Optimality of Message-Passing in Generalized Linear Models

• J. Barbier, FK, N. Macris, L. Miolane, L. Zdeborová, COLT 2018

Statistical and computational phase transitions in spiked tensor estimation

• T. Lesieur, L. Miolane, M. Lelarge, FK, L. Zdeborová, ISIT 2017

Information-theoretic thresholds from the cavity method

• A. Coja-Oghlan, FK, W. Perkins, L. Zdeborova STOC 2017 & Advances in Mathematics 2018

The Mutual Information in Random Linear Estimation

• J. Barbier, M. Dia, N. Macris, FK, Allerton 2017

Mutual information for symmetric rank-1 matrix estimation: A proof of the replica formula

• J. Barbier, M. Dia, N. Macris, FK, T. Lesieur, L. Zdeborova, NIPS 2017

Mutual Information in Rank-One Matrix Estimation FK, J. Xu, L. Zdeborová, ITW 2016

Thank you for your attention!