Variational inference, spin glasses, and TAP free energy Song Mei - - PowerPoint PPT Presentation

Variational inference, spin glasses, and TAP free energy

Song Mei

Stanford University

September 19, 2018

Joint work with Zhou Fan and Andrea Montanari

Song Mei (Stanford University) TAP free energy September 19, 2018 1 / 29

General motivation

◮ Bayesian inference: high dimensional integration is hard! ◮ Variational inference: integration/summation ✦ optimization.

A popular objective function: “mean field free energy”.

◮ Applications: topic modeling, stochastic block model, low rank

matrix estimation, compressed sensing.... ... within which “MF free energy” is known to be not optimal.

◮ Today: introduce the optimal objective “TAP free energy”, and

provide rigorous results.

Song Mei (Stanford University) TAP free energy September 19, 2018 2 / 29

General motivation

◮ Bayesian inference: high dimensional integration is hard! ◮ Variational inference: integration/summation ✦ optimization.

A popular objective function: “mean field free energy”.

◮ Applications: topic modeling, stochastic block model, low rank

matrix estimation, compressed sensing.... ... within which “MF free energy” is known to be not optimal.

◮ Today: introduce the optimal objective “TAP free energy”, and

provide rigorous results.

Song Mei (Stanford University) TAP free energy September 19, 2018 2 / 29

General motivation

◮ Bayesian inference: high dimensional integration is hard! ◮ Variational inference: integration/summation ✦ optimization.

A popular objective function: “mean field free energy”.

◮ Applications: topic modeling, stochastic block model, low rank

matrix estimation, compressed sensing.... ... within which “MF free energy” is known to be not optimal.

◮ Today: introduce the optimal objective “TAP free energy”, and

provide rigorous results.

Song Mei (Stanford University) TAP free energy September 19, 2018 2 / 29

General motivation

◮ Bayesian inference: high dimensional integration is hard! ◮ Variational inference: integration/summation ✦ optimization.

A popular objective function: “mean field free energy”.

◮ Applications: topic modeling, stochastic block model, low rank

matrix estimation, compressed sensing.... ... within which “MF free energy” is known to be not optimal.

◮ Today: introduce the optimal objective “TAP free energy”, and

provide rigorous results.

Song Mei (Stanford University) TAP free energy September 19, 2018 2 / 29

General motivation

◮ Bayesian inference: high dimensional integration is hard! ◮ Variational inference: integration/summation ✦ optimization.

A popular objective function: “mean field free energy”.

◮ Applications: topic modeling, stochastic block model, low rank

matrix estimation, compressed sensing.... ... within which “MF free energy” is known to be not optimal.

◮ Today: introduce the optimal objective “TAP free energy”, and

provide rigorous results.

Song Mei (Stanford University) TAP free energy September 19, 2018 2 / 29

General motivation

◮ Bayesian inference: high dimensional integration is hard! ◮ Variational inference: integration/summation ✦ optimization.

A popular objective function: “mean field free energy”.

◮ Applications: topic modeling, stochastic block model, low rank

matrix estimation, compressed sensing.... ... within which “MF free energy” is known to be not optimal.

◮ Today: introduce the optimal objective “TAP free energy”, and

provide rigorous results.

Song Mei (Stanford University) TAP free energy September 19, 2018 2 / 29

Z✷ synchronization

◮ Signal:

x ❂ ❬①✶❀ ✿ ✿ ✿ ❀ ①♥❪T ✷ Z♥

✷ ❀

①✐

✐✿✐✿❞✿

✘ ❯♥✐❢✭Z✷✮❀ Z✷ ❂ ❢✰✶❀ ✶❣✿

◮ Observation: for ✶ ✔ ✐ ❁ ❥ ✔ ♥

❨✐❥ ❂ ✕ ♥①✐①❥ ✰ ❲✐❥✿

◮ Noise ❲✐❥ ✘ ◆✭✵❀ ✶❂♥✮. ◮ SNR ✕ ✷ ❬✵❀ ✶✮ fixed, dimension ♥ ✦ ✶. ◮ In matrix notation:

Y ❂ ✕ ♥xxT ✰ W ✿

◮ Task: given Y ❂ ✭❨✐❥✮, estimate x (or say X ❂ xxT).

Song Mei (Stanford University) TAP free energy September 19, 2018 3 / 29

Z✷ synchronization

◮ Signal:

x ❂ ❬①✶❀ ✿ ✿ ✿ ❀ ①♥❪T ✷ Z♥

✷ ❀

①✐

✐✿✐✿❞✿

✘ ❯♥✐❢✭Z✷✮❀ Z✷ ❂ ❢✰✶❀ ✶❣✿

◮ Observation: for ✶ ✔ ✐ ❁ ❥ ✔ ♥

❨✐❥ ❂ ✕ ♥①✐①❥ ✰ ❲✐❥✿

◮ Noise ❲✐❥ ✘ ◆✭✵❀ ✶❂♥✮. ◮ SNR ✕ ✷ ❬✵❀ ✶✮ fixed, dimension ♥ ✦ ✶. ◮ In matrix notation:

Y ❂ ✕ ♥xxT ✰ W ✿

◮ Task: given Y ❂ ✭❨✐❥✮, estimate x (or say X ❂ xxT).

Song Mei (Stanford University) TAP free energy September 19, 2018 3 / 29

Z✷ synchronization

◮ Signal:

x ❂ ❬①✶❀ ✿ ✿ ✿ ❀ ①♥❪T ✷ Z♥

✷ ❀

①✐

✐✿✐✿❞✿

✘ ❯♥✐❢✭Z✷✮❀ Z✷ ❂ ❢✰✶❀ ✶❣✿

◮ Observation: for ✶ ✔ ✐ ❁ ❥ ✔ ♥

❨✐❥ ❂ ✕ ♥①✐①❥ ✰ ❲✐❥✿

◮ Noise ❲✐❥ ✘ ◆✭✵❀ ✶❂♥✮. ◮ SNR ✕ ✷ ❬✵❀ ✶✮ fixed, dimension ♥ ✦ ✶. ◮ In matrix notation:

Y ❂ ✕ ♥xxT ✰ W ✿

◮ Task: given Y ❂ ✭❨✐❥✮, estimate x (or say X ❂ xxT).

Song Mei (Stanford University) TAP free energy September 19, 2018 3 / 29

Z✷ synchronization

◮ Signal:

x ❂ ❬①✶❀ ✿ ✿ ✿ ❀ ①♥❪T ✷ Z♥

✷ ❀

①✐

✐✿✐✿❞✿

✘ ❯♥✐❢✭Z✷✮❀ Z✷ ❂ ❢✰✶❀ ✶❣✿

◮ Observation: for ✶ ✔ ✐ ❁ ❥ ✔ ♥

❨✐❥ ❂ ✕ ♥①✐①❥ ✰ ❲✐❥✿

◮ Noise ❲✐❥ ✘ ◆✭✵❀ ✶❂♥✮. ◮ SNR ✕ ✷ ❬✵❀ ✶✮ fixed, dimension ♥ ✦ ✶. ◮ In matrix notation:

Y ❂ ✕ ♥xxT ✰ W ✿

◮ Task: given Y ❂ ✭❨✐❥✮, estimate x (or say X ❂ xxT).

Song Mei (Stanford University) TAP free energy September 19, 2018 3 / 29

Z✷ synchronization

◮ Signal:

x ❂ ❬①✶❀ ✿ ✿ ✿ ❀ ①♥❪T ✷ Z♥

✷ ❀

①✐

✐✿✐✿❞✿

✘ ❯♥✐❢✭Z✷✮❀ Z✷ ❂ ❢✰✶❀ ✶❣✿

◮ Observation: for ✶ ✔ ✐ ❁ ❥ ✔ ♥

❨✐❥ ❂ ✕ ♥①✐①❥ ✰ ❲✐❥✿

◮ Noise ❲✐❥ ✘ ◆✭✵❀ ✶❂♥✮. ◮ SNR ✕ ✷ ❬✵❀ ✶✮ fixed, dimension ♥ ✦ ✶. ◮ In matrix notation:

Y ❂ ✕ ♥xxT ✰ W ✿

◮ Task: given Y ❂ ✭❨✐❥✮, estimate x (or say X ❂ xxT).

Song Mei (Stanford University) TAP free energy September 19, 2018 3 / 29

Z✷ synchronization

◮ Signal:

x ❂ ❬①✶❀ ✿ ✿ ✿ ❀ ①♥❪T ✷ Z♥

✷ ❀

①✐

✐✿✐✿❞✿

✘ ❯♥✐❢✭Z✷✮❀ Z✷ ❂ ❢✰✶❀ ✶❣✿

◮ Observation: for ✶ ✔ ✐ ❁ ❥ ✔ ♥

❨✐❥ ❂ ✕ ♥①✐①❥ ✰ ❲✐❥✿

◮ Noise ❲✐❥ ✘ ◆✭✵❀ ✶❂♥✮. ◮ SNR ✕ ✷ ❬✵❀ ✶✮ fixed, dimension ♥ ✦ ✶. ◮ In matrix notation:

Y ❂ ✕ ♥xxT ✰ W ✿

◮ Task: given Y ❂ ✭❨✐❥✮, estimate x (or say X ❂ xxT).

Song Mei (Stanford University) TAP free energy September 19, 2018 3 / 29

Bayes estimation in Z✷ synchronization

◮ Settings:

x ✘ ❯♥✐❢✭Z♥

✷ ✮❀

Y ❂ ✭✕❂♥✮xxT ✰ W ✿

◮ Estimate X ❂ xxT with loss:

❵✭X❀ ❝ X✮ ❂ ✭✶❂♥✷✮❦X ❝ X❦✷

❋ ✿ ◮ For ✕ ❁ ✶, estimation is impossible. ◮ For ✕ ❃ ✶, estimation is possible and efficient, e.g., spectral

estimator (Baik, Ben Arous, Peche phase transition).

◮ The optimal estimator is the Bayes estimator (also minimax

estimator):

❝

X❇❛②❡s ❂ E❬xxT❥Y ❪✿

Song Mei (Stanford University) TAP free energy September 19, 2018 4 / 29

Bayes estimation in Z✷ synchronization

◮ Settings:

x ✘ ❯♥✐❢✭Z♥

✷ ✮❀

Y ❂ ✭✕❂♥✮xxT ✰ W ✿

◮ Estimate X ❂ xxT with loss:

❵✭X❀ ❝ X✮ ❂ ✭✶❂♥✷✮❦X ❝ X❦✷

❋ ✿ ◮ For ✕ ❁ ✶, estimation is impossible. ◮ For ✕ ❃ ✶, estimation is possible and efficient, e.g., spectral

estimator (Baik, Ben Arous, Peche phase transition).

◮ The optimal estimator is the Bayes estimator (also minimax

estimator):

❝

X❇❛②❡s ❂ E❬xxT❥Y ❪✿

Song Mei (Stanford University) TAP free energy September 19, 2018 4 / 29

Bayes estimation in Z✷ synchronization

◮ Settings:

x ✘ ❯♥✐❢✭Z♥

✷ ✮❀

Y ❂ ✭✕❂♥✮xxT ✰ W ✿

◮ Estimate X ❂ xxT with loss:

❵✭X❀ ❝ X✮ ❂ ✭✶❂♥✷✮❦X ❝ X❦✷

❋ ✿ ◮ For ✕ ❁ ✶, estimation is impossible. ◮ For ✕ ❃ ✶, estimation is possible and efficient, e.g., spectral

estimator (Baik, Ben Arous, Peche phase transition).

◮ The optimal estimator is the Bayes estimator (also minimax

estimator):

❝

X❇❛②❡s ❂ E❬xxT❥Y ❪✿

Song Mei (Stanford University) TAP free energy September 19, 2018 4 / 29

Bayes estimation in Z✷ synchronization

◮ Settings:

x ✘ ❯♥✐❢✭Z♥

✷ ✮❀

Y ❂ ✭✕❂♥✮xxT ✰ W ✿

◮ Estimate X ❂ xxT with loss:

❵✭X❀ ❝ X✮ ❂ ✭✶❂♥✷✮❦X ❝ X❦✷

❋ ✿ ◮ For ✕ ❁ ✶, estimation is impossible. ◮ For ✕ ❃ ✶, estimation is possible and efficient, e.g., spectral

estimator (Baik, Ben Arous, Peche phase transition).

◮ The optimal estimator is the Bayes estimator (also minimax

estimator):

❝

X❇❛②❡s ❂ E❬xxT❥Y ❪✿

Song Mei (Stanford University) TAP free energy September 19, 2018 4 / 29

Bayes estimation in Z✷ synchronization

◮ Settings:

x ✘ ❯♥✐❢✭Z♥

✷ ✮❀

Y ❂ ✭✕❂♥✮xxT ✰ W ✿

◮ Estimate X ❂ xxT with loss:

❵✭X❀ ❝ X✮ ❂ ✭✶❂♥✷✮❦X ❝ X❦✷

❋ ✿ ◮ For ✕ ❁ ✶, estimation is impossible. ◮ For ✕ ❃ ✶, estimation is possible and efficient, e.g., spectral

estimator (Baik, Ben Arous, Peche phase transition).

◮ The optimal estimator is the Bayes estimator (also minimax

estimator):

❝

X❇❛②❡s ❂ E❬xxT❥Y ❪✿

Song Mei (Stanford University) TAP free energy September 19, 2018 4 / 29

Bayes estimation in Z✷ synchronization

◮ Settings:

x ✘ ❯♥✐❢✭Z♥

✷ ✮❀

Y ❂ ✭✕❂♥✮xxT ✰ W ✿

◮ Risk:

▼❙❊✕✭❝ X✮ ❂ ✭✶❂♥✷✮E❬❦xxT ❝ X❦✷

❋ ❪✿

0.5 1 1.5 2 2.5 3 3.5 4 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 MSE Bayes MSE PCA MSE

Song Mei (Stanford University) TAP free energy September 19, 2018 5 / 29

Compute the Bayesian estimator

◮ The Bayesian estimator:

❝

X❇❛②❡s ❂ E❬xxT❥Y ❪ ❂

❳

σ✷Z♥

✷

σσT♣✭σ❥Y ✮✿

◮ The posterior distribution:

♣✭σ❥Y ✮ ❂ ✶ ❩ ❡①♣❢✕❤σ❀ Y σ✐❂✷❣✿

Song Mei (Stanford University) TAP free energy September 19, 2018 6 / 29

Compute the Bayesian estimator

◮ The Bayesian estimator:

❝

X❇❛②❡s ❂ E❬xxT❥Y ❪ ❂

❳

σ✷Z♥

✷

σσT♣✭σ❥Y ✮✿

◮ The posterior distribution:

♣✭σ❥Y ✮ ❂ ✶ ❩ ❡①♣❢✕❤σ❀ Y σ✐❂✷❣✿

Song Mei (Stanford University) TAP free energy September 19, 2018 6 / 29

Mean field variational inference

◮ The posterior distribution:

♣✭σ❥Y ✮ ❂ ✶ ❩ ❡①♣❢✕❤σ❀ Y σ✐❂✷❣✿

◮ Approximate ♣✭σ❥Y ✮ by q ✷ P▼❋:

P▼❋ ❂

♥

q✭σ✮ ❂

♥

❨

✐❂✶

q✐✭✛✐✮ ✿ q✐ ✷ P✭Z✷✮

♦ ✘

❂ ❬✶❀ ✶❪♥✿

◮ Minimize the relative entropy between q and ♣✭σ❥Y ✮:

♠✐♥

q✷P▼❋ D❦❧✭q❦♣✭σ❥Y ✮✮✿ ◮ Equivalently minimizing ♠✐♥m✷❬✶❀✶❪♥ ❋▼❋✭m✮

❋▼❋✭m✮ ✑

♥

❳

✐❂✶

h✭♠✐✮ ✕❤m❀ Y m✐❂✷ ✕ ❧♦❣ ❩❀ where h✭♠✮ ❂ ✶♠

✷

❧♦❣✭ ✶♠

✷ ✮ ✶✰♠ ✷

❧♦❣✭ ✶✰♠

✷ ✮.

Song Mei (Stanford University) TAP free energy September 19, 2018 7 / 29

Mean field variational inference

◮ The posterior distribution:

♣✭σ❥Y ✮ ❂ ✶ ❩ ❡①♣❢✕❤σ❀ Y σ✐❂✷❣✿

◮ Approximate ♣✭σ❥Y ✮ by q ✷ P▼❋:

P▼❋ ❂

♥

q✭σ✮ ❂

♥

❨

✐❂✶

q✐✭✛✐✮ ✿ q✐ ✷ P✭Z✷✮

♦ ✘

❂ ❬✶❀ ✶❪♥✿

◮ Minimize the relative entropy between q and ♣✭σ❥Y ✮:

♠✐♥

q✷P▼❋ D❦❧✭q❦♣✭σ❥Y ✮✮✿ ◮ Equivalently minimizing ♠✐♥m✷❬✶❀✶❪♥ ❋▼❋✭m✮

❋▼❋✭m✮ ✑

♥

❳

✐❂✶

h✭♠✐✮ ✕❤m❀ Y m✐❂✷ ✕ ❧♦❣ ❩❀ where h✭♠✮ ❂ ✶♠

✷

❧♦❣✭ ✶♠

✷ ✮ ✶✰♠ ✷

❧♦❣✭ ✶✰♠

✷ ✮.

Song Mei (Stanford University) TAP free energy September 19, 2018 7 / 29

Mean field variational inference

◮ The posterior distribution:

♣✭σ❥Y ✮ ❂ ✶ ❩ ❡①♣❢✕❤σ❀ Y σ✐❂✷❣✿

◮ Approximate ♣✭σ❥Y ✮ by q ✷ P▼❋:

P▼❋ ❂

♥

q✭σ✮ ❂

♥

❨

✐❂✶

q✐✭✛✐✮ ✿ q✐ ✷ P✭Z✷✮

♦ ✘

❂ ❬✶❀ ✶❪♥✿

◮ Minimize the relative entropy between q and ♣✭σ❥Y ✮:

♠✐♥

q✷P▼❋ D❦❧✭q❦♣✭σ❥Y ✮✮✿ ◮ Equivalently minimizing ♠✐♥m✷❬✶❀✶❪♥ ❋▼❋✭m✮

❋▼❋✭m✮ ✑

♥

❳

✐❂✶

h✭♠✐✮ ✕❤m❀ Y m✐❂✷ ✕ ❧♦❣ ❩❀ where h✭♠✮ ❂ ✶♠

✷

❧♦❣✭ ✶♠

✷ ✮ ✶✰♠ ✷

❧♦❣✭ ✶✰♠

✷ ✮.

Song Mei (Stanford University) TAP free energy September 19, 2018 7 / 29

Mean field variational inference

◮ The posterior distribution:

♣✭σ❥Y ✮ ❂ ✶ ❩ ❡①♣❢✕❤σ❀ Y σ✐❂✷❣✿

◮ Approximate ♣✭σ❥Y ✮ by q ✷ P▼❋:

P▼❋ ❂

♥

q✭σ✮ ❂

♥

❨

✐❂✶

q✐✭✛✐✮ ✿ q✐ ✷ P✭Z✷✮

♦ ✘

❂ ❬✶❀ ✶❪♥✿

◮ Minimize the relative entropy between q and ♣✭σ❥Y ✮:

♠✐♥

q✷P▼❋ D❦❧✭q❦♣✭σ❥Y ✮✮✿ ◮ Equivalently minimizing ♠✐♥m✷❬✶❀✶❪♥ ❋▼❋✭m✮

❋▼❋✭m✮ ✑

♥

❳

✐❂✶

h✭♠✐✮ ✕❤m❀ Y m✐❂✷ ✕ ❧♦❣ ❩❀ where h✭♠✮ ❂ ✶♠

✷

❧♦❣✭ ✶♠

✷ ✮ ✶✰♠ ✷

❧♦❣✭ ✶✰♠

✷ ✮.

Song Mei (Stanford University) TAP free energy September 19, 2018 7 / 29

Mean field variational inference

◮ Mean field free energy:

❋▼❋✭m✮ ✑

♥

❳

✐❂✶

h✭♠✐✮ ✕❤m❀ Y m✐❂✷✿

◮ For m❄ ❂ ❛r❣ ♠✐♥m ❋▼❋✭m✮, we hope

m❄mT

❄ ✙ ❝

X❇❛②❡s ❂ E❬xxT❥Y ❪✿

◮ It was shown that m❄mT ❄ ✻✙ E❬xxT❥Y ❪ [Ghorbani, Javadi, and

Montanari, 2017].

◮ The assumption that posterior distribution can be approximately

factorized into the product of marginals is wrong!

Song Mei (Stanford University) TAP free energy September 19, 2018 8 / 29

Mean field variational inference

◮ Mean field free energy:

❋▼❋✭m✮ ✑

♥

❳

✐❂✶

h✭♠✐✮ ✕❤m❀ Y m✐❂✷✿

◮ For m❄ ❂ ❛r❣ ♠✐♥m ❋▼❋✭m✮, we hope

m❄mT

❄ ✙ ❝

X❇❛②❡s ❂ E❬xxT❥Y ❪✿

◮ It was shown that m❄mT ❄ ✻✙ E❬xxT❥Y ❪ [Ghorbani, Javadi, and

Montanari, 2017].

◮ The assumption that posterior distribution can be approximately

factorized into the product of marginals is wrong!

Song Mei (Stanford University) TAP free energy September 19, 2018 8 / 29

Mean field variational inference

◮ Mean field free energy:

❋▼❋✭m✮ ✑

♥

❳

✐❂✶

h✭♠✐✮ ✕❤m❀ Y m✐❂✷✿

◮ For m❄ ❂ ❛r❣ ♠✐♥m ❋▼❋✭m✮, we hope

m❄mT

❄ ✙ ❝

X❇❛②❡s ❂ E❬xxT❥Y ❪✿

◮ It was shown that m❄mT ❄ ✻✙ E❬xxT❥Y ❪ [Ghorbani, Javadi, and

Montanari, 2017].

◮ The assumption that posterior distribution can be approximately

factorized into the product of marginals is wrong!

Song Mei (Stanford University) TAP free energy September 19, 2018 8 / 29

Mean field variational inference

◮ Mean field free energy:

❋▼❋✭m✮ ✑

♥

❳

✐❂✶

h✭♠✐✮ ✕❤m❀ Y m✐❂✷✿

◮ For m❄ ❂ ❛r❣ ♠✐♥m ❋▼❋✭m✮, we hope

m❄mT

❄ ✙ ❝

X❇❛②❡s ❂ E❬xxT❥Y ❪✿

◮ It was shown that m❄mT ❄ ✻✙ E❬xxT❥Y ❪ [Ghorbani, Javadi, and

Montanari, 2017].

◮ The assumption that posterior distribution can be approximately

factorized into the product of marginals is wrong!

Song Mei (Stanford University) TAP free energy September 19, 2018 8 / 29

The TAP free energy

◮ Thouless, Anderson, and Palmer (1977) proposed the TAP free

energy when they study the Sherrington-Kirkpatrick model, whose Gibbs measure gives

• ☞❀✕✭σ✮ ❂

✶ ❩☞❀✕ ❡①♣❢☞❤σ❀ Y σ✐❣✿ where ❨✐❥ ✘ ◆✭✕❂♥❀ ✶❂♥✮.

◮ When ☞ ❂ ✕, the Gibbs measure of SK model is the same as the

posterior of Z✷ synchronization

◮ The TAP free energy (when ☞ ❂ ✕) gives

❋❚❆P✭m✮ ✑

♥

❳

✐❂✶

h✭♠✐✮ ✕ ✷ ❤m❀ Y m✐

⑤ ④③ ⑥

❋▼❋

♥✕✷ ✹

❤

✶ ❦m❦✷

✷

♥

✐✷ ⑤ ④③ ⑥

Onsager’s correction term

✿

Song Mei (Stanford University) TAP free energy September 19, 2018 9 / 29

The TAP free energy

◮ Thouless, Anderson, and Palmer (1977) proposed the TAP free

energy when they study the Sherrington-Kirkpatrick model, whose Gibbs measure gives

• ☞❀✕✭σ✮ ❂

✶ ❩☞❀✕ ❡①♣❢☞❤σ❀ Y σ✐❣✿ where ❨✐❥ ✘ ◆✭✕❂♥❀ ✶❂♥✮.

◮ When ☞ ❂ ✕, the Gibbs measure of SK model is the same as the

posterior of Z✷ synchronization

◮ The TAP free energy (when ☞ ❂ ✕) gives

❋❚❆P✭m✮ ✑

♥

❳

✐❂✶

h✭♠✐✮ ✕ ✷ ❤m❀ Y m✐

⑤ ④③ ⑥

❋▼❋

♥✕✷ ✹

❤

✶ ❦m❦✷

✷

♥

✐✷ ⑤ ④③ ⑥

Onsager’s correction term

✿

Song Mei (Stanford University) TAP free energy September 19, 2018 9 / 29

The TAP free energy

◮ Thouless, Anderson, and Palmer (1977) proposed the TAP free

energy when they study the Sherrington-Kirkpatrick model, whose Gibbs measure gives

• ☞❀✕✭σ✮ ❂

✶ ❩☞❀✕ ❡①♣❢☞❤σ❀ Y σ✐❣✿ where ❨✐❥ ✘ ◆✭✕❂♥❀ ✶❂♥✮.

◮ When ☞ ❂ ✕, the Gibbs measure of SK model is the same as the

posterior of Z✷ synchronization

◮ The TAP free energy (when ☞ ❂ ✕) gives

❋❚❆P✭m✮ ✑

♥

❳

✐❂✶

h✭♠✐✮ ✕ ✷ ❤m❀ Y m✐

⑤ ④③ ⑥

❋▼❋

♥✕✷ ✹

❤

✶ ❦m❦✷

✷

♥

✐✷ ⑤ ④③ ⑥

Onsager’s correction term

✿

Song Mei (Stanford University) TAP free energy September 19, 2018 9 / 29

The TAP free energy

◮ The TAP free energy

❋❚❆P✭m✮ ✑

♥

❳

✐❂✶

h✭♠✐✮ ✕ ✷ ❤m❀ Y m✐

⑤ ④③ ⑥

❋▼❋

♥✕✷ ✹

❤

✶ ❦m❦✷

✷

♥

✐✷ ⑤ ④③ ⑥

Onsager’s correction term

✿

◮ For m❄ ❂ ❛r❣ ♠✐♥m ❋❚❆P✭m✮, we hope

m❄mT

❄ ✙ ❝

X❇❛②❡s ❂ E❬xxT❥Y ❪✿

◮ Our main theorem shows that this is correct.

Song Mei (Stanford University) TAP free energy September 19, 2018 10 / 29

The TAP free energy

◮ The TAP free energy

❋❚❆P✭m✮ ✑

♥

❳

✐❂✶

h✭♠✐✮ ✕ ✷ ❤m❀ Y m✐

⑤ ④③ ⑥

❋▼❋

♥✕✷ ✹

❤

✶ ❦m❦✷

✷

♥

✐✷ ⑤ ④③ ⑥

Onsager’s correction term

✿

◮ For m❄ ❂ ❛r❣ ♠✐♥m ❋❚❆P✭m✮, we hope

m❄mT

❄ ✙ ❝

X❇❛②❡s ❂ E❬xxT❥Y ❪✿

◮ Our main theorem shows that this is correct.

Song Mei (Stanford University) TAP free energy September 19, 2018 10 / 29

The TAP free energy

◮ The TAP free energy

❋❚❆P✭m✮ ✑

♥

❳

✐❂✶

h✭♠✐✮ ✕ ✷ ❤m❀ Y m✐

⑤ ④③ ⑥

❋▼❋

♥✕✷ ✹

❤

✶ ❦m❦✷

✷

♥

✐✷ ⑤ ④③ ⑥

Onsager’s correction term

✿

◮ For m❄ ❂ ❛r❣ ♠✐♥m ❋❚❆P✭m✮, we hope

m❄mT

❄ ✙ ❝

X❇❛②❡s ❂ E❬xxT❥Y ❪✿

◮ Our main theorem shows that this is correct.

Song Mei (Stanford University) TAP free energy September 19, 2018 10 / 29

Proof of the main theorem

Theorem (Fan, M., Montanari, 2018)

Denote ❈✕❀♥ ❂ ❢m ✷ ❬✶❀ ✶❪♥ ✿ r❋❚❆P✭m✮ ❂ 0❀ ❋❚❆P✭m✮ ✔ ✕✷❂✸❣. There exists ✕✵ ❃ ✵, such that for any ✕ ❃ ✕✵, we have ❧✐♠

♥✦✶ E

❤

s✉♣

m✷❈✕❀♥

✶ ♥✷ ❦mmT ❝ X❇❛②❡s❦✷

❋ ❫ ✶

✐

❂ ✵✿ (1) All the critical points (below a threshold) are close to the Bayesian estimator.

Song Mei (Stanford University) TAP free energy September 19, 2018 11 / 29

Proof of the main theorem

Theorem (Fan, M., Montanari, 2018)

Denote ❈✕❀♥ ❂ ❢m ✷ ❬✶❀ ✶❪♥ ✿ r❋❚❆P✭m✮ ❂ 0❀ ❋❚❆P✭m✮ ✔ ✕✷❂✸❣. There exists ✕✵ ❃ ✵, such that for any ✕ ❃ ✕✵, we have ❧✐♠

♥✦✶ E

❤

s✉♣

m✷❈✕❀♥

✶ ♥✷ ❦mmT ❝ X❇❛②❡s❦✷

❋ ❫ ✶

✐

❂ ✵✿ (1) All the critical points (below a threshold) are close to the Bayesian estimator.

Song Mei (Stanford University) TAP free energy September 19, 2018 11 / 29

Relationship with AMP

◮ Another way to construct the Bayes estimator is approximate message

passing [Donoho, Maleki, and Montanari, 2009], [Bolthausen, 2014]: m❦✰✶ ❂ t❛♥❤✭✕Y m❦ ✕✷❬✶ ❦m❦❦✷

✷❂♥❪m❦✶✮ ✿

◮ Fixed point of AMP is a critical point of the TAP free energy. ◮ The risk of AMP iterations converge to the Bayes risk [Deshpande,

Abbes, and Montanari, 2016], [Montanari and Venkataramanan, 2017]: ❧✐♠

❦✦✶ ❧✐♠ ♥✦✶

✶ ♥✷ ❦m❦✭m❦✮T xxT❦✷

❋ ❂ ❧✐♠ ♥✦✶ ▼❙❊♥✭❝

X❇❛②❡s✮✿

◮ But it is not known if AMP will converge to a fixed point (It is still an

• pen problem).

Song Mei (Stanford University) TAP free energy September 19, 2018 12 / 29

Relationship with AMP

◮ Another way to construct the Bayes estimator is approximate message

passing [Donoho, Maleki, and Montanari, 2009], [Bolthausen, 2014]: m❦✰✶ ❂ t❛♥❤✭✕Y m❦ ✕✷❬✶ ❦m❦❦✷

✷❂♥❪m❦✶✮ ✿

◮ Fixed point of AMP is a critical point of the TAP free energy. ◮ The risk of AMP iterations converge to the Bayes risk [Deshpande,

Abbes, and Montanari, 2016], [Montanari and Venkataramanan, 2017]: ❧✐♠

❦✦✶ ❧✐♠ ♥✦✶

✶ ♥✷ ❦m❦✭m❦✮T xxT❦✷

❋ ❂ ❧✐♠ ♥✦✶ ▼❙❊♥✭❝

X❇❛②❡s✮✿

◮ But it is not known if AMP will converge to a fixed point (It is still an

• pen problem).

Song Mei (Stanford University) TAP free energy September 19, 2018 12 / 29

Relationship with AMP

◮ Another way to construct the Bayes estimator is approximate message

passing [Donoho, Maleki, and Montanari, 2009], [Bolthausen, 2014]: m❦✰✶ ❂ t❛♥❤✭✕Y m❦ ✕✷❬✶ ❦m❦❦✷

✷❂♥❪m❦✶✮ ✿

◮ Fixed point of AMP is a critical point of the TAP free energy. ◮ The risk of AMP iterations converge to the Bayes risk [Deshpande,

Abbes, and Montanari, 2016], [Montanari and Venkataramanan, 2017]: ❧✐♠

❦✦✶ ❧✐♠ ♥✦✶

✶ ♥✷ ❦m❦✭m❦✮T xxT❦✷

❋ ❂ ❧✐♠ ♥✦✶ ▼❙❊♥✭❝

X❇❛②❡s✮✿

◮ But it is not known if AMP will converge to a fixed point (It is still an

• pen problem).

Song Mei (Stanford University) TAP free energy September 19, 2018 12 / 29

Relationship with AMP

◮ Another way to construct the Bayes estimator is approximate message

passing [Donoho, Maleki, and Montanari, 2009], [Bolthausen, 2014]: m❦✰✶ ❂ t❛♥❤✭✕Y m❦ ✕✷❬✶ ❦m❦❦✷

✷❂♥❪m❦✶✮ ✿

◮ Fixed point of AMP is a critical point of the TAP free energy. ◮ The risk of AMP iterations converge to the Bayes risk [Deshpande,

Abbes, and Montanari, 2016], [Montanari and Venkataramanan, 2017]: ❧✐♠

❦✦✶ ❧✐♠ ♥✦✶

✶ ♥✷ ❦m❦✭m❦✮T xxT❦✷

❋ ❂ ❧✐♠ ♥✦✶ ▼❙❊♥✭❝

X❇❛②❡s✮✿

◮ But it is not known if AMP will converge to a fixed point (It is still an

• pen problem).

Song Mei (Stanford University) TAP free energy September 19, 2018 12 / 29

Related literatures in spin glass theory

TAP free energy in unbiased SK.

◮ TAP equations: [Talagrand, 2004], [Chatterjee, 2009], [Chen, 2011],

[Auffinger and Jagannath, 2016], Posterior means/Pure states satisfy TAP equations.

◮ TAP free energy: [Chen and Panchenko, 2017], constrained TAP

minimum are exact. Calculating the complexity.

◮

[Auffinger, Ben Arous, and Cerny, 2010], [Subag, 2016].

Song Mei (Stanford University) TAP free energy September 19, 2018 13 / 29

Proof of the main theorem

Theorem (Fan, M., Montanari, 2018)

Denote ❈✕❀♥ ❂ ❢m ✷ ❬✶❀ ✶❪♥ ✿ r❋❚❆P✭m✮ ❂ 0❀ ❋❚❆P✭m✮ ✔ ✕✷❂✸❣. There exists ✕✵ ❃ ✵, such that for any ✕ ❃ ✕✵, we have ❧✐♠

♥✦✶ E

❤

s✉♣

m✷❈✕❀♥

✶ ♥✷ ❦mmT ❝ X❇❛②❡s❦✷

❋ ❫ ✶

✐

❂ ✵✿ (1) All the critical points (below a threshold) are close to the Bayesian estimator.

Song Mei (Stanford University) TAP free energy September 19, 2018 14 / 29

Proof idea - Count the number of critical points

◮ Recall

❋❚❆P✭m✮ ✑

♥

❳

✐❂✶

h✭♠✐✮ ✕ ✷ ❤m❀ Y m✐ ♥✕✷ ✹ ❤ ✶ ❦m❦✷

✷

♥ ✐✷ ✿

◮ Define some important statistics of m:

❊✭m✮ ❂ ❋❚❆P✭m✮❂♥❀ ◗✭m✮ ❂ ❦m❦✷

✷❂♥❀

▼✭m✮ ❂ ❤m❀ x✐❂♥✿

◮ For any ❯ ✒ R✸, define

❈r✐t♥✭❯✮ ✑ ★❢m ✿ r❊✭m✮ ❂ 0❀ ✭◗✭m✮❀ ▼✭m✮❀ ❊✭m✮✮ ✷ ❯❣✿ (2)

Proposition

E❬❈r✐t♥✭❯✮❪ ✔ ❡①♣

♥

♥ s✉♣

✭q❀✬❀❡✮✷❯

❙❄✭q❀ ✬❀ ❡✮ ✰ ♦✭♥✮

♦

✿

Song Mei (Stanford University) TAP free energy September 19, 2018 15 / 29

Proof idea - Count the number of critical points

❙❄✭q❀ ✬❀ ❡✮ ❂ s✉♣

❛✷R

✐♥❢

✭✖❀✗❀✜❀✌✮✷R✹ ❙✭q❀ ✬❀ ❛❀ ❡❀ ✖❀ ✗❀ ✜❀ ✌✮❀

where ❙✭q❀ ✬❀ ❛❀ ❡❀ ✖❀ ✗❀ ✜❀ ✌✮ ❂ ✶ ✹☞✷ ❤❛ q ☞✕✬✷ q ☞✷✭✶ q✮ ✐✷ q✖ ✬✗ ❛✜ ❤ ☞✷ ✹ ✭✶ q✷✮ ✰ ❛ ✷ ❡ ✐ ✌ ✰ ❧♦❣ ■❀ and ■ ❂ ❩ ✶

✶

✶ ✭✷✙☞✷q✮✶❂✷ ❡①♣ ♥ ✭① ☞✕✬✮✷ ✷☞✷q ✰ ✖ t❛♥❤✷✭①✮ ✰ ✗ t❛♥❤✭①✮ ✰ ✜① t❛♥❤✭①✮ ✰ ✌ ❧♦❣❬✷ ❝♦s❤✭①✮❪ ♦ ❞①✿

Song Mei (Stanford University) TAP free energy September 19, 2018 16 / 29

Proof idea - Count the number of critical points

◮ Key proposition: for ❯ ✒ R✸,

E❬❈r✐t♥✭❯✮❪ ✔ ❡①♣

♥

♥

❚✭❯✮

③ ⑥⑤ ④

s✉♣

✭q❀✬❀❡✮✷❯

❙❄✭q❀ ✬❀ ❡✮ ✰♦✭♥✮

♦

❀

◮ For any ❯ such that ❚✭❯✮ ❃ ✵, there could potentially be critical

points of ❋❚❆P in ❯.

◮ For any ❯ such that ❚✭❯✮ ❁ ✵, there is no critical points of ❋❚❆P

in ❯, with high probability.

◮ If we admit the key proposition, suffice to show that ❚✭❯✮ ❁ ✵

unless ❯ contains a neighborhood of the Bayes estimator.

Song Mei (Stanford University) TAP free energy September 19, 2018 17 / 29

Proof idea - Count the number of critical points

◮ Key proposition: for ❯ ✒ R✸,

E❬❈r✐t♥✭❯✮❪ ✔ ❡①♣

♥

♥

❚✭❯✮

③ ⑥⑤ ④

s✉♣

✭q❀✬❀❡✮✷❯

❙❄✭q❀ ✬❀ ❡✮ ✰♦✭♥✮

♦

❀

◮ For any ❯ such that ❚✭❯✮ ❃ ✵, there could potentially be critical

points of ❋❚❆P in ❯.

◮ For any ❯ such that ❚✭❯✮ ❁ ✵, there is no critical points of ❋❚❆P

in ❯, with high probability.

◮ If we admit the key proposition, suffice to show that ❚✭❯✮ ❁ ✵

unless ❯ contains a neighborhood of the Bayes estimator.

Song Mei (Stanford University) TAP free energy September 19, 2018 17 / 29

Proof idea - Count the number of critical points

◮ Key proposition: for ❯ ✒ R✸,

E❬❈r✐t♥✭❯✮❪ ✔ ❡①♣

♥

♥

❚✭❯✮

③ ⑥⑤ ④

s✉♣

✭q❀✬❀❡✮✷❯

❙❄✭q❀ ✬❀ ❡✮ ✰♦✭♥✮

♦

❀

◮ For any ❯ such that ❚✭❯✮ ❃ ✵, there could potentially be critical

points of ❋❚❆P in ❯.

◮ For any ❯ such that ❚✭❯✮ ❁ ✵, there is no critical points of ❋❚❆P

in ❯, with high probability.

◮ If we admit the key proposition, suffice to show that ❚✭❯✮ ❁ ✵

unless ❯ contains a neighborhood of the Bayes estimator.

Song Mei (Stanford University) TAP free energy September 19, 2018 17 / 29

Proof idea - Count the number of critical points

◮ Key proposition: for ❯ ✒ R✸,

E❬❈r✐t♥✭❯✮❪ ✔ ❡①♣

♥

♥

❚✭❯✮

③ ⑥⑤ ④

s✉♣

✭q❀✬❀❡✮✷❯

❙❄✭q❀ ✬❀ ❡✮ ✰♦✭♥✮

♦

❀

◮ For any ❯ such that ❚✭❯✮ ❃ ✵, there could potentially be critical

points of ❋❚❆P in ❯.

◮ For any ❯ such that ❚✭❯✮ ❁ ✵, there is no critical points of ❋❚❆P

in ❯, with high probability.

◮ If we admit the key proposition, suffice to show that ❚✭❯✮ ❁ ✵

unless ❯ contains a neighborhood of the Bayes estimator.

Song Mei (Stanford University) TAP free energy September 19, 2018 17 / 29

Proof idea - the complexity function ❙❄

◮ ❙❄✭❡✮ ❂ s✉♣q❀✬ ❙❄✭q❀ ✬❀ ❡✮.

E

• 2.3
• 2.2
• 2.1
• 2
• 1.9
• 1.8
• 1.7
• 1.6
• 1.5

Complexity

• 0.07
• 0.06
• 0.05
• 0.04
• 0.03
• 0.02
• 0.01

0.01 0.02 0.03 lambda = 2

◮ At ❡❄, ❙❄✭❡❄✮ ❂ ✵.

Song Mei (Stanford University) TAP free energy September 19, 2018 18 / 29

Proof idea - the complexity function ❙❄

◮ ❙❄✭✬✮ ❂ s✉♣q❀❡ ❙❄✭q❀ ✬❀ ❡✮.

M 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Complexity

• 0.07
• 0.06
• 0.05
• 0.04
• 0.03
• 0.02
• 0.01

0.01 0.02 0.03 lambda = 2

◮ At ✬❄, ❙❄✭✬❄✮ ❂ ✵.

Song Mei (Stanford University) TAP free energy September 19, 2018 19 / 29

Proof idea - the complexity function ❙❄

◮ ❙❄✭q✮ ❂ s✉♣✬❀❡ ❙❄✭q❀ ✬❀ ❡✮.

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Q

• 0.1
• 0.08
• 0.06
• 0.04
• 0.02

0.02 0.04 0.06 0.08 0.1 Complexity lambda = 2

◮ At q❄, ❙❄✭q❄✮ ❂ ✵.

Song Mei (Stanford University) TAP free energy September 19, 2018 20 / 29

Proof idea - the complexity function ❙❄

There exists ✕✵, for ✕ ✕ ✕✵,

◮ ❙❄✭q❄❀ ✬❄❀ ❡❄✮ ❂ ✵, where ✭q❄❀ ✬❄❀ ❡❄✮ ✙ ✭◗✭m❄✮❀ ▼✭m❄✮❀ ❊✭m❄✮✮

for ❝ X❇❛②❡s ✙ m❄mT

❄ . ◮ ❙❄✭q❀ ✬❀ ❡✮ ❁ ✵ for any ❡ ✔ ✕✷❂✸ and ✭q❀ ✬❀ ❡✮ ✻❂ ✭q❄❀ ✬❄❀ ❡❄✮.

The proof of these two properties is more than calculus. It requires bounds using concentration inequalities. Combining with the key inequality it is easy to show the main theorem. E❬❈r✐t♥✭❯✮❪ ✔ ❡①♣

♥

♥ s✉♣

✭q❀✬❀❡✮✷❯

❙❄✭q❀ ✬❀ ❡✮ ✰ ♦✭♥✮

♦

✿ Now suffice to show the key inequality.

Song Mei (Stanford University) TAP free energy September 19, 2018 21 / 29

Proof idea - the complexity function ❙❄

There exists ✕✵, for ✕ ✕ ✕✵,

◮ ❙❄✭q❄❀ ✬❄❀ ❡❄✮ ❂ ✵, where ✭q❄❀ ✬❄❀ ❡❄✮ ✙ ✭◗✭m❄✮❀ ▼✭m❄✮❀ ❊✭m❄✮✮

for ❝ X❇❛②❡s ✙ m❄mT

❄ . ◮ ❙❄✭q❀ ✬❀ ❡✮ ❁ ✵ for any ❡ ✔ ✕✷❂✸ and ✭q❀ ✬❀ ❡✮ ✻❂ ✭q❄❀ ✬❄❀ ❡❄✮.

The proof of these two properties is more than calculus. It requires bounds using concentration inequalities. Combining with the key inequality it is easy to show the main theorem. E❬❈r✐t♥✭❯✮❪ ✔ ❡①♣

♥

♥ s✉♣

✭q❀✬❀❡✮✷❯

❙❄✭q❀ ✬❀ ❡✮ ✰ ♦✭♥✮

♦

✿ Now suffice to show the key inequality.

Song Mei (Stanford University) TAP free energy September 19, 2018 21 / 29

Proof idea - the complexity function ❙❄

There exists ✕✵, for ✕ ✕ ✕✵,

◮ ❙❄✭q❄❀ ✬❄❀ ❡❄✮ ❂ ✵, where ✭q❄❀ ✬❄❀ ❡❄✮ ✙ ✭◗✭m❄✮❀ ▼✭m❄✮❀ ❊✭m❄✮✮

for ❝ X❇❛②❡s ✙ m❄mT

❄ . ◮ ❙❄✭q❀ ✬❀ ❡✮ ❁ ✵ for any ❡ ✔ ✕✷❂✸ and ✭q❀ ✬❀ ❡✮ ✻❂ ✭q❄❀ ✬❄❀ ❡❄✮.

The proof of these two properties is more than calculus. It requires bounds using concentration inequalities. Combining with the key inequality it is easy to show the main theorem. E❬❈r✐t♥✭❯✮❪ ✔ ❡①♣

♥

♥ s✉♣

✭q❀✬❀❡✮✷❯

❙❄✭q❀ ✬❀ ❡✮ ✰ ♦✭♥✮

♦

✿ Now suffice to show the key inequality.

Song Mei (Stanford University) TAP free energy September 19, 2018 21 / 29

Calculating the ❈r✐t: Kac-Rice formula

Lemma (Kac-Rice formula, c.f. [Adler and Taylor, 2007)

] Let ❢ ✿ R❞ ✦ R be a “sufficiently regular” random morse function. Let ♣m✭z✮ be the density of r❢✭m✮ at z. For any Borel measurable set ❚ ✒ R❞, denote ❈r✐t✭❚✮ ❂ ★❢m ✷ ❚ ✿ r❢✭m✮ ❂ 0❣✿ Then E❬❈r✐t✭❚✮❪ ❂E ❤ ❩

❚

☞ ☞ ❞❡t r✷❢✭m✮ ☞ ☞ ✁ ✍✭r❢✭m✮✮ ✁ dm ✐ ❂ ❩

❚

E ❤☞ ☞ ❞❡t r✷❢✭m✮ ☞ ☞ ☞ ☞ ☞r❢✭m✮ ❂ 0 ✐ ♣m✭0✮dm✿

◮ ❥ ❞❡t r✷❢✭m✮❥ is the correct weight function so that each critical point

count exactly once.

Song Mei (Stanford University) TAP free energy September 19, 2018 22 / 29

Dealing with determinant of Hessian

◮ The conditional Hessian is distributed as (up to some scaling)

❬r✷❋❚❆P✭m✮❥r❋❚❆P✭m✮ ❂ 0❪ ❞ ❂ D ✰ W ✰ low rank perturbation❀ where D ❂ diag✭❞✐✮, and W ✘ ●❖❊✭♥✮.

◮ The low rank perturbation has vanishing effects. Therefore, we

just need to calculate E❬❥ ❞❡t✭H✮❥❪, with H ❂ D ✰ W ✿

Song Mei (Stanford University) TAP free energy September 19, 2018 23 / 29

Dealing with determinant of Hessian

◮ The conditional Hessian is distributed as (up to some scaling)

❬r✷❋❚❆P✭m✮❥r❋❚❆P✭m✮ ❂ 0❪ ❞ ❂ D ✰ W ✰ low rank perturbation❀ where D ❂ diag✭❞✐✮, and W ✘ ●❖❊✭♥✮.

◮ The low rank perturbation has vanishing effects. Therefore, we

just need to calculate E❬❥ ❞❡t✭H✮❥❪, with H ❂ D ✰ W ✿

Song Mei (Stanford University) TAP free energy September 19, 2018 23 / 29

Dealing with determinant of Hessian

H ❂ D ✰ W ❂ ❞✐❛❣♦♥❛❧ ✰ ●❖❊✿ ✶ ♥ ❧♦❣ E❬❥ ❞❡t✭H✮❥❪ ❂ ✶ ♥ ❧♦❣ E

❤ ♥ ❨

✐❂✶

❥✕✐✭H✮❥

✐

✙ ✶ ♥ ❧♦❣

❤ ♥ ❨

✐❂✶

❥✕✐✭H✮❥

✐

❂ ✶ ♥

♥

❳

✐❂✶

❧♦❣ ❥✕✐✭H✮❥ ❂

❩

R

❧♦❣ ❥①❥ ✁ ✖H✭d①✮ ✙ E

❤ ❩

R

❧♦❣ ❥①❥ ✁ ✖H✭d①✮

✐

✿ where ✖H ❂ ✭✶❂♥✮ P♥

✐❂✶ ✍✭✕✐✭H✮✮. ◮ Approximate equalities are due to concentration. ◮ The Stieltjes transform of ✖H can be approximately calculated

using free probability theory.

◮ Once the Stieltjes transform of ✖H is known, the quantity

E

❤ ❘

R✭❧♦❣ ❥①❥✮✖H✭d①✮

✐

can be computed.

Song Mei (Stanford University) TAP free energy September 19, 2018 24 / 29

Dealing with determinant of Hessian

H ❂ D ✰ W ❂ ❞✐❛❣♦♥❛❧ ✰ ●❖❊✿ ✶ ♥ ❧♦❣ E❬❥ ❞❡t✭H✮❥❪ ❂ ✶ ♥ ❧♦❣ E

❤ ♥ ❨

✐❂✶

❥✕✐✭H✮❥

✐

✙ ✶ ♥ ❧♦❣

❤ ♥ ❨

✐❂✶

❥✕✐✭H✮❥

✐

❂ ✶ ♥

♥

❳

✐❂✶

❧♦❣ ❥✕✐✭H✮❥ ❂

❩

R

❧♦❣ ❥①❥ ✁ ✖H✭d①✮ ✙ E

❤ ❩

R

❧♦❣ ❥①❥ ✁ ✖H✭d①✮

✐

✿ where ✖H ❂ ✭✶❂♥✮ P♥

✐❂✶ ✍✭✕✐✭H✮✮. ◮ Approximate equalities are due to concentration. ◮ The Stieltjes transform of ✖H can be approximately calculated

using free probability theory.

◮ Once the Stieltjes transform of ✖H is known, the quantity

E

❤ ❘

R✭❧♦❣ ❥①❥✮✖H✭d①✮

✐

can be computed.

Song Mei (Stanford University) TAP free energy September 19, 2018 24 / 29

Free convolution of two distribution

Let A ✷ R♥✂♥, and ✖A ❂ ✭✶❂♥✮ P♥

✐❂✶ ✍✭✕✐✭A✮✮. For any ③ ✷ C✰, the

Stieltjes transform of ✖A is defined as ❣A✭③✮ ❂

❩

R

✶ ① ③ ✖A✭d①✮ ❂ ✶ ♥

♥

❳

✐❂✶

✶ ✕✐✭A✮ ③ ✿

Lemma (Due to free probability theory)

Let D ❂ diag✭❞✐✮ be a diagonal matrix, and let H ❂ D ✰ W . Then E❣H✭③✮ ❂ ✶ ♥

♥

❳

✐❂✶

✶ ❞✐ ③ E❣H✭③✮ ✰ ♦♥✭✶✮✿ (3)

Song Mei (Stanford University) TAP free energy September 19, 2018 25 / 29

Free convolution of two distribution

Let A ✷ R♥✂♥, and ✖A ❂ ✭✶❂♥✮ P♥

✐❂✶ ✍✭✕✐✭A✮✮. For any ③ ✷ C✰, the

Stieltjes transform of ✖A is defined as ❣A✭③✮ ❂

❩

R

✶ ① ③ ✖A✭d①✮ ❂ ✶ ♥

♥

❳

✐❂✶

✶ ✕✐✭A✮ ③ ✿

Lemma (Due to free probability theory)

Let D ❂ diag✭❞✐✮ be a diagonal matrix, and let H ❂ D ✰ W . Then E❣H✭③✮ ❂ ✶ ♥

♥

❳

✐❂✶

✶ ❞✐ ③ E❣H✭③✮ ✰ ♦♥✭✶✮✿ (3)

Song Mei (Stanford University) TAP free energy September 19, 2018 25 / 29

Dealing with determinant of Hessian

H ❂ D ✰ W ❂ ❞✐❛❣♦♥❛❧ ✰ ●❖❊✿ ✶ ♥ ❧♦❣ E❬❥ ❞❡t✭H✮❥❪ ❂ ✶ ♥ ❧♦❣ E

❤ ♥ ❨

✐❂✶

❥✕✐✭H✮❥

✐

✙ ✶ ♥ ❧♦❣

❤ ♥ ❨

✐❂✶

❥✕✐✭H✮❥

✐

❂ ✶ ♥

♥

❳

✐❂✶

❧♦❣ ❥✕✐✭H✮❥ ❂

❩

R

❧♦❣ ❥①❥ ✁ ✖H✭d①✮ ✙ E

❤ ❩

R

❧♦❣ ❥①❥ ✁ ✖H✭d①✮

✐

✿ where ✖H ❂ ✭✶❂♥✮ P♥

✐❂✶ ✍✭✕✐✭H✮✮. ◮ Approximate equalities are due to concentration. ◮ The Stieltjes transform of ✖H can be approximately calculated

using free probability theory.

◮ Once the Stieltjes transform of ✖H is known, the quantity

E

❤ ❘

R✭❧♦❣ ❥①❥✮✖H✭d①✮

✐

can be computed.

Song Mei (Stanford University) TAP free energy September 19, 2018 26 / 29

Calculate E❬

❘ R ❧♦❣ ❥①❥ ✁ ✖H✭d①✮❪

◮ Define

❇✭t✮ ❂ E

❩

R

❧♦❣✭① it✮✖H✭d①✮✿

◮ We have

❁❇✭✵✰✮ ❂E

❩

R

❧♦❣ ❥①❥ ✁ ✖H✭d①✮❀ ❇✵✭t✮ ❂ iE

❩

R

❬✶❂✭① it✮❪✖H✭d①✮ ❂ iE❬❣H✭it✮❪✿

◮ We guess a formula

⑦ ❇✭t✮ ❂ ✶ ♥

♥

❳

✐❂✶

❧♦❣✭❞✐ it E❣H✭it✮✮ ✰ ✶ ✷❬E❣H✭it✮❪✷✿ Then ⑦ ❇✭t✮ satisfy all the conditions that ❇✭t✮ approximately satisfy, so that ⑦ ❇✭t✮ ❂ ❇✭t✮ ✰ ♦♥✭✶✮.

◮ Hence

✶ ♥ ❧♦❣ E❬❥ ❞❡t✭H✮❥❪ ❂ ⑦ ❇✭✵✮ ✰ ♦♥✭✶✮✿

Song Mei (Stanford University) TAP free energy September 19, 2018 27 / 29

Calculate E❬

❘ R ❧♦❣ ❥①❥ ✁ ✖H✭d①✮❪

◮ Define

❇✭t✮ ❂ E

❩

R

❧♦❣✭① it✮✖H✭d①✮✿

◮ We have

❁❇✭✵✰✮ ❂E

❩

R

❧♦❣ ❥①❥ ✁ ✖H✭d①✮❀ ❇✵✭t✮ ❂ iE

❩

R

❬✶❂✭① it✮❪✖H✭d①✮ ❂ iE❬❣H✭it✮❪✿

◮ We guess a formula

⑦ ❇✭t✮ ❂ ✶ ♥

♥

❳

✐❂✶

❧♦❣✭❞✐ it E❣H✭it✮✮ ✰ ✶ ✷❬E❣H✭it✮❪✷✿ Then ⑦ ❇✭t✮ satisfy all the conditions that ❇✭t✮ approximately satisfy, so that ⑦ ❇✭t✮ ❂ ❇✭t✮ ✰ ♦♥✭✶✮.

◮ Hence

✶ ♥ ❧♦❣ E❬❥ ❞❡t✭H✮❥❪ ❂ ⑦ ❇✭✵✮ ✰ ♦♥✭✶✮✿

Song Mei (Stanford University) TAP free energy September 19, 2018 27 / 29

Calculate E❬

❘ R ❧♦❣ ❥①❥ ✁ ✖H✭d①✮❪

◮ Define

❇✭t✮ ❂ E

❩

R

❧♦❣✭① it✮✖H✭d①✮✿

◮ We have

❁❇✭✵✰✮ ❂E

❩

R

❧♦❣ ❥①❥ ✁ ✖H✭d①✮❀ ❇✵✭t✮ ❂ iE

❩

R

❬✶❂✭① it✮❪✖H✭d①✮ ❂ iE❬❣H✭it✮❪✿

◮ We guess a formula

⑦ ❇✭t✮ ❂ ✶ ♥

♥

❳

✐❂✶

❧♦❣✭❞✐ it E❣H✭it✮✮ ✰ ✶ ✷❬E❣H✭it✮❪✷✿ Then ⑦ ❇✭t✮ satisfy all the conditions that ❇✭t✮ approximately satisfy, so that ⑦ ❇✭t✮ ❂ ❇✭t✮ ✰ ♦♥✭✶✮.

◮ Hence

✶ ♥ ❧♦❣ E❬❥ ❞❡t✭H✮❥❪ ❂ ⑦ ❇✭✵✮ ✰ ♦♥✭✶✮✿

Song Mei (Stanford University) TAP free energy September 19, 2018 27 / 29

Calculate E❬

❘ R ❧♦❣ ❥①❥ ✁ ✖H✭d①✮❪

◮ Define

❇✭t✮ ❂ E

❩

R

❧♦❣✭① it✮✖H✭d①✮✿

◮ We have

❁❇✭✵✰✮ ❂E

❩

R

❧♦❣ ❥①❥ ✁ ✖H✭d①✮❀ ❇✵✭t✮ ❂ iE

❩

R

❬✶❂✭① it✮❪✖H✭d①✮ ❂ iE❬❣H✭it✮❪✿

◮ We guess a formula

⑦ ❇✭t✮ ❂ ✶ ♥

♥

❳

✐❂✶

❧♦❣✭❞✐ it E❣H✭it✮✮ ✰ ✶ ✷❬E❣H✭it✮❪✷✿ Then ⑦ ❇✭t✮ satisfy all the conditions that ❇✭t✮ approximately satisfy, so that ⑦ ❇✭t✮ ❂ ❇✭t✮ ✰ ♦♥✭✶✮.

◮ Hence

✶ ♥ ❧♦❣ E❬❥ ❞❡t✭H✮❥❪ ❂ ⑦ ❇✭✵✮ ✰ ♦♥✭✶✮✿

Song Mei (Stanford University) TAP free energy September 19, 2018 27 / 29

Dealing with determinant of Hessian

H ❂ D ✰ W ❂ ❞✐❛❣♦♥❛❧ ✰ ●❖❊✿ ✶ ♥ ❧♦❣ E❬❥ ❞❡t✭H✮❥❪ ❂ ✶ ♥ ❧♦❣ E

❤ ♥ ❨

✐❂✶

❥✕✐✭H✮❥

✐

✙ ✶ ♥ ❧♦❣

❤ ♥ ❨

✐❂✶

❥✕✐✭H✮❥

✐

❂ ✶ ♥

♥

❳

✐❂✶

❧♦❣ ❥✕✐✭H✮❥ ❂

❩

R

❧♦❣ ❥①❥ ✁ ✖H✭d①✮ ✙ E

❤ ❩

R

❧♦❣ ❥①❥ ✁ ✖H✭d①✮

✐

✿ where ✖H ❂ ✭✶❂♥✮ P♥

✐❂✶ ✍✭✕✐✭H✮✮. ◮ Approximate equalities are due to concentration. ◮ The Stieltjes transform of ✖H can be approximately calculated

using free probability theory.

◮ Once the Stieltjes transform of ✖H is known, the quantity

E

❤ ❘

R✭❧♦❣ ❥①❥✮✖H✭d①✮

✐

can be computed.

Song Mei (Stanford University) TAP free energy September 19, 2018 28 / 29

Summary

◮ TAP free energy is accurate for Z✷ synchronization. ◮ Can be generalized to topic modeling, low rank matrix estimation,

compressed sensing, etc...

◮ It is interesting to study and apply variational inference beyond

mean field.

Song Mei (Stanford University) TAP free energy September 19, 2018 29 / 29

Summary

◮ TAP free energy is accurate for Z✷ synchronization. ◮ Can be generalized to topic modeling, low rank matrix estimation,

compressed sensing, etc...

◮ It is interesting to study and apply variational inference beyond

mean field.

Song Mei (Stanford University) TAP free energy September 19, 2018 29 / 29

Summary

◮ TAP free energy is accurate for Z✷ synchronization. ◮ Can be generalized to topic modeling, low rank matrix estimation,

compressed sensing, etc...

◮ It is interesting to study and apply variational inference beyond

mean field.

Song Mei (Stanford University) TAP free energy September 19, 2018 29 / 29