variational inference spin glasses and tap free energy
play

Variational inference, spin glasses, and TAP free energy Song Mei - PowerPoint PPT Presentation

Sep 12, 2023 •271 likes •995 views

Variational inference, spin glasses, and TAP free energy Song Mei Stanford University September 19, 2018 Joint work with Zhou Fan and Andrea Montanari Song Mei (Stanford University) TAP free energy September 19, 2018 1 / 29 General

  1. Variational inference, spin glasses, and TAP free energy Song Mei Stanford University September 19, 2018 Joint work with Zhou Fan and Andrea Montanari Song Mei (Stanford University) TAP free energy September 19, 2018 1 / 29

  2. General motivation ◮ Bayesian inference: high dimensional integration is hard! ◮ Variational inference: integration/summation ✦ optimization. A popular objective function: “mean field free energy”. ◮ Applications: topic modeling, stochastic block model, low rank matrix estimation, compressed sensing.... ... within which “MF free energy” is known to be not optimal. ◮ Today: introduce the optimal objective “TAP free energy”, and provide rigorous results. Song Mei (Stanford University) TAP free energy September 19, 2018 2 / 29

  3. General motivation ◮ Bayesian inference: high dimensional integration is hard! ◮ Variational inference: integration/summation ✦ optimization. A popular objective function: “mean field free energy”. ◮ Applications: topic modeling, stochastic block model, low rank matrix estimation, compressed sensing.... ... within which “MF free energy” is known to be not optimal. ◮ Today: introduce the optimal objective “TAP free energy”, and provide rigorous results. Song Mei (Stanford University) TAP free energy September 19, 2018 2 / 29

  4. General motivation ◮ Bayesian inference: high dimensional integration is hard! ◮ Variational inference: integration/summation ✦ optimization. A popular objective function: “mean field free energy”. ◮ Applications: topic modeling, stochastic block model, low rank matrix estimation, compressed sensing.... ... within which “MF free energy” is known to be not optimal. ◮ Today: introduce the optimal objective “TAP free energy”, and provide rigorous results. Song Mei (Stanford University) TAP free energy September 19, 2018 2 / 29

  5. General motivation ◮ Bayesian inference: high dimensional integration is hard! ◮ Variational inference: integration/summation ✦ optimization. A popular objective function: “mean field free energy”. ◮ Applications: topic modeling, stochastic block model, low rank matrix estimation, compressed sensing.... ... within which “MF free energy” is known to be not optimal. ◮ Today: introduce the optimal objective “TAP free energy”, and provide rigorous results. Song Mei (Stanford University) TAP free energy September 19, 2018 2 / 29

  6. General motivation ◮ Bayesian inference: high dimensional integration is hard! ◮ Variational inference: integration/summation ✦ optimization. A popular objective function: “mean field free energy”. ◮ Applications: topic modeling, stochastic block model, low rank matrix estimation, compressed sensing.... ... within which “MF free energy” is known to be not optimal. ◮ Today: introduce the optimal objective “TAP free energy”, and provide rigorous results. Song Mei (Stanford University) TAP free energy September 19, 2018 2 / 29

  7. General motivation ◮ Bayesian inference: high dimensional integration is hard! ◮ Variational inference: integration/summation ✦ optimization. A popular objective function: “mean field free energy”. ◮ Applications: topic modeling, stochastic block model, low rank matrix estimation, compressed sensing.... ... within which “MF free energy” is known to be not optimal. ◮ Today: introduce the optimal objective “TAP free energy”, and provide rigorous results. Song Mei (Stanford University) TAP free energy September 19, 2018 2 / 29

  8. Z ✷ synchronization ◮ Signal: x ❂ ❬ ① ✶ ❀ ✿ ✿ ✿ ❀ ① ♥ ❪ T ✷ Z ♥ ✐✿✐✿❞✿ ✷ ❀ ① ✐ ✘ ❯♥✐❢✭ Z ✷ ✮ ❀ Z ✷ ❂ ❢ ✰✶ ❀ � ✶ ❣ ✿ ◮ Observation: for ✶ ✔ ✐ ❁ ❥ ✔ ♥ ❨ ✐❥ ❂ ✕ ♥① ✐ ① ❥ ✰ ❲ ✐❥ ✿ ◮ Noise ❲ ✐❥ ✘ ◆ ✭✵ ❀ ✶ ❂♥ ✮ . ◮ SNR ✕ ✷ ❬✵ ❀ ✶ ✮ fixed, dimension ♥ ✦ ✶ . ◮ In matrix notation: Y ❂ ✕ ♥ xx T ✰ W ✿ ◮ Task: given Y ❂ ✭ ❨ ✐❥ ✮ , estimate x (or say X ❂ xx T ). Song Mei (Stanford University) TAP free energy September 19, 2018 3 / 29

  9. Z ✷ synchronization ◮ Signal: x ❂ ❬ ① ✶ ❀ ✿ ✿ ✿ ❀ ① ♥ ❪ T ✷ Z ♥ ✐✿✐✿❞✿ ✷ ❀ ① ✐ ✘ ❯♥✐❢✭ Z ✷ ✮ ❀ Z ✷ ❂ ❢ ✰✶ ❀ � ✶ ❣ ✿ ◮ Observation: for ✶ ✔ ✐ ❁ ❥ ✔ ♥ ❨ ✐❥ ❂ ✕ ♥① ✐ ① ❥ ✰ ❲ ✐❥ ✿ ◮ Noise ❲ ✐❥ ✘ ◆ ✭✵ ❀ ✶ ❂♥ ✮ . ◮ SNR ✕ ✷ ❬✵ ❀ ✶ ✮ fixed, dimension ♥ ✦ ✶ . ◮ In matrix notation: Y ❂ ✕ ♥ xx T ✰ W ✿ ◮ Task: given Y ❂ ✭ ❨ ✐❥ ✮ , estimate x (or say X ❂ xx T ). Song Mei (Stanford University) TAP free energy September 19, 2018 3 / 29

  10. Z ✷ synchronization ◮ Signal: x ❂ ❬ ① ✶ ❀ ✿ ✿ ✿ ❀ ① ♥ ❪ T ✷ Z ♥ ✐✿✐✿❞✿ ✷ ❀ ① ✐ ✘ ❯♥✐❢✭ Z ✷ ✮ ❀ Z ✷ ❂ ❢ ✰✶ ❀ � ✶ ❣ ✿ ◮ Observation: for ✶ ✔ ✐ ❁ ❥ ✔ ♥ ❨ ✐❥ ❂ ✕ ♥① ✐ ① ❥ ✰ ❲ ✐❥ ✿ ◮ Noise ❲ ✐❥ ✘ ◆ ✭✵ ❀ ✶ ❂♥ ✮ . ◮ SNR ✕ ✷ ❬✵ ❀ ✶ ✮ fixed, dimension ♥ ✦ ✶ . ◮ In matrix notation: Y ❂ ✕ ♥ xx T ✰ W ✿ ◮ Task: given Y ❂ ✭ ❨ ✐❥ ✮ , estimate x (or say X ❂ xx T ). Song Mei (Stanford University) TAP free energy September 19, 2018 3 / 29

  11. Z ✷ synchronization ◮ Signal: x ❂ ❬ ① ✶ ❀ ✿ ✿ ✿ ❀ ① ♥ ❪ T ✷ Z ♥ ✐✿✐✿❞✿ ✷ ❀ ① ✐ ✘ ❯♥✐❢✭ Z ✷ ✮ ❀ Z ✷ ❂ ❢ ✰✶ ❀ � ✶ ❣ ✿ ◮ Observation: for ✶ ✔ ✐ ❁ ❥ ✔ ♥ ❨ ✐❥ ❂ ✕ ♥① ✐ ① ❥ ✰ ❲ ✐❥ ✿ ◮ Noise ❲ ✐❥ ✘ ◆ ✭✵ ❀ ✶ ❂♥ ✮ . ◮ SNR ✕ ✷ ❬✵ ❀ ✶ ✮ fixed, dimension ♥ ✦ ✶ . ◮ In matrix notation: Y ❂ ✕ ♥ xx T ✰ W ✿ ◮ Task: given Y ❂ ✭ ❨ ✐❥ ✮ , estimate x (or say X ❂ xx T ). Song Mei (Stanford University) TAP free energy September 19, 2018 3 / 29

  12. Z ✷ synchronization ◮ Signal: x ❂ ❬ ① ✶ ❀ ✿ ✿ ✿ ❀ ① ♥ ❪ T ✷ Z ♥ ✐✿✐✿❞✿ ✷ ❀ ① ✐ ✘ ❯♥✐❢✭ Z ✷ ✮ ❀ Z ✷ ❂ ❢ ✰✶ ❀ � ✶ ❣ ✿ ◮ Observation: for ✶ ✔ ✐ ❁ ❥ ✔ ♥ ❨ ✐❥ ❂ ✕ ♥① ✐ ① ❥ ✰ ❲ ✐❥ ✿ ◮ Noise ❲ ✐❥ ✘ ◆ ✭✵ ❀ ✶ ❂♥ ✮ . ◮ SNR ✕ ✷ ❬✵ ❀ ✶ ✮ fixed, dimension ♥ ✦ ✶ . ◮ In matrix notation: Y ❂ ✕ ♥ xx T ✰ W ✿ ◮ Task: given Y ❂ ✭ ❨ ✐❥ ✮ , estimate x (or say X ❂ xx T ). Song Mei (Stanford University) TAP free energy September 19, 2018 3 / 29

  13. Z ✷ synchronization ◮ Signal: x ❂ ❬ ① ✶ ❀ ✿ ✿ ✿ ❀ ① ♥ ❪ T ✷ Z ♥ ✐✿✐✿❞✿ ✷ ❀ ① ✐ ✘ ❯♥✐❢✭ Z ✷ ✮ ❀ Z ✷ ❂ ❢ ✰✶ ❀ � ✶ ❣ ✿ ◮ Observation: for ✶ ✔ ✐ ❁ ❥ ✔ ♥ ❨ ✐❥ ❂ ✕ ♥① ✐ ① ❥ ✰ ❲ ✐❥ ✿ ◮ Noise ❲ ✐❥ ✘ ◆ ✭✵ ❀ ✶ ❂♥ ✮ . ◮ SNR ✕ ✷ ❬✵ ❀ ✶ ✮ fixed, dimension ♥ ✦ ✶ . ◮ In matrix notation: Y ❂ ✕ ♥ xx T ✰ W ✿ ◮ Task: given Y ❂ ✭ ❨ ✐❥ ✮ , estimate x (or say X ❂ xx T ). Song Mei (Stanford University) TAP free energy September 19, 2018 3 / 29

  14. Bayes estimation in Z ✷ synchronization ◮ Settings: Y ❂ ✭ ✕❂♥ ✮ xx T ✰ W ✿ x ✘ ❯♥✐❢✭ Z ♥ ✷ ✮ ❀ ◮ Estimate X ❂ xx T with loss: ❵ ✭ X ❀ ❝ X ✮ ❂ ✭✶ ❂♥ ✷ ✮ ❦ X � ❝ X ❦ ✷ ❋ ✿ ◮ For ✕ ❁ ✶ , estimation is impossible. ◮ For ✕ ❃ ✶ , estimation is possible and efficient, e.g., spectral estimator (Baik, Ben Arous, Peche phase transition). ◮ The optimal estimator is the Bayes estimator (also minimax estimator): X ❇❛②❡s ❂ E ❬ xx T ❥ Y ❪ ✿ ❝ Song Mei (Stanford University) TAP free energy September 19, 2018 4 / 29

  15. Bayes estimation in Z ✷ synchronization ◮ Settings: Y ❂ ✭ ✕❂♥ ✮ xx T ✰ W ✿ x ✘ ❯♥✐❢✭ Z ♥ ✷ ✮ ❀ ◮ Estimate X ❂ xx T with loss: ❵ ✭ X ❀ ❝ X ✮ ❂ ✭✶ ❂♥ ✷ ✮ ❦ X � ❝ X ❦ ✷ ❋ ✿ ◮ For ✕ ❁ ✶ , estimation is impossible. ◮ For ✕ ❃ ✶ , estimation is possible and efficient, e.g., spectral estimator (Baik, Ben Arous, Peche phase transition). ◮ The optimal estimator is the Bayes estimator (also minimax estimator): X ❇❛②❡s ❂ E ❬ xx T ❥ Y ❪ ✿ ❝ Song Mei (Stanford University) TAP free energy September 19, 2018 4 / 29

  16. Bayes estimation in Z ✷ synchronization ◮ Settings: Y ❂ ✭ ✕❂♥ ✮ xx T ✰ W ✿ x ✘ ❯♥✐❢✭ Z ♥ ✷ ✮ ❀ ◮ Estimate X ❂ xx T with loss: ❵ ✭ X ❀ ❝ X ✮ ❂ ✭✶ ❂♥ ✷ ✮ ❦ X � ❝ X ❦ ✷ ❋ ✿ ◮ For ✕ ❁ ✶ , estimation is impossible. ◮ For ✕ ❃ ✶ , estimation is possible and efficient, e.g., spectral estimator (Baik, Ben Arous, Peche phase transition). ◮ The optimal estimator is the Bayes estimator (also minimax estimator): X ❇❛②❡s ❂ E ❬ xx T ❥ Y ❪ ✿ ❝ Song Mei (Stanford University) TAP free energy September 19, 2018 4 / 29

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Variational Inference for GPs: Presenters Group1: Stochastic variational inference. Slides 2 - 28

Variational Inference for GPs: Presenters Group1: Stochastic variational inference. Slides 2 - 28

Variational Inference for GPs: Presenters Group1: Stochastic variational inference. Slides 2 - 28 Chaoqi Wang Sana Tonekaboni Will Grathwohl Group2: Variational inference for GPs. Slides 29 - 57 Trefor Evans Kingsley Chang Shems Saleh James

1.32k views • 98 slides
Spin glasses, concepts and analysis of susceptibility P.C.W. Holdsworth Ecole Normale Suprieure

Spin glasses, concepts and analysis of susceptibility P.C.W. Holdsworth Ecole Normale Suprieure

Spin glasses, concepts and analysis of susceptibility P.C.W. Holdsworth Ecole Normale Suprieure de Lyon 1. A (brief) review of spin glasses 2. Slow dynamics, aging phenomena 3. An illustration from a simple model My sources - Binder and Young

787 views • 53 slides
Recent advances on mean-field spin glasses Wei-Kuo Chen University of Minnesota Joint work with

Recent advances on mean-field spin glasses Wei-Kuo Chen University of Minnesota Joint work with

Recent advances on mean-field spin glasses Wei-Kuo Chen University of Minnesota Joint work with A. Auffinger, M. Handschy, D. Gamarnik, G. Lerman, D. Panchenko, M. Rahman, A. Sen, Q. Zeng July, 2018 1 / 17 What are spin glasses? 2 / 17

465 views • 34 slides
CS480/680 Machine Learning Lecture 11: February 11 th , 2020 Variational Inference Zahra

CS480/680 Machine Learning Lecture 11: February 11 th , 2020 Variational Inference Zahra

CS480/680 Machine Learning Lecture 11: February 11 th , 2020 Variational Inference Zahra Sheikhbahaee VARIATIONAL ALGORITHMS FOR APPROXIMATE BAYESIAN INFERENCE (Beal 2003, chapter 2) Variational Inference: A Review for Statisticians (Blei et al.

713 views • 28 slides
Spin glasses and Adiabatic Quantum Computing A.P. Young Talk at the Workshop on Theory and

Spin glasses and Adiabatic Quantum Computing A.P. Young Talk at the Workshop on Theory and

Spin glasses and Adiabatic Quantum Computing A.P. Young Talk at the Workshop on Theory and Practice of Adiabatic Quantum Computers and Quantum Simulation, ICTP, Trieste, August 22-26, 2016 Spin Glasses The problems studied on current quantum

537 views • 24 slides
Rejection Sampling Variational Inference Karan Grewal CSC2547 / STA4273 Overview Variational

Rejection Sampling Variational Inference Karan Grewal CSC2547 / STA4273 Overview Variational

Rejection Sampling Variational Inference Karan Grewal CSC2547 / STA4273 Overview Variational Inference Interested in computing posterior , but it is often intractable parametrize a variational family of distributions to

623 views • 12 slides
Deep Variational Inference FLARE Reading Group Presentation Wesley Tansey 9/28/2016 What is

Deep Variational Inference FLARE Reading Group Presentation Wesley Tansey 9/28/2016 What is

Deep Variational Inference FLARE Reading Group Presentation Wesley Tansey 9/28/2016 What is Variational Inference? What is Variational Inference? p*(x) Want to estimate some distribution, p*(x) What is Variational Inference?

684 views • 52 slides
Lecture Variational 13 Inference Panini Kaushal Scribes : - Margulies Smedeuranh Niklas

Lecture Variational 13 Inference Panini Kaushal Scribes : - Margulies Smedeuranh Niklas

Lecture Variational 13 Inference Panini Kaushal Scribes : - Margulies Smedeuranh Niklas Variational Inference Approximate by posterior Idea maximizing , bound lower Ptt variational fly ply ? ) a , , ) " 9) * =

709 views • 15 slides
Variational Inference for Bayes vMF Mixture Hanxiao Liu September 23, 2014 1 / 14 Variational

Variational Inference for Bayes vMF Mixture Hanxiao Liu September 23, 2014 1 / 14 Variational

Variational Inference for Bayes vMF Mixture Hanxiao Liu September 23, 2014 1 / 14 Variational Inference Review Lower bound the likelihood L ( ; X ) = E q log p ( X | ) log p ( X , Z | ) q ( Z ) = E q + E q log q (

310 views • 14 slides
On the replica approach for statistical mechanics of random system Giorgio Parisi Spin

On the replica approach for statistical mechanics of random system Giorgio Parisi Spin

On the replica approach for statistical mechanics of random system Giorgio Parisi Spin glasses in the infinite range limit (the Sherrington Kirkpatrick model). The computation of the free energy in the SK model (mean field

440 views • 19 slides
Spin Glasses and Information Processing Pavithran S Iyer Guide: Prof. V.V Sreedhar Chennai

Spin Glasses and Information Processing Pavithran S Iyer Guide: Prof. V.V Sreedhar Chennai

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Spin Glasses and Information Processing Pavithran S Iyer Guide: Prof. V.V Sreedhar Chennai Mathematical Institute April

748 views • 55 slides
Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family

Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family

Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations Wu Lin (UBC) June 11, 2019 Joint work with Mohammad Emtiyaz Khan (AIP, RIKEN) and Mark Schmidt (UBC) 1 / 7 Variational Inference (VI) VI

119 views • 7 slides
Variational Inference CMSC 691 UMBC Goal: Posterior Inference Hyperparameters Unknown

Variational Inference CMSC 691 UMBC Goal: Posterior Inference Hyperparameters Unknown

Approximate Inference: Variational Inference CMSC 691 UMBC Goal: Posterior Inference Hyperparameters Unknown parameters Data: p ( | ) Likelihood model: p( | ) (Some) Learning Techniques MAP/MLE: Point

1.09k views • 81 slides
You, the product! Me, my object. My glasses How I see/look without them My glasses = my normal

You, the product! Me, my object. My glasses How I see/look without them My glasses = my normal

You, the product! Me, my object. My glasses How I see/look without them My glasses = my normal vision. My glasses = beautiful finish of my appearance. My glasses = a big part of me . DISABILITY. ATTACHMENT. IDENTITY? Glasses or

677 views • 19 slides
Neural Variational Inference and Learning Andriy Mnih, Karol Gregor 22 June 2014 1 / 14

Neural Variational Inference and Learning Andriy Mnih, Karol Gregor 22 June 2014 1 / 14

Neural Variational Inference and Learning Andriy Mnih, Karol Gregor 22 June 2014 1 / 14 Introduction Training directed latent variable models is difficult because inference in them is intractable. Both MCMC and traditional variational

322 views • 14 slides
Lecture 14: Inference in Dirichlet Processes (Blei & Jordan, Variational inference for

Lecture 14: Inference in Dirichlet Processes (Blei & Jordan, Variational inference for

CS598JHM: Advanced NLP (Spring 2013) http://courses.engr.illinois.edu/cs598jhm/ Lecture 14: Inference in Dirichlet Processes (Blei & Jordan, Variational inference for Dirichlet Process Mixture models, Bayesian Analysis 2006) Julia

354 views • 21 slides
TERM PROJECT Classifying Tweets Using Nave Bayes Classifier CSC 177 Spring 2020 Andrew

TERM PROJECT Classifying Tweets Using Nave Bayes Classifier CSC 177 Spring 2020 Andrew

TERM PROJECT Classifying Tweets Using Nave Bayes Classifier CSC 177 Spring 2020 Andrew Flores, Hera Flores Agenda Demo Motivation Background Knowledge Lessons Learned Scope of the Project Future work Approach

609 views • 34 slides
Machine Learning in Spam Filtering A Crash Course in ML Konstantin Tretyakov kt@ut.ee

Machine Learning in Spam Filtering A Crash Course in ML Konstantin Tretyakov kt@ut.ee

Machine Learning in Spam Filtering A Crash Course in ML Konstantin Tretyakov kt@ut.ee Institute of Computer Science, University of Tartu Overview Spam is Evil ML for Spam Filtering: General Idea, Problems. Some Algorithms Nave Bayesian

391 views • 35 slides
Introduction to Adaptive Designs FUNDAMENTAL DESIGN PRINCIPLES, CASE STUDIES, AND HANDS-ON

Introduction to Adaptive Designs FUNDAMENTAL DESIGN PRINCIPLES, CASE STUDIES, AND HANDS-ON

Introduction to Adaptive Designs FUNDAMENTAL DESIGN PRINCIPLES, CASE STUDIES, AND HANDS-ON PRACTICE Minh Huynh, Ph.D. Aaron Heuser, Ph.D. Chunxiao Zhou, Ph.D. Course Objectives Gain familiarity with the basic principles of adaptive 1.

2.13k views • 174 slides
Reducing Uncertainty and Increasing Confidence in Reservoir Seismic Characterisation Erick

Reducing Uncertainty and Increasing Confidence in Reservoir Seismic Characterisation Erick

Reducing Uncertainty and Increasing Confidence in Reservoir Seismic Characterisation Erick Alvarez Team Leader Reservoir Seismic Characterisation Senergy Society of Petroleum Engineers Distinguished Lecturer Program www.spe.org/dl 1

275 views • 24 slides
Sentiment Analysis and Movie Reviews By: Donovan Ambler Overview Problem Description

Sentiment Analysis and Movie Reviews By: Donovan Ambler Overview Problem Description

Sentiment Analysis and Movie Reviews By: Donovan Ambler Overview Problem Description Sentiment Analysis and Naive Bayes Experimental Design and Procedures Results Problem Description For this problem, I will be taking a

159 views • 11 slides
High-Dimensional Classification Methods for Sparse Signals and Their Applications in Gene

High-Dimensional Classification Methods for Sparse Signals and Their Applications in Gene

High-Dimensional Classification Methods for Sparse Signals and Their Applications in Gene Expression Data Dawit Tadesse, Ph.D. Department of Mathematical Sciences University of Cincinnati Biostatistics Epidemiology & Research Design

809 views • 69 slides
Language Ted Dunning Kristinn Reykjavk University Languages

Language Ted Dunning Kristinn Reykjavk University Languages

Statistical Identification of Language Ted Dunning Kristinn Reykjavk University Languages Hall Hello Hallo Hola Bonjour 2 Languages Hall

166 views • 14 slides
Bayesian inference from all-sky SETI surveys Claudio Grimaldi C. Grimaldi, Sci. Rep. 7, 46273

Bayesian inference from all-sky SETI surveys Claudio Grimaldi C. Grimaldi, Sci. Rep. 7, 46273

Bayesian inference from all-sky SETI surveys Claudio Grimaldi C. Grimaldi, Sci. Rep. 7, 46273 (2017) C. Grimaldi, G.W. Marcy, N.K. Tellis, F. Drake, PASP 130, 054101 (2018) C. Grimaldi, G.W. Marcy, PNAS 115 , E9755 (2018) Bayes rule Bayes

510 views • 18 slides

More recommend