spectral approximate inference
play

Spectral Approximate Inference Speaker: Sejun Park 1 Joint work with - PowerPoint PPT Presentation

Mar 02, 2023 •259 likes •526 views

Spectral Approximate Inference Speaker: Sejun Park 1 Joint work with Eunho Yang 1,2 , Se-Young Yun 1 and Jinwoo Shin 1,2 1 Korea Advanced Institute of Science and Technology (KAIST) 2 AITRICS June 13th, 2019 Goal: Partition Function Approximation

  1. Spectral Approximate Inference Speaker: Sejun Park 1 Joint work with Eunho Yang 1,2 , Se-Young Yun 1 and Jinwoo Shin 1,2 1 Korea Advanced Institute of Science and Technology (KAIST) 2 AITRICS June 13th, 2019

  2. Goal: Partition Function Approximation in GMs Pairwise binary graphical model (GM) is a joint distribution, factorized by • computer vision [Freeman et al., 2000] , social science [Scott, 2017] and deep learning [Hinton et al., 2006] Partition function Z is essential for inference, but it is NP-hard even to approximate [Jerrum, 1993] Park et al. (KAIST) Spectral Approximate Inference 2019.06.13

  3. Goal: Partition Function Approximation in GMs Pairwise binary graphical model (GM) is a joint distribution, factorized by • computer vision [Freeman et al., 2000] , social science [Scott, 2017] and deep learning [Hinton et al., 2006] Partition function Z is essential for inference, but it is NP-hard even to approximate [Jerrum, 1993] In theory, Z of only a few restricted classes of GM can be approximated in polynomial time 1. Structured GMs: e.g., A is an adjacency matrix of tree/planar graphs [Temperley et al., 1961; Pearl, 1982] 2. GMs with homogeneous parameters: e.g., [Jerrum, 1993; Li et al., 2013; Liu, 2018] 3. GMs under correlation decay/tree uniqueness: e.g., [Li et al., 2013] Park et al. (KAIST) Spectral Approximate Inference 2019.06.13

  4. Goal: Partition Function Approximation in GMs Pairwise binary graphical model (GM) is a joint distribution, factorized by • computer vision [Freeman et al., 2000] , social science [Scott, 2017] and deep learning [Hinton et al., 2006] Partition function Z is essential for inference, but it is NP-hard even to approximate [Jerrum, 1993] In practice, approximation algorithms based on certain local structures/consistency have been used 1. Markov chain Monte Carlo: e.g., annealed importance sampling [Neal, 2001] 2. Variational inference: e.g., belief propagation [Pearl, 1982], mean-field approximation [Parisi, 1988] 3. Variable elimination: e.g., minibucket [Dechter et al., 2003] , weighted minibucket [Lie et al., 2011] However, due to their local nature, they often fails under large global correlation (i.e., large A) Park et al. (KAIST) Spectral Approximate Inference 2019.06.13

  5. Goal: Partition Function Approximation in GMs Pairwise binary graphical model (GM) is a joint distribution, factorized by • computer vision [Freeman et al., 2000] , social science [Scott, 2017] and deep learning [Hinton et al., 2006] We study the spectral properties Partition function Z is essential for inference, but it is NP-hard even to approximate [Jerrum, 1993] of the parameter matrix A for more robust approximate inference In practice, approximation algorithms based on certain local structures/consistency have been used 1. Markov chain Monte Carlo: e.g., annealed importance sampling [Neal, 2001] 2. Variational inference: e.g., belief propagation [Pearl, 1982], mean-field approximation [Parisi, 1988] 3. Variable elimination: e.g., minibucket [Dechter et al., 2003] , weighted minibucket [Lie et al., 2011] However, due to their local nature, they often fails under large global correlation (i.e., large A) Park et al. (KAIST) Spectral Approximate Inference 2019.06.13

  6. Spectral Approximate Inference for Low-Rank GMs Provable approximate inference algorithm for low-rank GMs (low-rank A)

  7. Spectral Approximate Inference for Low-Rank GMs Proposed algorithm using spectral properties of A ( ) Eigenvalue decomposition Park et al. (KAIST) Spectral Approximate Inference 2019.06.13

  8. Spectral Approximate Inference for Low-Rank GMs Proposed algorithm using spectral properties of A ( ) 1. Transform the domain of integration from to using the identity n-dimensional integration 1-dimensional integration Park et al. (KAIST) Spectral Approximate Inference 2019.06.13

  9. Spectral Approximate Inference for Low-Rank GMs Proposed algorithm using spectral properties of A ( ) 1. Transform the domain of integration from to using the identity 2. Approximate 1-dimensional integration into a polynomial number of summations using histogram 1-dimensional integration Polynomial # summations (Riemann sum / histogram) Park et al. (KAIST) Spectral Approximate Inference 2019.06.13

  10. Spectral Approximate Inference for Low-Rank GMs Proposed algorithm using spectral properties of A ( ) 1. Transform the domain of integration from to using the identity 2. Approximate 1-dimensional integration into a polynomial number of summations using histogram 3. Compute the weight of the histogram recursively For differing only at Weight of histogram Park et al. (KAIST) Spectral Approximate Inference 2019.06.13

  11. Spectral Approximate Inference for Low-Rank GMs Proposed algorithm using spectral properties of A ( ) 1. Transform the domain of integration from to using the identity 2. Approximate 1-dimensional integration into a polynomial number of summations using histogram 3. Compute the weight of the histogram recursively 4. Compute the approximated Z from using the histogram Polynomial # summations (Riemann sum / histogram) Park et al. (KAIST) Spectral Approximate Inference 2019.06.13

  12. Spectral Approximate Inference for Low-Rank GMs Proposed algorithm using spectral properties of A • The procedure for rank-1 GMs generalizes to arbitrary GMs by considering the histogram of -dimension Theorem [Park et al., 2019] For any , the algorithm outputs such that in time Park et al. (KAIST) Spectral Approximate Inference 2019.06.13

  13. Spectral Approximate Inference for Low-Rank GMs Proposed algorithm using spectral properties of A • The procedure for rank-1 GMs generalizes to arbitrary GMs by considering the histogram of -dimension Theorem [Park et al., 2019] For any , the algorithm outputs such that in time • The proposed algorithm is a fully polynomial-time approximation scheme (FPTAS) for • However, it is hard to use the algorithm for general GMs due to its complexity, exponential to Park et al. (KAIST) Spectral Approximate Inference 2019.06.13

  14. Spectral Approximate Inference for Low-Rank GMs Proposed algorithm using spectral properties of A • The procedure for rank-1 GMs generalizes to arbitrary GMs by considering the histogram of -dimension Theorem [Park et al., 2019] For any , the algorithm outputs such that in time • The proposed algorithm is a fully polynomial-time approximation scheme (FPTAS) for • However, it is hard to use the algorithm for general GMs due to its complexity, exponential to Next: we propose an algorithm for general high-rank GMs Park et al. (KAIST) Spectral Approximate Inference 2019.06.13

  15. Spectral Approximate Inference for High-Rank GMs Approximation algorithm for general high-rank GMs

  16. Spectral Approximate Inference for High-Rank GMs Mean-field approximation Park et al. (KAIST) Spectral Approximate Inference 2019.06.13

  17. Spectral Approximate Inference for High-Rank GMs Mean-field approximation Transform summation into expectation Park et al. (KAIST) Spectral Approximate Inference 2019.06.13

  18. Spectral Approximate Inference for High-Rank GMs Mean-field approximation Eigenvalue decomposition Park et al. (KAIST) Spectral Approximate Inference 2019.06.13

  19. Spectral Approximate Inference for High-Rank GMs Mean-field approximation Mean-field approximation Product of rank-1 expectations Park et al. (KAIST) Spectral Approximate Inference 2019.06.13

  20. Spectral Approximate Inference for High-Rank GMs Mean-field approximation Mean-field approximation Product of rank-1 expectations Controlling the mean-field approximation by varying the spectral property Park et al. (KAIST) Spectral Approximate Inference 2019.06.13

  21. Spectral Approximate Inference for High-Rank GMs Mean-field approximation Mean-field approximation Product of rank-1 expectations Controlling the mean-field approximation by varying the spectral property Free parameter: diagonal matrix D Park et al. (KAIST) Spectral Approximate Inference 2019.06.13

  22. Spectral Approximate Inference for High-Rank GMs Mean-field approximation with a diagonal matrix D Goal: Reduce the error by choosing proper D Controlling the mean-field approximation by varying the spectral property Free parameter: diagonal matrix D Park et al. (KAIST) Spectral Approximate Inference 2019.06.13

  23. Spectral Approximate Inference for High-Rank GMs Mean-field approximation with a diagonal matrix D Goal: Reduce the error by choosing proper D Optimizing diagonal matrix D for reducing the approximation error Semi-definite programming Park et al. (KAIST) Spectral Approximate Inference 2019.06.13

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Approximate inference: Sampling methods Probabilistic Graphical Models Sharif University of

Approximate inference: Sampling methods Probabilistic Graphical Models Sharif University of

Approximate inference: Sampling methods Probabilistic Graphical Models Sharif University of Technology Spring 2018 Soleymani Approximate inference Approximate inference techniques Deterministic approximation Variational algorithms

1.95k views • 97 slides
Two Approximate- Programmability Birds, One Statistical- Inference Stone Adrian Sampson

Two Approximate- Programmability Birds, One Statistical- Inference Stone Adrian Sampson

Two Approximate- Programmability Birds, One Statistical- Inference Stone Adrian Sampson University of Washington sa pa APPROX 2014 Assisted approximate programming Statistical inference Cheap check generation 1 Assisted approximate

668 views • 25 slides
Inference in Bayesian networks Chapter 14.45 Chapter 14.45 1 Outline Exact inference

Inference in Bayesian networks Chapter 14.45 Chapter 14.45 1 Outline Exact inference

Inference in Bayesian networks Chapter 14.45 Chapter 14.45 1 Outline Exact inference by enumeration Exact inference by variable elimination Approximate inference by stochastic simulation Approximate inference by Markov

530 views • 38 slides
Bayesian networks: approximate inference Machine Intelligence Thomas D. Nielsen September 2008

Bayesian networks: approximate inference Machine Intelligence Thomas D. Nielsen September 2008

Bayesian networks: approximate inference Machine Intelligence Thomas D. Nielsen September 2008 Approximative inference September 2008 1 / 25 Approximate Inference Motivation Because of the (worst-case) intractability of exact inference in

1.13k views • 37 slides
Approximate Inference: Randomized Methods October 15, 2015 Topics Hard Inference

Approximate Inference: Randomized Methods October 15, 2015 Topics Hard Inference

Approximate Inference: Randomized Methods October 15, 2015 Topics Hard Inference Local search & hill climbing Stochastic hill climbing / Simulated Annealing Soft Inference Monte-Carlo approximations Markov-Chain

975 views • 66 slides
Approximate Inference 9.520 Class 19 Ruslan Salakhutdinov BCS and CSAIL, MIT 1 Plan 1.

Approximate Inference 9.520 Class 19 Ruslan Salakhutdinov BCS and CSAIL, MIT 1 Plan 1.

Approximate Inference 9.520 Class 19 Ruslan Salakhutdinov BCS and CSAIL, MIT 1 Plan 1. Introduction/Notation. 2. Examples of successful Bayesian models. 3. Laplace and Variational Inference. 4. Basic Sampling Algorithms. 5. Markov chain

833 views • 45 slides
Travel Time Estimation using Approximate Belief States on a Hidden Markov Model Walid Krichene

Travel Time Estimation using Approximate Belief States on a Hidden Markov Model Walid Krichene

Travel Time Estimation using Approximate Belief States on a Hidden Markov Model Walid Krichene Overview Context Inference on a HMM Modeling framework and exact inference Approximate Inference: the Boyen-Koller algorithm Graph Partitioning

1.04k views • 82 slides
Modern Gaussian Processes: Scalable Inference and Novel Applications (Part II-b) Approximate

Modern Gaussian Processes: Scalable Inference and Novel Applications (Part II-b) Approximate

Modern Gaussian Processes: Scalable Inference and Novel Applications (Part II-b) Approximate Inference Edwin V. Bonilla and Maurizio Filippone CSIROs Data61, Sydney, Australia and EURECOM, Sophia Antipolis, France July 14 th , 2019 1

1.14k views • 44 slides
Approximate Inference by Stochastic Simulation/Sampling Methods Zhenke Wu Department of

Approximate Inference by Stochastic Simulation/Sampling Methods Zhenke Wu Department of

Approximate Inference by Stochastic Simulation/Sampling Methods Zhenke Wu Department of Biostatistics University of Michigan October 20, 2016 Inference Techniques Central task of applying probabilistic models: Evaluate the posterior:

331 views • 16 slides
Approximate Inference Henrik I. Christensen Robotics & Intelligent Machines @ GT Georgia

Approximate Inference Henrik I. Christensen Robotics & Intelligent Machines @ GT Georgia

Introduction Variational Inference Mixture of Gaussians Exponential Family Expectation Propagation Summary Approximate Inference Henrik I. Christensen Robotics & Intelligent Machines @ GT Georgia Institute of Technology, Atlanta, GA

628 views • 36 slides
Approximate Bayesian inference for latent Gaussian models avard Rue 1 H Department of

Approximate Bayesian inference for latent Gaussian models avard Rue 1 H Department of

Approximative Bayesian inference Approximate Bayesian inference for latent Gaussian models avard Rue 1 H Department of Mathematical Sciences NTNU, Norway December 4, 2009 1 With S.Martino/N.Chopin Approximative Bayesian inference Overview

1.61k views • 159 slides
Spectral Inference under Complex Temporal Dynamics Jun Yang joint work with Zhou Zhou

Spectral Inference under Complex Temporal Dynamics Jun Yang joint work with Zhou Zhou

Spectral Inference under Complex Temporal Dynamics Jun Yang joint work with Zhou Zhou Department of Statistical Sciences University of Toronto SSC 2019, University of Calgary Spectral Inference (Jun Yang) 1 Motivation Frequency content of

395 views • 12 slides
Approximate inference (Ch. 14) Likelihood Weighting P(b|a) 1 P(a) 0.5 A B P(b|a) 0.2 In

Approximate inference (Ch. 14) Likelihood Weighting P(b|a) 1 P(a) 0.5 A B P(b|a) 0.2 In

Approximate inference (Ch. 14) Likelihood Weighting P(b|a) 1 P(a) 0.5 A B P(b|a) 0.2 In LW, say we generated 2 samples: [a] : w = 1, [a], w=0.2 If we did rejection sampling, we need about 5 a to actually get a b, so 10

645 views • 33 slides
ECE 4524 Artificial Intelligence and Engineering Applications Lecture 20: Approximate Inference

ECE 4524 Artificial Intelligence and Engineering Applications Lecture 20: Approximate Inference

ECE 4524 Artificial Intelligence and Engineering Applications Lecture 20: Approximate Inference Reading: AIAMA 14.5, see also MacKay Chapters 29 and Chapters 27 and 33 Todays Schedule: Inference by simulation sampling random

285 views • 12 slides
Approximate Spectral Clustering via Randomized Sketching Christos Boutsidis Yahoo! Labs, New

Approximate Spectral Clustering via Randomized Sketching Christos Boutsidis Yahoo! Labs, New

Approximate Spectral Clustering via Randomized Sketching Christos Boutsidis Yahoo! Labs, New York Joint work with Alex Gittens (Ebay), Anju Kambadur (IBM) The big picture: sketch and solve Tradeoff : Speed (depends on the size of A)

378 views • 22 slides
Approximate inference on graphical models: variational methods Alexandre Bouchard-C ot e

Approximate inference on graphical models: variational methods Alexandre Bouchard-C ot e

Approximate inference on graphical models: variational methods Alexandre Bouchard-C ot e Exact inference in general graphs. . . Recall: we now have an exact, general and efficient inference algorithm: the Junction-Tree algorithm

781 views • 39 slides
Structured Policy Iteration for Linear Quadratic Regulator Youngsuk Park 1 with R. Rossi 2 , Z.

Structured Policy Iteration for Linear Quadratic Regulator Youngsuk Park 1 with R. Rossi 2 , Z.

Structured Policy Iteration for Linear Quadratic Regulator Youngsuk Park 1 with R. Rossi 2 , Z. Wen 3 , G. Wu 2 , and H. Zhao 2 Stanford University 1 , Adobe research 2 , Deepmind 3 July 14, 2020 Introduction reinforcement learning (RL) is

319 views • 18 slides
Hibachi: A Cooperative Hybrid Cache with NVRAM and DRAM for Storage Arrays Ziqi Fan, Fenggang Wu,

Hibachi: A Cooperative Hybrid Cache with NVRAM and DRAM for Storage Arrays Ziqi Fan, Fenggang Wu,

Hibachi: A Cooperative Hybrid Cache with NVRAM and DRAM for Storage Arrays Ziqi Fan, Fenggang Wu, Dongchul Park 1 , Jim Diehl, Doug Voigt 2 , and David H.C. Du University of Minnesota, 1 Intel, 2 HP Enterprise May 18, 2017 C enter for R esearch in

875 views • 20 slides
FAST: Quick Application Launch on Solid-State Drives Yongsoo Joo 1 , Junhee Ryu 2 , Sangsoo Park

FAST: Quick Application Launch on Solid-State Drives Yongsoo Joo 1 , Junhee Ryu 2 , Sangsoo Park

FAST: Quick Application Launch on Solid-State Drives Yongsoo Joo 1 , Junhee Ryu 2 , Sangsoo Park 1 , and Kang G. Shin 1,3 1 Ewha Womans University, Korea 2 Seoul National University, Korea 3 University of Michigan, USA Application Launch Delay

402 views • 36 slides
Yun-Hee Park 1 , Arastoo Pour Biazar 1 , Richard T. McNider 1 , Bright Dornblaser 3 , Maudood Khan

Yun-Hee Park 1 , Arastoo Pour Biazar 1 , Richard T. McNider 1 , Bright Dornblaser 3 , Maudood Khan

Yun-Hee Park 1 , Arastoo Pour Biazar 1 , Richard T. McNider 1 , Bright Dornblaser 3 , Maudood Khan 2 , Kevin Doty 1 1. University of Alabama in Huntsville 2. University Space Research Association (USRA) 3. Texas Commission on Environmental

498 views • 29 slides
An Empirical Study on Reducing Omission Errors in Practice Jihun

An Empirical Study on Reducing Omission Errors in Practice Jihun

ASE 2014 An Empirical Study on Reducing Omission Errors in Practice Jihun Park 1 , Miryung Kim 2 , Doo-Hwan Bae 1 1. KAIST, South Korea 2. University of California,

300 views • 14 slides
Evidence for Episodic Accretion in Class I Source, IRAS 16316-1540 Sung-Yong Yoon 1 , Jeong-Eun

Evidence for Episodic Accretion in Class I Source, IRAS 16316-1540 Sung-Yong Yoon 1 , Jeong-Eun

Evidence for Episodic Accretion in Class I Source, IRAS 16316-1540 Sung-Yong Yoon 1 , Jeong-Eun Lee 1 , Seokho Lee 1 , Sunkyung Park 1 1 Kyung Hee University Contents 1.Introduction 1.1. Inner Region of Protoplanetary Disk 1.2. IGRINS Survey

573 views • 20 slides
Charm++ as an Energy Efficient Runtime 1 4/18/17 BILGE ACUN - CHARM++ WORKSHOP 2017 Interaction

Charm++ as an Energy Efficient Runtime 1 4/18/17 BILGE ACUN - CHARM++ WORKSHOP 2017 Interaction

Charm++ as an Energy Efficient Runtime 1 4/18/17 BILGE ACUN - CHARM++ WORKSHOP 2017 Interaction Between the Runtime System and the Resource Manager Allows dynamic interaction between the system resource manager or scheduler and the job

523 views • 25 slides
LOBBY 10 1 PAINLESS ADVOCACY: The Art of Successfully Engaging with Your Elected officials

LOBBY 10 1 PAINLESS ADVOCACY: The Art of Successfully Engaging with Your Elected officials

LOBBY 10 1 PAINLESS ADVOCACY: The Art of Successfully Engaging with Your Elected officials PRESENTED BY Ellen Teller, Food Research and Action Center Carrie Calvert, Feeding America February 21, 2017, 4:00 PM EST Local Voices Matter!

436 views • 19 slides

More recommend