mmse approximation for the sparse prior using stochastic
play

MMSE Approximation for the Sparse Prior Using Stochastic Resonance - PowerPoint PPT Presentation

Feb 05, 2024 •6 likes •986 views

MMSE Approximation for the Sparse Prior Using Stochastic Resonance Dror Simon Computer Science, Technion, Israel January 15, 2020 Joint work with Jeremias Sulam, Yaniv Romano, Yue M. Lu and Michael Elad Dror Simon (Technion) MMSE for Sparse

  1. MMSE Approximation for the Sparse Prior Using Stochastic Resonance Dror Simon Computer Science, Technion, Israel January 15, 2020 Joint work with Jeremias Sulam, Yaniv Romano, Yue M. Lu and Michael Elad Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 1 / 46

  2. Noise Removal Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 2 / 46

  3. Noise Removal Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 2 / 46

  4. Noise Removal Why denoising? Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 2 / 46

  5. Noise Removal Why denoising? A simple testing ground for novel concepts in signal processing. Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 2 / 46

  6. Noise Removal Why denoising? A simple testing ground for novel concepts in signal processing. Can be generalized to other, more complicated applications. Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 2 / 46

  7. Noise Removal Noisy Signal Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 3 / 46

  8. Noise Removal Noisy Signal Clean Estimate Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 3 / 46

  9. Noise Removal Noisy Signal Signal Clean Restoration Estimate Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 3 / 46

  10. Noise Removal Noisy Signal Signal Clean Restoration Estimate Prior Knowledge Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 3 / 46

  11. Noise Removal – Bayesian Standpoint Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 4 / 46

  12. Noise Removal – Bayesian Standpoint Denoising Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 4 / 46

  13. Noise Removal – Bayesian Standpoint Signal Estimation Denoising Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 4 / 46

  14. Noise Removal – Bayesian Standpoint Signal Approx. Estimation Denoising Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 4 / 46

  15. Noise Removal – Bayesian Standpoint Suboptimal Signal Approx. Estimation Denoising Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 4 / 46

  16. Noise Removal – Bayesian Standpoint Suboptimal Signal Approx. Estimation Denoising Optimal Signal Estimation Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 4 / 46

  17. Noise Removal – Bayesian Standpoint Suboptimal Signal Approx. Estimation Denoising Optimal Signal Approx. Estimation Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 4 / 46

  18. Outline Bayesian Framework 1 The Generative Model Bayesian Estimators MMSE Approximation 2 Previous Work Stochastic Resonance 3 Can Noise Help Denoising? Our Proposed Method 4 The Algorithm Unitary Case Analysis Image Denoising Conclusions 5 Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 5 / 46

  19. Outline Bayesian Framework 1 The Generative Model Bayesian Estimators MMSE Approximation 2 Previous Work Stochastic Resonance 3 Can Noise Help Denoising? Our Proposed Method 4 The Algorithm Unitary Case Analysis Image Denoising Conclusions 5 Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 6 / 46

  20. The Generative Model D ∈ R n × m is a dictionary with normalized columns. m n D Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 7 / 46

  21. The Generative Model Each element i in α is non zero with probability p i ≪ 1. m n D α Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 7 / 46

  22. The Generative Model The non-zero elements of the sparse representation, denoted by α s , are � � 0 , σ 2 sampled from a Gaussian distribution α s | s ∼ N . α I | s | m n D α Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 7 / 46

  23. The Generative Model The product D α leads to a signal x . m = n D x α Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 7 / 46

  24. The Generative Model We are given noisy measurements y = D α + ν , where ν is a white � � 0 , σ 2 Gaussian noise ν ∼ N . ν I n m = n D x α Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 7 / 46

  25. The Generative Model – Results 1 Turek, Javier S., Irad Yavneh, and Michael Elad, 2011. ”On MMSE and MAP denoising under sparse representation modeling over a unitary dictionary.” Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 8 / 46

  26. The Generative Model – Results The prior probability of a support (Bernoulli): p ( s ) = � � i ∈ s p i ∈ s (1 − p j ). j / 1 Turek, Javier S., Irad Yavneh, and Michael Elad, 2011. ”On MMSE and MAP denoising under sparse representation modeling over a unitary dictionary.” Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 8 / 46

  27. The Generative Model – Results The prior probability of a support (Bernoulli): p ( s ) = � � i ∈ s p i ∈ s (1 − p j ). j / When the support is known, y and α s are jointly Gaussian y = D s α s + ν , leading to 1 1 Turek, Javier S., Irad Yavneh, and Michael Elad, 2011. ”On MMSE and MAP denoising under sparse representation modeling over a unitary dictionary.” Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 8 / 46

  28. The Generative Model – Results The prior probability of a support (Bernoulli): p ( s ) = � � i ∈ s p i ∈ s (1 − p j ). j / When the support is known, y and α s are jointly Gaussian y = D s α s + ν , leading to 1 y | s is Gaussian: y | s ∼ N ( 0 , C s ). 1 Turek, Javier S., Irad Yavneh, and Michael Elad, 2011. ”On MMSE and MAP denoising under sparse representation modeling over a unitary dictionary.” Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 8 / 46

  29. The Generative Model – Results The prior probability of a support (Bernoulli): p ( s ) = � � i ∈ s p i ∈ s (1 − p j ). j / When the support is known, y and α s are jointly Gaussian y = D s α s + ν , leading to 1 y | s is Gaussian: y | s ∼ N ( 0 , C s ). � � D s α s , σ 2 y | α s , s is Gaussian: y | α s , s ∼ N ν I n . 1 Turek, Javier S., Irad Yavneh, and Michael Elad, 2011. ”On MMSE and MAP denoising under sparse representation modeling over a unitary dictionary.” Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 8 / 46

  30. The Generative Model – Results The prior probability of a support (Bernoulli): p ( s ) = � � i ∈ s p i ∈ s (1 − p j ). j / When the support is known, y and α s are jointly Gaussian y = D s α s + ν , leading to 1 y | s is Gaussian: y | s ∼ N ( 0 , C s ). � � D s α s , σ 2 y | α s , s is Gaussian: y | α s , s ∼ N ν I n . � � 1 ν Q − 1 s D T s y , Q − 1 α s | y , s is Gaussian: α s | y , s ∼ N . σ 2 s Q s = 1 I | s | + 1 C s = σ 2 α D s D T s + σ 2 D T ν I n , s D s σ 2 σ ν α 1 Turek, Javier S., Irad Yavneh, and Michael Elad, 2011. ”On MMSE and MAP denoising under sparse representation modeling over a unitary dictionary.” Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 8 / 46

  31. Outline Bayesian Framework 1 The Generative Model Bayesian Estimators MMSE Approximation 2 Previous Work Stochastic Resonance 3 Can Noise Help Denoising? Our Proposed Method 4 The Algorithm Unitary Case Analysis Image Denoising Conclusions 5 Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 9 / 46

  32. Bayesian Estimators The goal: Estimate α given the noisy measurements y , i.e. denoise the signal. Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 10 / 46

  33. Bayesian Estimators The goal: Estimate α given the noisy measurements y , i.e. denoise the signal. Many estimators can be proposed. We focus our attention on three: Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 10 / 46

  34. Bayesian Estimators The goal: Estimate α given the noisy measurements y , i.e. denoise the signal. Many estimators can be proposed. We focus our attention on three: 1 The oracle estimator. Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 10 / 46

  35. Bayesian Estimators The goal: Estimate α given the noisy measurements y , i.e. denoise the signal. Many estimators can be proposed. We focus our attention on three: 1 The oracle estimator. 2 The Maximum A-posteriori Probability (MAP) estimator. Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 10 / 46

  36. Bayesian Estimators The goal: Estimate α given the noisy measurements y , i.e. denoise the signal. Many estimators can be proposed. We focus our attention on three: 1 The oracle estimator. 2 The Maximum A-posteriori Probability (MAP) estimator. 3 The Minimum Mean Square Error (MMSE) estimator. Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 10 / 46

  37. The Bayesian Estimators Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 11 / 46

  38. The Bayesian Estimators Oracle Estimator = E { α s | s , y } = 1 α Oracle Q − 1 s D T � s y s σ 2 ν Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 11 / 46

  39. The Bayesian Estimators MAP Support Estimator � s MAP = arg max p ( s | y ) s Dror Simon (Technion) MMSE for Sparse Prior January 15, 2020 11 / 46

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Adaptive Stochastic Collocation for PDE-Constrained Optimization under Uncertainty using Sparse

Adaptive Stochastic Collocation for PDE-Constrained Optimization under Uncertainty using Sparse

Two-Level Approximation Error-Aware Trust-Region Model Management Numerical Experiments Adaptive Stochastic Collocation for PDE-Constrained Optimization under Uncertainty using Sparse Grids and Model Reduction Matthew J. Zahr Advisor: Charbel

612 views • 46 slides
Bridging the gap between Stochastic Approximation and Markov chains Aymeric DIEULEVEUT ENS

Bridging the gap between Stochastic Approximation and Markov chains Aymeric DIEULEVEUT ENS

Bridging the gap between Stochastic Approximation and Markov chains Aymeric DIEULEVEUT ENS Paris, INRIA 17 november 2017 Joint work with Francis Bach and Alain Durmus. 1 Outline Introduction to Stochastic Approximation for Machine

725 views • 56 slides
Empirical Testing of Sparse Approximation and Matrix Completion Algorithms Jared Tanner Workshop

Empirical Testing of Sparse Approximation and Matrix Completion Algorithms Jared Tanner Workshop

Sparse Approximation Phase Transitions Matrix completion Empirical Testing of Sparse Approximation and Matrix Completion Algorithms Jared Tanner Workshop on Sparsity, Compressed Sensing and Applications University of Oxford

655 views • 34 slides
Stochastic Approximation in Hilbert Spaces Aymeric DIEULEVEUT Supervised by Francis BACH

Stochastic Approximation in Hilbert Spaces Aymeric DIEULEVEUT Supervised by Francis BACH

Stochastic Approximation in Hilbert Spaces Aymeric DIEULEVEUT Supervised by Francis BACH September 28, 2017 0 Outline 1. Introduction: Supervised Machine Learning Stochastic Approximation 2. Finite dimensional results 3. Infinite

1.25k views • 121 slides
Multi-level stochastic approximation algorithms Noufel Frikha Universit e Paris Diderot, LPMA

Multi-level stochastic approximation algorithms Noufel Frikha Universit e Paris Diderot, LPMA

Introduction Analysis of the SA scheme Multi-level stochastic approximation algorithms Numerical results Multi-level stochastic approximation algorithms Noufel Frikha Universit e Paris Diderot, LPMA 28th October, 2013. Stochastic

746 views • 35 slides
Calmness in stochastic programming exact penalization and sample approximation techniques

Calmness in stochastic programming exact penalization and sample approximation techniques

Calmness in stochastic programming exact penalization and sample approximation techniques Martin Branda Charles University in Prague Faculty of Mathematics and Physics Department of Probability and Mathematical Statistics Stochastic

940 views • 69 slides
Recent Theoretical Advances in Sparse Approximation Joel A. Tropp

Recent Theoretical Advances in Sparse Approximation Joel A. Tropp

Recent Theoretical Advances in Sparse Approximation Joel A. Tropp <jtropp@ices.utexas.edu> Institute for Computational Engineering and Sciences The University of Texas at Austin Includes joint work with A. C. Gilbert, S.

1.17k views • 80 slides
Stochastic approximation for adaptive Markov chain Monte Carlo algorithms Gersende FORT LTCI /

Stochastic approximation for adaptive Markov chain Monte Carlo algorithms Gersende FORT LTCI /

Stochastic approximation for adaptive Markov chain Monte Carlo algorithms Stochastic approximation for adaptive Markov chain Monte Carlo algorithms Gersende FORT LTCI / CNRS - TELECOM ParisTech, France Stochastic approximation for adaptive

448 views • 44 slides
A regularized least-squares method for sparse low-rank approximation of multivariate functions

A regularized least-squares method for sparse low-rank approximation of multivariate functions

Workshop Numerical methods for high-dimensional problems April 18, 2014 A regularized least-squares method for sparse low-rank approximation of multivariate functions Mathilde Chevreuil joint work with Prashant Rai, Loic Giraldi, Anthony

1k views • 53 slides
On the weak approximation of solutions to stochastic partial differential equations Raphael Kruse

On the weak approximation of solutions to stochastic partial differential equations Raphael Kruse

On the weak approximation of solutions to stochastic partial differential equations Raphael Kruse (TU Berlin) (joint work with Adam Andersson und Stig Larsson) Berlin-Padova Young Researchers Meeting Stochastic Analysis and Applications in

677 views • 32 slides
Some References P. Carpentier Master MMMEF Cours MNOS 2014-2015 263 / 263 Stochastic

Some References P. Carpentier Master MMMEF Cours MNOS 2014-2015 263 / 263 Stochastic

Stochastic Approximation and Stochastic Gradient Optimization Theory and Large Scale Systems Dual Effect and Discretization Stochastic Programming and Stochastic Optimal Control Some References P. Carpentier Master MMMEF Cours MNOS

385 views • 9 slides
Tutorial: Sparse Recovery Using Sparse Matrices Piotr Indyk MIT Problem Formulation

Tutorial: Sparse Recovery Using Sparse Matrices Piotr Indyk MIT Problem Formulation

Tutorial: Sparse Recovery Using Sparse Matrices Piotr Indyk MIT Problem Formulation (approximation theory, learning Fourier coeffs, linear sketching, finite rate of innovation, compressed sensing...) Setup: Data/signal in n-dimensional

418 views • 25 slides
Sparse Approximation of Signals and Images Gerlind Plonka Institute for Numerical and Applied

Sparse Approximation of Signals and Images Gerlind Plonka Institute for Numerical and Applied

Sparse Approximation of Signals and Images Gerlind Plonka Institute for Numerical and Applied Mathematics University of G ottingen in collaboration with Dennis Heinen, Armin Iske (Hamburg), Thomas Peter, Daniela Ro sca (Cluj-Napoca),

704 views • 54 slides
A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation

A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation

A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation Dr. Richard E. Turner ( ret26@cam.ac.uk ) Computational and Biological Learning Lab, Department of Engineering, University of Cambridge ...joint

715 views • 68 slides
A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation

A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation

A Unifying Framework for Sparse Gaussian Process Approximation using Power Expectation Propagation Dr. Richard E. Turner ( ret26@cam.ac.uk ) Computational and Biological Learning Lab, Department of Engineering, University of Cambridge ...joint

884 views • 71 slides
Sparse Inverse Covariance Estimation Using Quadratic Approximation Inderjit S. Dhillon Dept of

Sparse Inverse Covariance Estimation Using Quadratic Approximation Inderjit S. Dhillon Dept of

Sparse Inverse Covariance Estimation Using Quadratic Approximation Inderjit S. Dhillon Dept of Computer Science UT Austin MLSLP Symposium Portland, Oregon Sept 14, 2012 Joint work with C. Hsieh, M. Sustik and P. Ravikumar Inderjit S.

393 views • 28 slides
GNSS at tide gauges for Mean Dynamic Topography: Conventional measurements and Multipath

GNSS at tide gauges for Mean Dynamic Topography: Conventional measurements and Multipath

GNSS at tide gauges for Mean Dynamic Topography: Conventional measurements and Multipath Reflectometry Simon Williams (NOC, Liverpool) Chris Hughes (University of Liverpool and NOC, Liverpool) Mean dynamic topography at tide gauges why?

452 views • 20 slides
Approximate Posterior Inference for Differential Equation Models Jaeyong Lee Department of

Approximate Posterior Inference for Differential Equation Models Jaeyong Lee Department of

Approximate Posterior Inference for Differential Equation Models Jaeyong Lee Department of Statistics Seoul National University June 24, 2014 jointly with Sarat C. Dass, Universiti Teknologi PETRONAS Kyungjae Lee, Seoul National University

339 views • 31 slides
Non-Iterative, Feature-Preserving Mesh Smoothing Thouis R. Jones (MIT), Frdo Durand (MIT),

Non-Iterative, Feature-Preserving Mesh Smoothing Thouis R. Jones (MIT), Frdo Durand (MIT),

Non-Iterative, Feature-Preserving Mesh Smoothing Thouis R. Jones (MIT), Frdo Durand (MIT), Mathieu Desbrun (USC) thouis@graphics.csail.mit.edu, fredo@graphics.csail.mit.edu, desbrun@usc.edu Why Smooth? 3D scanners are noisy... Jones, Durand,

829 views • 50 slides
Probabilistic Optimal Estimation and Filtering Least Squares and Randomized Algorithms Fabrizio

Probabilistic Optimal Estimation and Filtering Least Squares and Randomized Algorithms Fabrizio

Probabilistic Optimal Estimation and Filtering Least Squares and Randomized Algorithms Fabrizio Dabbene 1 Mario Sznaier 2 Roberto Tempo 1 1 CNR - IEIIT Politecnico di Torino 2 Northeastern University Boston Workshop on Uncertain Dynamical

789 views • 53 slides
Master Thesis Preparations: Modelling and Identification of the HalfWing Jonas Schlagenhauf

Master Thesis Preparations: Modelling and Identification of the HalfWing Jonas Schlagenhauf

Master Thesis Preparations: Modelling and Identification of the HalfWing Jonas Schlagenhauf September 5, 2016 1 / 1 Objective of the master thesis NMPC of a constrained model airplane Develop system and tools Record data

517 views • 14 slides
Da Data-Dr Driven Progn ognos ostics cs of of Lithium- Io Ion R n Rec echar hargeable B

Da Data-Dr Driven Progn ognos ostics cs of of Lithium- Io Ion R n Rec echar hargeable B

Da Data-Dr Driven Progn ognos ostics cs of of Lithium- Io Ion R n Rec echar hargeable B eable Batter eries U ies Using sing Bi Bilinear Kernel Regression Charlie Hubbard, John Bavlsik, Chinmay Hegde and Chao Hu Iowa State

456 views • 23 slides
Advances in Credit Scoring: combining performance and interpretation in Kernel Discriminant

Advances in Credit Scoring: combining performance and interpretation in Kernel Discriminant

Advances in Credit Scoring: combining performance and interpretation in Kernel Discriminant Analysis. Caterina Liberati DEMS Universit degli Studi di Milano-Bicocca, Milan, Italy caterina.liberati@unimib.it November 10 th 2017 Liberati 1 /

564 views • 38 slides
FORECASTING DIRECTION OF CHINA SECURITY INDEX 300 MOVEMENT WITH LEAST SQUARES SUPPORT VECTOR

FORECASTING DIRECTION OF CHINA SECURITY INDEX 300 MOVEMENT WITH LEAST SQUARES SUPPORT VECTOR

FORECASTING DIRECTION OF CHINA SECURITY INDEX 300 MOVEMENT WITH LEAST SQUARES SUPPORT VECTOR MACHINE Shuai Wang ,Wei Shang Academy of Mathematics and Systems Sciences Chinese Academy of Sciences Beijing, China , June 2014 Contents 1

356 views • 20 slides

More recommend