beyond the graphical lasso structure learning via inverse
play

Beyond the graphical Lasso: Structure learning via inverse - PowerPoint PPT Presentation

Mar 12, 2024 •370 likes •1.36k views

Beyond the graphical Lasso: Structure learning via inverse covariance estimation Po-Ling Loh UC Berkeley Department of Statistics ICML Workshop on Covariance Selection and Graphical Model Structure Learning June 26, 2014 Joint work with

  1. Beyond the graphical Lasso: Structure learning via inverse covariance estimation Po-Ling Loh UC Berkeley Department of Statistics ICML Workshop on Covariance Selection and Graphical Model Structure Learning June 26, 2014 Joint work with Martin Wainwright (UC Berkeley) & Peter B¨ uhlmann (ETH Z¨ urich) P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 1 / 40

  2. Outline Introduction 1 Generalized inverse covariances 2 Linear structural equation models 3 Corrupted data 4 P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 2 / 40

  3. Outline Introduction 1 Generalized inverse covariances 2 Linear structural equation models 3 Corrupted data 4 P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 3 / 40

  4. Undirected graphical models Undirected graph G = ( V , E ) Joint distribution of ( X 1 , . . . , X p ), where | V | = p X 3 X 1 X 2 X p P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 4 / 40

  5. Undirected graphical models Undirected graph G = ( V , E ) Joint distribution of ( X 1 , . . . , X p ), where | V | = p X 3 X 1 X 2 X p Markov property: ( s , t ) / ∈ E = ⇒ X s ⊥ ⊥ X t | X \{ s , t } P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 4 / 40

  6. Undirected graphical models Undirected graph G = ( V , E ) Joint distribution of ( X 1 , . . . , X p ), where | V | = p A X 3 X 1 B X 2 X p S More generally, X A ⊥ ⊥ X B | X S when S ⊆ V separates A from B P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 4 / 40

  7. Directed graphical models Directed acyclic graph G = ( V , E ) X 1 X 2 X 3 X p Markov property: X j ⊥ ⊥ X Nondesc( j ) | X Pa( j ) , ∀ j P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 5 / 40

  8. Structure learning � � n ( X ( i ) 1 , X ( i ) 2 , . . . , X ( i ) Goal: Edge recovery from n samples: p ) i =1 P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 6 / 40

  9. Structure learning � � n ( X ( i ) 1 , X ( i ) 2 , . . . , X ( i ) Goal: Edge recovery from n samples: p ) i =1 High-dimensional setting: p ≫ n , assume deg( G ) ≤ d P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 6 / 40

  10. Structure learning � � n ( X ( i ) 1 , X ( i ) 2 , . . . , X ( i ) Goal: Edge recovery from n samples: p ) i =1 High-dimensional setting: p ≫ n , assume deg( G ) ≤ d Sources of corruption: non-i.i.d. observations, contamination by noise/missing data P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 6 / 40

  11. Structure learning � � n ( X ( i ) 1 , X ( i ) 2 , . . . , X ( i ) Goal: Edge recovery from n samples: p ) i =1 High-dimensional setting: p ≫ n , assume deg( G ) ≤ d Sources of corruption: non-i.i.d. observations, contamination by noise/missing data Note: Structure learning generally harder for directed graphs (topological order unknown) P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 6 / 40

  12. Graphical Lasso When ( X 1 , . . . , X p ) ∼ N (0 , Σ), well-known fact: (Σ − 1 ) st = 0 ⇐ ⇒ ( s , t ) / ∈ E P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 7 / 40

  13. Graphical Lasso When ( X 1 , . . . , X p ) ∼ N (0 , Σ), well-known fact: (Σ − 1 ) st = 0 ⇐ ⇒ ( s , t ) / ∈ E Establishes statistical consistency of graphical Lasso (Yuan & Lin ’07):     � �  trace( � Θ ∈ arg min ΣΘ) − log det(Θ) + λ | Θ st |  Θ � 0 s � = t P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 7 / 40

  14. Some observations P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 8 / 40

  15. Some observations Only sample-based quantity is � Σ:     � �  trace( � Θ ∈ arg min ΣΘ) − log det(Θ) + λ | Θ st |  Θ � 0 s � = t P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 8 / 40

  16. Some observations Only sample-based quantity is � Σ:     � �  trace( � Θ ∈ arg min ΣΘ) − log det(Θ) + λ | Θ st |  Θ � 0 s � = t Although graphical Lasso is penalized Gaussian MLE , can always be used to estimate � Θ from � Σ: � � (Σ ∗ ) − 1 = arg min trace(Σ ∗ Θ) − log det(Θ) Θ P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 8 / 40

  17. Some observations Only sample-based quantity is � Σ:     � �  trace( � Θ ∈ arg min ΣΘ) − log det(Θ) + λ | Θ st |  Θ � 0 s � = t Although graphical Lasso is penalized Gaussian MLE , can always be used to estimate � Θ from � Σ: � � (Σ ∗ ) − 1 = arg min trace(Σ ∗ Θ) − log det(Θ) Θ We extend graphical Lasso to discrete-valued data (undirected case) and linear structural equation models (directed case) P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 8 / 40

  18. Theory for graphical Lasso If � � log p log p � � Σ − Σ ∗ � max � and λ � , n n then �� � log p � � Θ − Θ ∗ � max � + λ n P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 9 / 40

  19. Theory for graphical Lasso If � � log p log p � � Σ − Σ ∗ � max � and λ � , n n then �� � log p � � Θ − Θ ∗ � max � + λ n Deviation condition holds w.h.p. for various ensembles (e.g., sub-Gaussian) P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 9 / 40

  20. Theory for graphical Lasso If � � log p log p � � Σ − Σ ∗ � max � and λ � , n n then �� � log p � � Θ − Θ ∗ � max � + λ n Deviation condition holds w.h.p. for various ensembles (e.g., sub-Gaussian) � Thresholding � log p Θ at level yields correct support n P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 9 / 40

  21. Outline Introduction 1 Generalized inverse covariances 2 Linear structural equation models 3 Corrupted data 4 P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 10 / 40

  22. Non-Gaussian distributions (Liu et al. ’09, ’12): ( X 1 , . . . , X p ) follows nonparanormal distribution if ( f 1 ( X 1 ) , . . . , f p ( X p )) ∼ N (0 , Σ), and f j ’s monotone and differentiable P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 11 / 40

  23. Non-Gaussian distributions (Liu et al. ’09, ’12): ( X 1 , . . . , X p ) follows nonparanormal distribution if ( f 1 ( X 1 ) , . . . , f p ( X p )) ∼ N (0 , Σ), and f j ’s monotone and differentiable Then ( i , j ) / ∈ E iff Θ ij = 0 P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 11 / 40

  24. Non-Gaussian distributions (Liu et al. ’09, ’12): ( X 1 , . . . , X p ) follows nonparanormal distribution if ( f 1 ( X 1 ) , . . . , f p ( X p )) ∼ N (0 , Σ), and f j ’s monotone and differentiable Then ( i , j ) / ∈ E iff Θ ij = 0 In general non-Gaussian setting, relationship between entries of Θ = Σ − 1 and edges of G unknown P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 11 / 40

  25. Discrete graphical models Assume X i ’s take values in a discrete set: { 0 , 1 , . . . , m − 1 } P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 12 / 40

  26. Discrete graphical models Assume X i ’s take values in a discrete set: { 0 , 1 , . . . , m − 1 } Our results: Establish relationship between augmented inverse covariance matrices and edge structure New algorithms for structure learning in discrete graphs P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 12 / 40

  27. An illustrative example Binary Ising model:   � �  , P θ ( x 1 , . . . , x p ) ∝ exp θ s x s + θ st x s x t s ∈ V ( s , t ) ∈ E P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 13 / 40

  28. An illustrative example Binary Ising model:   � �  , P θ ( x 1 , . . . , x p ) ∝ exp θ s x s + θ st x s x t s ∈ V ( s , t ) ∈ E θ ∈ R p + ( p 2 ) , ( x 1 , . . . , x p ) ∈ { 0 , 1 } p P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 13 / 40

  29. An illustrative example Ising models with θ s = 0 . 1 , θ st = 2 X 1 X 4 X 2 X 3 X 1 X 4 X 2 X 3 P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 14 / 40

  30. An illustrative example Ising models with θ s = 0 . 1 , θ st = 2   X 1 X 4 9 . 80 − 3 . 59 0 0   − 3 . 59 34 . 30 − 4 . 77 0   Θ chain =   0 − 4 . 77 34 . 30 − 3 . 59 0 0 − 3 . 59 9 . 80 X 2 X 3   X 1 X 4 51 . 37 − 5 . 37 − 0 . 17 − 5 . 37   − 5 . 37 51 . 37 − 5 . 37 − 0 . 17   Θ loop =   − 0 . 17 − 5 . 37 51 . 37 − 5 . 37 − 5 . 37 − 0 . 17 − 5 . 37 51 . 37 X 2 X 3 P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 14 / 40

  31. An illustrative example Ising models with θ s = 0 . 1 , θ st = 2   X 1 X 4 9 . 80 − 3 . 59 0 0   − 3 . 59 34 . 30 − 4 . 77 0   Θ chain =   0 − 4 . 77 34 . 30 − 3 . 59 0 0 − 3 . 59 9 . 80 X 2 X 3   X 1 X 4 51 . 37 − 5 . 37 − 0 . 17 − 5 . 37   − 5 . 37 51 . 37 − 5 . 37 − 0 . 17   Θ loop =   − 0 . 17 − 5 . 37 51 . 37 − 5 . 37 − 5 . 37 − 0 . 17 − 5 . 37 51 . 37 X 2 X 3 Θ is graph-structured for chain, but not loop P. Loh (UC Berkeley) Beyond the graphical Lasso June 26, 2014 14 / 40

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Probabilistic Graphical Models Probabilistic Graphical Models Structure learning in Bayesian

Probabilistic Graphical Models Probabilistic Graphical Models Structure learning in Bayesian

Probabilistic Graphical Models Probabilistic Graphical Models Structure learning in Bayesian networks Siamak Ravanbakhsh Fall 2019 Learning objectives Learning objectives why structure learning is hard? two approaches to structure learning

1.04k views • 74 slides
Bilevel Learning of the Group Lasso Structure Jordan Frecon 1 , Saverio Salzo 1 , Massimiliano

Bilevel Learning of the Group Lasso Structure Jordan Frecon 1 , Saverio Salzo 1 , Massimiliano

Bilevel Learning of the Group Lasso Structure Jordan Frecon 1 , Saverio Salzo 1 , Massimiliano Pontil 1 , 2 1 CSML - Istituto Italiano di Tecnologia 2 Dept of Computer Science - University College London Thirty-second Conference on Neural

327 views • 9 slides
Structure' Learning' Daphne Koller Why Structure Learning To learn model for new queries,

Structure' Learning' Daphne Koller Why Structure Learning To learn model for new queries,

Learning' Probabilis2c' Graphical' BN'Structure' Models' Structure' Learning' Daphne Koller Why Structure Learning To learn model for new queries, when domain expertise is not perfect For structure discovery, when inferring network

752 views • 61 slides
Learning in Graphical Models Andrea Passerini passerini@disi.unitn.it Machine Learning Learning

Learning in Graphical Models Andrea Passerini passerini@disi.unitn.it Machine Learning Learning

Learning in Graphical Models Andrea Passerini passerini@disi.unitn.it Machine Learning Learning in Graphical Models Learning graphical models Parameter estimation We assume the structure of the model is given We are given a dataset of

268 views • 23 slides
Learning Bayesian network : Given structure and completely observed data Probabilistic Graphical

Learning Bayesian network : Given structure and completely observed data Probabilistic Graphical

Learning Bayesian network : Given structure and completely observed data Probabilistic Graphical Models Sharif University of Technology Spring 2017 Soleymani Learning problem Target: true distribution that maybe correspond to

766 views • 61 slides
Structure Learning: the good, the bad, the ugly Graphical Model 10708 Carlos Guestrin

Structure Learning: the good, the bad, the ugly Graphical Model 10708 Carlos Guestrin

Koller & Friedman Chapter 13 Structure Learning: the good, the bad, the ugly Graphical Model 10708 Carlos Guestrin Carnegie Mellon University October 24 th , 2005 Project feedback by e-mail soon Announcements Where are we?

760 views • 38 slides
1. Algorithms for Inverse Reinforcement Learning 2. Apprenticeship learning via Inverse

1. Algorithms for Inverse Reinforcement Learning 2. Apprenticeship learning via Inverse

1. Algorithms for Inverse Reinforcement Learning 2. Apprenticeship learning via Inverse Reinforcement Learning Algorithms for Inverse Reinforcement Learning Andrew Ng and Stuart Russell Motivation Given: (1) measurements of an agent's

379 views • 34 slides
Learning in Graphical Models Problem Dimensions Model Bayes Nets Markov Nets

Learning in Graphical Models Problem Dimensions Model Bayes Nets Markov Nets

Learning in Graphical Models Problem Dimensions Model Bayes Nets Markov Nets Structure Known Unknown (structure learning) Data Complete Incomplete (missing values or hidden variables) Expectation-Maximization

814 views • 28 slides
Learning the Structure of Mixed Graphical Models Jason Lee with Trevor Hastie, Michael Saunders,

Learning the Structure of Mixed Graphical Models Jason Lee with Trevor Hastie, Michael Saunders,

Learning the Structure of Mixed Graphical Models Jason Lee with Trevor Hastie, Michael Saunders, Yuekai Sun, and Jonathan Taylor Institute of Computational & Mathematical Engineering Stanford University June 26th, 2014 Examples of

581 views • 45 slides
ECE 6504: Advanced Topics in Machine Learning Probabilistic Graphical Models and Large-Scale

ECE 6504: Advanced Topics in Machine Learning Probabilistic Graphical Models and Large-Scale

ECE 6504: Advanced Topics in Machine Learning Probabilistic Graphical Models and Large-Scale Learning Topics Bayes Nets (Finish) Parameter Learning Structure Learning Readings: KF 18.1, 18.3; Barber 9.5, 10.4 Dhruv Batra Virginia

351 views • 32 slides
Probabilistic Graphical Models Probabilistic Graphical Models introduction to learning Siamak

Probabilistic Graphical Models Probabilistic Graphical Models introduction to learning Siamak

Probabilistic Graphical Models Probabilistic Graphical Models introduction to learning Siamak Ravanbakhsh Fall 2019 Learning objectives Learning objectives different goals of learning a graphical model effect of goals on the learning setup

273 views • 23 slides
Autonomous Navigation CSE 571 Inverse Optimal Control (Inverse Reinforcement Learning) Many

Autonomous Navigation CSE 571 Inverse Optimal Control (Inverse Reinforcement Learning) Many

5/25/20 Autonomous Navigation CSE 571 Inverse Optimal Control (Inverse Reinforcement Learning) Many slides by Drew Bagnell Carnegie Mellon University 1 2 Optimal Control Solution X X Learning Y Y Cost Map (Sensor Data) (Input)

489 views • 10 slides
Screening Rules for Lasso with Non-Convex Sparse Regularizers Joseph Salmon

Screening Rules for Lasso with Non-Convex Sparse Regularizers Joseph Salmon

Screening Rules for Lasso with Non-Convex Sparse Regularizers Joseph Salmon http://josephsalmon.eu Universit de Montpellier Joint work with A. Rakotomamonjy and G. Gasso 1 / 18 Motivation and objective Lasso and screening Learning

440 views • 24 slides
Graphical Models Independence & Factorization Including structure Complexity of

Graphical Models Independence & Factorization Including structure Complexity of

Graphical Models slides adopted from Sandro Schnborn 1 Graphical Models Independence & Factorization Including structure Complexity of multivariate problems Independence assumptions Graphical Models Graphs to depict

385 views • 23 slides
ECE 6504: Advanced Topics in Machine Learning Probabilistic Graphical Models and Large-Scale

ECE 6504: Advanced Topics in Machine Learning Probabilistic Graphical Models and Large-Scale

ECE 6504: Advanced Topics in Machine Learning Probabilistic Graphical Models and Large-Scale Learning Topics Bayes Nets (Finish) Structure Learning Readings: KF 18.4; Barber 9.5, 10.4 Dhruv Batra Virginia Tech Administrativia HW1

596 views • 30 slides
Probabilistic Graphical Models Probabilistic Graphical Models Parameter learning in Bayesian

Probabilistic Graphical Models Probabilistic Graphical Models Parameter learning in Bayesian

Probabilistic Graphical Models Probabilistic Graphical Models Parameter learning in Bayesian networks Siamak Ravanbakhsh Fall 2019 Learning objectives Learning objectives likelihood function and MLE role of the sufficient statistics MLE for

644 views • 60 slides
Kelly OBrien, Sandra Gardner, Ahmed Bayoumi, Sergio Rueda, Curtis Cooper, Trevor Hart, Patty

Kelly OBrien, Sandra Gardner, Ahmed Bayoumi, Sergio Rueda, Curtis Cooper, Trevor Hart, Patty

Relationships Between Dimensions of Disability Experienced by Adults Living with HIV: A Structural Equation Model Analysis using data from the OCS Study International Forum on HIV and Rehabilitation Research Thursday June 13 th , 2013 Kelly

416 views • 19 slides
Polynomial systems of graphical models Elizabeth Gross University of Hawaii at M anoa

Polynomial systems of graphical models Elizabeth Gross University of Hawaii at M anoa

Polynomial systems of graphical models Elizabeth Gross University of Hawaii at M anoa ICERM Nonlinear Algebra, Graphical Models Working Group: Bibhas Adhikari, Alexandros Grosdos, Marc H ark onen, Cvetelina Hill, Sara Lamboglia,

420 views • 24 slides
A Unified Structural Equation Modeling Approach for the Decomposition of Rank-Dependent

A Unified Structural Equation Modeling Approach for the Decomposition of Rank-Dependent

A Unified Structural Equation Modeling Approach for the Decomposition of Rank-Dependent Indicators of Socioeconomic Inequality of Health Roselinde Kessels & Guido Erreygers Friday 5 September 2014 Socioeconomic Inequality of Health

419 views • 26 slides
Nonrecursive Model for Peer-Influences Data Variables in the Model The R Statistical Computing

Nonrecursive Model for Peer-Influences Data Variables in the Model The R Statistical Computing

x 3 x 1 x 4 y 5 y 6 e 7 e 8 s 78 s 14 g 51 g 52 g 63 g 64 b 56 b 65 x 2 Nonrecursive Model for Peer-Influences Data Variables in the Model The R Statistical Computing Environment A nonrecursive model, from Duncan, Haller, and Portess (1968)

95 views • 7 slides
Type-Based Structural Analysis for Modular Systems of Equations Linkping University, 19 June

Type-Based Structural Analysis for Modular Systems of Equations Linkping University, 19 June

Type-Based Structural Analysis for Modular Systems of Equations Linkping University, 19 June 2014 Henrik Nilsson Joint work with John Capper School of Computer Science University of Nottingham Type-Based Structural Analysis for Modular

1.07k views • 105 slides
using Stan mc-stan.org About the speaker Robert Grant is senior lecturer in health & social

using Stan mc-stan.org About the speaker Robert Grant is senior lecturer in health & social

An introduction to Bayesian computation & evidence synthesis using Stan mc-stan.org About the speaker Robert Grant is senior lecturer in health & social care statistics at Kingston University & St George's, University of London,

3.21k views • 30 slides
E2E Provisioning Workshop Dr. Jan Gruntord CEO CESNET, Czech Republic, Member of GN3 Executive

E2E Provisioning Workshop Dr. Jan Gruntord CEO CESNET, Czech Republic, Member of GN3 Executive

E2E Provisioning Workshop Dr. Jan Gruntord CEO CESNET, Czech Republic, Member of GN3 Executive Committee jan.gruntorad@cesnet.cz E2E Provisioning Workshop Prague, November 29 - 30, 2010 Presentation Overview 1. CESNET Basic Data 2.

648 views • 15 slides
Smart Regions Smart Solutions A place-based approach to decarbonisation: can Horizon Europe

Smart Regions Smart Solutions A place-based approach to decarbonisation: can Horizon Europe

Smart Regions Smart Solutions A place-based approach to decarbonisation: can Horizon Europe meet th the expectations? Cities Northern Netherlands 4 th Floor 19 June 2019, 14:00-15:30 @ERRINNetwork www.errin.eu AGENDA OF THE MEETING

195 views • 6 slides

More recommend