Sparse Logistic Regression Learns All Discrete Pairwise Graphical - - PowerPoint PPT Presentation

▶

Oct 05, 2022 281 likes •396 views

Sparse Logistic Regression Learns All Discrete Pairwise Graphical Models Shanshan Wu , Sujay Sanghavi, Alex Dimakis University of Texas at Austin Graphical models are used to describe complex dependency structures This is a Markov model

SLIDE 1

Sparse Logistic Regression Learns All Discrete Pairwise Graphical Models

Shanshan Wu, Sujay Sanghavi, Alex Dimakis University of Texas at Austin

SLIDE 2

Graphical models are used to describe complex dependency structures

J. Guo et al., “Estimating heterogeneous graphical models for discrete data with an application to roll call voting”. Annuals of Statistics, 2015.
H. Kamisetty et al., “Free Energy Estimates of All-atom Protein Structures Using Generalized Belief Propagation”, RECOMB 2007

Social network analysis Natural language processing Biology

This is a Markov model

SLIDE 3

Discrete pairwise graphical model

Binary case (aka Ising model):
Non-binary case (alphabet size 𝑙): 𝑎 ∈ 1,2, … , 𝑙 (

𝑗 𝑘

𝐵,- ∈ ℝ

An undirected graph on 𝑜 nodes ℙ 𝑎 = 𝑨 ∝ exp( 8

9:,;-:(

𝐵,-𝑨,𝑨- + 8

,∈[(]

𝜄,𝑨,) A distribution over 𝑎 ∈ −1,1 ( External field Edge weight b/t 𝑗 & 𝑘

SLIDE 4

The structure learning problem

[-1 1 -1 -1 -1 1 …… 1] [1 -1 -1 1 -1 -1 …… 1] … 𝑎, 𝑎

𝑎9 𝑎B 𝑎C 𝑎D 𝑎E 𝑎F …… 𝑎(

Given: i.i.d. samples from an unknown graphical model Goal: Recover the graph, i.e., identify the edges

Sample 1 Sample 2

…

Algorithms: Ravikumar et al.’2010, Jalali et al.’2011, Bresler’2015, Vuffray et

al.’2016, Lokhov et al.’2018, Hamilton et al.’2017, Klivans and Meka’2017, Rigollet and Hütter’2019, Vuffray et al.’2019 …

SLIDE 5

A simple approach…

Maximize the conditional log-likelihood ℓ9-regularized logistic regression [Ravikumar et al.’10] ℓB,9-regularized logistic regression [Jalali et al.’11] Binary case Non-binary case

SLIDE 6

Limitation of [Ravikumar et al.’10, Jalali et al.’11]

Assuming that the graphical models satisfy an incoherence condition, sparse logistic regression provably recover the graph structure.

SLIDE 7

Our contribution

Assuming that the graphical models satisfy an incoherence condition, For all graphical models, sparse logistic regression provably recover the graph structure.

SLIDE 8

Our contribution

Let 𝑜 = # variables, alphabet size 𝑙, width 𝜇, minimum edge weight 𝜃

Algorithm Sample complexity Greedy algorithm [Hamilton et al.’17] 𝑃(exp( 𝑙K L exp 𝑒B𝜇 𝜃K 9 )ln(𝑜𝑙)) Sparsitron [Klivans and Meka’17] 𝑃(𝜇B𝑙E exp 14𝜇 𝜃D ln 𝑜𝑙 𝜃 ) ℓB,9-constrained logistic regression [Our work] 𝑃(𝜇B𝑙D exp 14𝜇 𝜃D ln 𝑜𝑙 )

Improves from 𝑙E to 𝑙D!

SLIDE 9

Experiments (grid graph)

Sparse logistic regression requires fewer samples for graph recovery.

Sparsitron Logistic reg Sparsitron Sparsitron Logistic reg Logistic reg

SLIDE 10

Sparse Logistic Regression Learns All Discrete Pairwise Graphical - - PowerPoint PPT Presentation

Sparse Logistic Regression Learns All Discrete Pairwise Graphical Models

Shanshan Wu, Sujay Sanghavi, Alex Dimakis University of Texas at Austin

Graphical models are used to describe complex dependency structures

Discrete pairwise graphical model

The structure learning problem

A simple approach…

Maximize the conditional log-likelihood ℓ9-regularized logistic regression [Ravikumar et al.’10] ℓB,9-regularized logistic regression [Jalali et al.’11] Binary case Non-binary case

Limitation of [Ravikumar et al.’10, Jalali et al.’11]

Assuming that the graphical models satisfy an incoherence condition, sparse logistic regression provably recover the graph structure.

Our contribution

Assuming that the graphical models satisfy an incoherence condition, For all graphical models, sparse logistic regression provably recover the graph structure.

Our contribution

Improves from 𝑙E to 𝑙D!

Experiments (grid graph)

Sparse logistic regression requires fewer samples for graph recovery.

Poster #183

183

Today 10:45 AM -- 12:45 PM @East Exhibition Hall B + C