Negative Dependence, Stable Polynomials etc in ML Part 1 STEFANIE - - PowerPoint PPT Presentation

Part 1 STEFANIE JEGELKA & SUVRIT SRA

Dept of EECS & CSAIL

Massachusetts Institute of Technology

ml.mit.edu

Neural information Processing Systems, 2018

Negative Dependence, 
 Stable Polynomials etc in ML

Negative Dependence

Discrete probability measure
 Equivalently: binary random variables


1 1 1

negative dependence

µ(S), S ⊆ V

n

Xi

P( ∈ S | ∈ S) ≤ P( ∈ S)

S

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 3

Negative dependence - where?

Priors on diversity Interpretability

Basal-like HER2 LumA Basal-like HER2 LumA

Negative Dependence Indian Buffet Process

(Xu, Müller, Telesca 2015)

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 4

Negative dependence - where?

Compressing data and models

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 5

Negative dependence - where?

Exploration, active learning, experimental design, Bayesian Optimization

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu)

Outline

2 1

Theory & Applications

Learning a DPP (and some variants) Approximating partition functions Applications Perspectives and wrap-up

Intro & Theory

Introduction


Prominent example: Determinantal Point Processes

Stronger notions of negative dependence Implications: Sampling

—BREAK—

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 7

Capturing Diversity: Determinantal Point Processes

µ(S) ∝ Vol2({vi}i∈S)

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 8

Determinantal Point Processes

1 2 3 4 5 6 7 8 9 10

1 2 3 4 5 6 7 8 9 10

PSD similarity matrix L

LS

µ(S) ∝ det(LS)

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 9

Determinantal Point Processes (DPPs)

1 2 3 4 5 6 7 8 9 10

1 2 3 4 5 6 7 8 9 10

PSD similarity matrix L

LS

µ(S) = det(LS) det(L + I)

✦ Macchi 1975: “fermion processes” ✦ Borodin & Olshanski 2000: “Determinantal Point 


Process”

✦ Introduction to ML: Kulesza & Taskar

(Hough, Krishnapur, Peres, Virag 2006; Lyons 2014; Lyons & Peres 2016; Pemantle 2000)

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 10

Combinatorial Examples

random spanning trees ascents in random sequences

0 2 8 0 9 7 4 5 2 4 9 5 5 2 4

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 11

DPP samples

lij = exp(

1 2σ2 kxi xjk2)

DPP uniform

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 12

Repulsion in nature

termite mounds in Brazil

(Martin, Funch, Hanson, Yoo, Current Biology 2018, 
 http://djalil.chafai.net/blog/2018/11/23/yet-another-determinantal-point-process-in-nature/, )

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu)

Outline

1

Intro & Theory

Introduction


Examples
 Determinantal Point Processes

Stronger notions of negative dependence


Relations to polynomials

Implications: Sampling

Negative Dependence

Discrete probability measure
 Equivalently: binary random variables


1 1 1

negative dependence

n

µ(S), S ⊆ V

Xi

S

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu)

I

J

15

Stronger Notions

EXiXj ≤ EXi EXj

equivalently:

S

Negative Association For all monotone increasing functions G(S), H(S)

E[G(S ∩ I)H(S ∩ J )] ≤ EG(S ∩ I) EH(S ∩ J )

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 16

Example

if thirsty researcher goes to coffee shop i


• therwise

Bi = ( 1

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 17

Detour: Positive Dependence

Positive Association:
 For all monotone increasing functions

E[G(S)H(S)] ≥ EG(S) EH(S)

G(S), H(S)

Analog does not hold for negative dependence!


negative association implies log-submodularity, but not conversely

FKG Lattice Condition, Multivariate Totally Positive:
 log-supermodularity 
 
 
 implies positive association (Fortuin,Kasteleyn,Ginibre 1971) µ(S)µ(T) ≤ µ(S ∪ T)µ(S ∩ T) ∀S, T ⊆ V

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu)

The “right notion” should imply:

✦ Negative Association ✦ Stochastic Covering ✦ Log-concave rank sequences ✦ Closed under “natural” operations ✦ conditioning ✦ marginalization ✦ products …

ak = P(|S| = k)

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 19

Generating Polynomial

qµ(z) = X

S⊆V

µ(S) Y

i∈S

zi, z ∈ Cn

Operations on polynomial = operations on distribution Example: 
 2 items

qµ(z) = µ∅ + µ1z1 + µ2z2 + µ1,2z1z2

Obtain coefficient µ1 = µ({1})

∂ ∂z1 qµ(z) = µ1 + µ1,2z2

• 1. differentiate wrt :

z1

• 2. set z = 0

∂ ∂z1 qµ(z)

• z=0

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu)

P(1 ∈ S)

P(1 / ∈ S)

20

Generating Polynomial

qµ(z) = X

S⊆V

µ(S) Y

i∈S

zi, z ∈ Cn

Example: 
 2 items

qµ(z) = µ∅ + µ1z1 + µ2z2 + µ1,2z1z2

Marginalization: π({1}) = P(1 ∈ S)

Set z2 = 1

qµ(z1, 1) = [µ∅ + µ2] + [µ1 + µ1,2]z1

= qπ(z1)

Properties of polynomial Properties of distribution

⇔

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 21

Strongly Rayleigh Measures

qµ(z) = X

S⊆V

µ(S) Y

i∈S

zi, z ∈ Cn

is Strongly Rayleigh (SR) if is real stable: qµ(z)

µ(S)

Im(zi) > 0 8i ) qµ(z) 6= 0

SR implies almost all conditions laid out by Pemantle!


(Borcea, Bränden, Liggett 2009) Determinantal Point Process: essentially multivariate 
 variant of the characteristic polynomial

qµ(z)

det(ziI − L)

LS

Operations on polynomial Operations on distribution

⇔

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 22

Real Stable Polynomials

Im(zi) > 0 8i ) qµ(z) 6= 0

✦ Deep mathematical connections ✦ Univariate case goes back to Newton (at least) ✦ Powerful class of closure properties

∂q(x) ∂zi ∂q(x) ∂zj ≥ q(x)∂2q(x) ∂zi∂zj , x ∈ Rn

Generating polynomial q(z) is SR is equivalent to ∀i, j

⇒ µ(S)µ(T) ≥ µ(S ∪ T)µ(S ∩ T) ∀S, T ⊆ V

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 23

Nice properties of SR

✦ Closed under marginalization, conditioning on
 … |S| = k, Xi = 1, Xi = 0

✦ Implies many other types of negative dependence, 
 e.g. negative association and E Y

i

Xi ≤ Y

i

EXi

✦ Concentration of measure: behave like 
 independent random variables
 sum , Lipschitz functions F(S), matrices


(Panconesi,Srinivasan 1997, Dubhashi,Ranjan 1998, Farcomeni 2008, 
 Pemantle,Peres 2011, Kyng,Song 2018, Garbe,Vondrak 2018,…)

Xi

Xn

i=1 Xi

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 24

Algorithmic Implications

✦ Approximation Algorithms


(Panconesi,Srinivasan 1997, Hayes 2003, Considine,Byers,Mitzenmacher 
 2004, Asadpour,Goemans,Madry,Oveis-Gharan,Saberi 2010, Gharan, 
 Saberi,Singh 2011, Spielman,Srivastava 2011, …)

✦ Approximate partition functions, permanents, volumes,

counting


(Gurvits 2006, Nikolov-Singh 2016, Straszak-Vishnoi 2016, Anari,Gharan, Saberi,Singh 2016, Anari,Gharan 2017…)

✦ Sampling


(Féder,Mihail 1992, Jerrum,Son 2002, Jerrum,Son,Tetali,Vigoda 2004, 
 Anari,Gharan,Rezaei 2016, Li,Jegelka,Sra 2016)

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 25

Other SR measures

NOT a DPP

P(S) ∝ det(ASA>

S )

X Dual Volume Sampling A

… but Strongly Rayleigh!

(Avron,Boutsidis 2013, Li,Jegelka,Sra 2017, Derezinski,Warmuth 2017)

proof: closeness properties of polynomials

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 26

… and limits

is SR. What about ?


(Kulesza,Taskar 2012, Zou,Adams 2012, Gillenwater 2014, Anari,
 Gharan 2017, Mariet,Sra,Jegelka 2018)

µ(S)

µ(S)p

In general not SR! Conditioning on combinatorial constraints: at most 


• ne item per group.


(Celis,Deshpande,Kathuria,Straszak,Vishnoi 2017, Celis,Keswani, 
 Straszak,Deshpande,Kathuria,Vishnoi 2018)

In general not SR!

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu)

Outline

1

Intro & Theory

Introduction


Examples
 Determinantal Point Processes

Stronger notions of negative dependence


Relations to polynomials
 Strongly Rayleigh measures

Implications: Sampling

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 28

Sampling for Determinantal Point Processes

EVD/SVD (Hough, Krishnapur, Peres, Virág 2006)

✦ sample subspace via eigenvalues ✦ sequentially sample items

LS

Faster via A: “dual” sampling (Kulesza,Taskar 2010)

L = A>A

n × d

O(n3)

Acceleration: approximate L or A


volume-preserving sketching (Magen,Zouzias 2008; Deshpande, Rademacher 2010, Gillenwater, Kulesza, Taskar 2012) Nyström (Affandi, Kulesza, Fox, Taskar 2013) MCMC (Bardenet,Hardy 2016) Coresets (Li,Jegelka,Sra 2016) R-DPP (Derezinski 2018)

Other approaches (Derezinski, Warmuth 2018,Derezinski, Warmuth, 


Hsu 2018, Mariet, Sra 2016,…)

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 29

Sampling for general SR measures

Markov Chain Monte Carlo (MCMC) a b

if conditioning on : only swaps

|S| = k

in iteration t:
 sample , uniformly at random

a ∈ St−1 b / ∈ St−1

randomly do one of

• swap: remove b, add a
• add b
• remove a
• retain St = St−1

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 30

Sampling empirically

# iters 1000 2000 3000 4000 5000 Fnorm error 0.01 0.02 0.03 Size=4000

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 31

How many iterations?

Mixing time: #iterations until kµt µkTV  ✏

balanced matroids:


(Féder,Mihail 1992, Jerrum,Son 2002 
 Jerrum,Son,Tetali,Vigoda, 2004)

fixed-cardinality SR:


(Anari,Oveis-Gharan,Rezaei 2016)

arbitrary SR:


(Li,Jegelka,Sra 2016)

|S| = k

}

O

• n2 log(

1 ✏µ(S0))

• <latexit sha1_base64="kCf8r1KNEF0lOl2YCsA2umz/kZ4=">ACG3icbVDLSgMxFM3UV62vUZdugkWYbspMEXRZdOPOivYBnTpk0kwbmkmGJCOUof/hxl9x40IRV4IL/8a0nYW2HgczjmXm3vChFGlXfbKqysrq1vFDdLW9s7u3v2/kFLiVRi0sSCdkJkSKMctLUVDPSRBchIOxdTv32A5GKCn6nxwnpxWjAaUQx0kYK7Nq1H9KBw+9r0Gdi4EBfRxLhzJtkPkUZYJDP06d28CtTCrTbCWwy27VnQEuEy8nZCjEdifl/gNCZcY4aU6npuonsZkpiRiYlP1UkQXiEBqRrKEcxUb1sdtsEnhilDyMhzeMaztTfExmKlRrHoUnGSA/VojcV/O6qY7OexnlSaoJx/NFUcqgFnBaFOxTSbBmY0MQltT8FeIhMt1oU2fJlOAtnrxMWrWqZ/jNabl+kdRBEfgGDjA2egDq5AzQBo/gGbyCN+vJerHerY95tGDlM4fgD6yvH8RBn/w=</latexit><latexit sha1_base64="kCf8r1KNEF0lOl2YCsA2umz/kZ4=">ACG3icbVDLSgMxFM3UV62vUZdugkWYbspMEXRZdOPOivYBnTpk0kwbmkmGJCOUof/hxl9x40IRV4IL/8a0nYW2HgczjmXm3vChFGlXfbKqysrq1vFDdLW9s7u3v2/kFLiVRi0sSCdkJkSKMctLUVDPSRBchIOxdTv32A5GKCn6nxwnpxWjAaUQx0kYK7Nq1H9KBw+9r0Gdi4EBfRxLhzJtkPkUZYJDP06d28CtTCrTbCWwy27VnQEuEy8nZCjEdifl/gNCZcY4aU6npuonsZkpiRiYlP1UkQXiEBqRrKEcxUb1sdtsEnhilDyMhzeMaztTfExmKlRrHoUnGSA/VojcV/O6qY7OexnlSaoJx/NFUcqgFnBaFOxTSbBmY0MQltT8FeIhMt1oU2fJlOAtnrxMWrWqZ/jNabl+kdRBEfgGDjA2egDq5AzQBo/gGbyCN+vJerHerY95tGDlM4fgD6yvH8RBn/w=</latexit><latexit sha1_base64="kCf8r1KNEF0lOl2YCsA2umz/kZ4=">ACG3icbVDLSgMxFM3UV62vUZdugkWYbspMEXRZdOPOivYBnTpk0kwbmkmGJCOUof/hxl9x40IRV4IL/8a0nYW2HgczjmXm3vChFGlXfbKqysrq1vFDdLW9s7u3v2/kFLiVRi0sSCdkJkSKMctLUVDPSRBchIOxdTv32A5GKCn6nxwnpxWjAaUQx0kYK7Nq1H9KBw+9r0Gdi4EBfRxLhzJtkPkUZYJDP06d28CtTCrTbCWwy27VnQEuEy8nZCjEdifl/gNCZcY4aU6npuonsZkpiRiYlP1UkQXiEBqRrKEcxUb1sdtsEnhilDyMhzeMaztTfExmKlRrHoUnGSA/VojcV/O6qY7OexnlSaoJx/NFUcqgFnBaFOxTSbBmY0MQltT8FeIhMt1oU2fJlOAtnrxMWrWqZ/jNabl+kdRBEfgGDjA2egDq5AzQBo/gGbyCN+vJerHerY95tGDlM4fgD6yvH8RBn/w=</latexit><latexit sha1_base64="kCf8r1KNEF0lOl2YCsA2umz/kZ4=">ACG3icbVDLSgMxFM3UV62vUZdugkWYbspMEXRZdOPOivYBnTpk0kwbmkmGJCOUof/hxl9x40IRV4IL/8a0nYW2HgczjmXm3vChFGlXfbKqysrq1vFDdLW9s7u3v2/kFLiVRi0sSCdkJkSKMctLUVDPSRBchIOxdTv32A5GKCn6nxwnpxWjAaUQx0kYK7Nq1H9KBw+9r0Gdi4EBfRxLhzJtkPkUZYJDP06d28CtTCrTbCWwy27VnQEuEy8nZCjEdifl/gNCZcY4aU6npuonsZkpiRiYlP1UkQXiEBqRrKEcxUb1sdtsEnhilDyMhzeMaztTfExmKlRrHoUnGSA/VojcV/O6qY7OexnlSaoJx/NFUcqgFnBaFOxTSbBmY0MQltT8FeIhMt1oU2fJlOAtnrxMWrWqZ/jNabl+kdRBEfgGDjA2egDq5AzQBo/gGbyCN+vJerHerY95tGDlM4fgD6yvH8RBn/w=</latexit>

O

• nk log(

1 ✏µ(S0))

• <latexit sha1_base64="f9wtpX+ZXUH0+EwDul6CBM8y3k=">ACGnicbVDLSgMxFM3UV62vqks3wSJMN2VGBF0W3bizon1Ap5RMmpmG5jEkGaEM/Q43/obF4q4Ezf+jZm2C209EDicy4394QJo9p43rdTWFldW98obpa2tnd298r7By0tU4VJE0smVSdEmjAqSNQw0gnUQTxkJF2OLrK/fYDUZpKcW/GCelxFAsaUYyMlfpl/yYIaeyKEQyYjF0YmEghnPmTLCJpkwKGPDUvet71Uk1j1b75YpX86aAy8SfkwqYo9EvfwYDiVNOhMEMad31vcT0MqQMxYxMSkGqSYLwCMWka6lAnOheNj1tAk+sMoCRVPYJA6fq74kMca3HPLRJjsxQL3q5+J/XTU10cuoSFJDBJ4tilIGjYR5T3BAFcGjS1BWFH7V4iHyHZjbJslW4K/ePIyaZ3WfMtvzyr1y3kdRXAEjoELfHAO6uAaNEATYPAInsEreHOenBfn3fmYRQvOfOYQ/IHz9QNeDJ/N</latexit><latexit sha1_base64="f9wtpX+ZXUH0+EwDul6CBM8y3k=">ACGnicbVDLSgMxFM3UV62vqks3wSJMN2VGBF0W3bizon1Ap5RMmpmG5jEkGaEM/Q43/obF4q4Ezf+jZm2C209EDicy4394QJo9p43rdTWFldW98obpa2tnd298r7By0tU4VJE0smVSdEmjAqSNQw0gnUQTxkJF2OLrK/fYDUZpKcW/GCelxFAsaUYyMlfpl/yYIaeyKEQyYjF0YmEghnPmTLCJpkwKGPDUvet71Uk1j1b75YpX86aAy8SfkwqYo9EvfwYDiVNOhMEMad31vcT0MqQMxYxMSkGqSYLwCMWka6lAnOheNj1tAk+sMoCRVPYJA6fq74kMca3HPLRJjsxQL3q5+J/XTU10cuoSFJDBJ4tilIGjYR5T3BAFcGjS1BWFH7V4iHyHZjbJslW4K/ePIyaZ3WfMtvzyr1y3kdRXAEjoELfHAO6uAaNEATYPAInsEreHOenBfn3fmYRQvOfOYQ/IHz9QNeDJ/N</latexit><latexit sha1_base64="f9wtpX+ZXUH0+EwDul6CBM8y3k=">ACGnicbVDLSgMxFM3UV62vqks3wSJMN2VGBF0W3bizon1Ap5RMmpmG5jEkGaEM/Q43/obF4q4Ezf+jZm2C209EDicy4394QJo9p43rdTWFldW98obpa2tnd298r7By0tU4VJE0smVSdEmjAqSNQw0gnUQTxkJF2OLrK/fYDUZpKcW/GCelxFAsaUYyMlfpl/yYIaeyKEQyYjF0YmEghnPmTLCJpkwKGPDUvet71Uk1j1b75YpX86aAy8SfkwqYo9EvfwYDiVNOhMEMad31vcT0MqQMxYxMSkGqSYLwCMWka6lAnOheNj1tAk+sMoCRVPYJA6fq74kMca3HPLRJjsxQL3q5+J/XTU10cuoSFJDBJ4tilIGjYR5T3BAFcGjS1BWFH7V4iHyHZjbJslW4K/ePIyaZ3WfMtvzyr1y3kdRXAEjoELfHAO6uAaNEATYPAInsEreHOenBfn3fmYRQvOfOYQ/IHz9QNeDJ/N</latexit><latexit sha1_base64="f9wtpX+ZXUH0+EwDul6CBM8y3k=">ACGnicbVDLSgMxFM3UV62vqks3wSJMN2VGBF0W3bizon1Ap5RMmpmG5jEkGaEM/Q43/obF4q4Ezf+jZm2C209EDicy4394QJo9p43rdTWFldW98obpa2tnd298r7By0tU4VJE0smVSdEmjAqSNQw0gnUQTxkJF2OLrK/fYDUZpKcW/GCelxFAsaUYyMlfpl/yYIaeyKEQyYjF0YmEghnPmTLCJpkwKGPDUvet71Uk1j1b75YpX86aAy8SfkwqYo9EvfwYDiVNOhMEMad31vcT0MqQMxYxMSkGqSYLwCMWka6lAnOheNj1tAk+sMoCRVPYJA6fq74kMca3HPLRJjsxQL3q5+J/XTU10cuoSFJDBJ4tilIGjYR5T3BAFcGjS1BWFH7V4iHyHZjbJslW4K/ePIyaZ3WfMtvzyr1y3kdRXAEjoELfHAO6uAaNEATYPAInsEreHOenBfn3fmYRQvOfOYQ/IHz9QNeDJ/N</latexit>

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 32

Sampling: from fixed-cardinality to general SR

V

Vnew = V ∪ V 0

V 0

T ⊂ Vnew

• sample n out of 2n, but use only T ∩ V

• extend measure to “shadow set”
• Key: measure on 2n is Strongly Rayleigh by 


closedness properties of real stable polynomials
 (“symmetric homogenization”)

(Li, Jegelka, Sra 2016)

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu) 33

Sampling

✦ generic method: MCMC; 
 different algorithms for special cases (DPP , dual volume) ✦ Accelerating each iteration of DPP-MCMC: lazy 
 computations with Gauss quadrature (Li,Sra,Jegelka 2016) ✦ SR sufficient for fast mixing but not necessary: e.g., 
 complete log concavity (Anari,Liu,Gharan,Vinzant 2018) ✦ Sampling log-submodular distributions (negative 
 lattice condition) (Rebeschini,Karbasi 2015, Gotovos,Hassani,


Krause 2018, Gotovos,Hassani,Krause,Jegelka 2018)

✦ Continuous DPP sampling 
 (Affandi,Fox,Taskar 2013, Gharan,Rezaei 2018, …)

Negative dependence, stable polynomials etc. in ML - part 1

Stefanie Jegelka (stefje@mit.edu)

Outline

2 1

Theory & Applications

Learning a DPP (and some variants) Approximating partition functions Applications Perspectives and wrap-up

Intro & Theory

Introduction


Examples, Determinantal Point Processes

Stronger notions of negative dependence
 Strongly Rayleigh measures and real stable polynomials Implications: Sampling

—BREAK—