Stochastic Blockmodels meet Graph Neural Networks Nikhil Mehta - PowerPoint PPT Presentation

Stochastic Blockmodels meet Graph Neural Networks Nikhil Mehta Lawrence Carin Piyush Rai I nternational Conference on Machine Learning (ICML) 2019, Long Beach, CA

Problem Statement ➢ Goal: Learn sparse node embeddings for graphs. ➢ Motivation: ▪ Can be used for downstream machine learning tasks – link/edge prediction, node classification, community discovery. ➢ Some notation ▪ Consider a graph associated with an adjacency matrix: 𝑂×𝑂 𝐵 ∈ 0,1 ▪ Additional side information associated with each node: 𝑌 ∈ R 𝑂×𝐸 Stochastic blockmodels meet Graph Neural Networks

Some Existing Work • Probabilistic Methods: • A simple class of models: Stochastic Block Models (SBM) [Nowicki & Snijders, 2001] 𝑈 𝑋𝑨 𝑨 𝑗 ∼ 𝑁𝑣𝑚𝑢𝑗𝑜𝑝𝑣𝑚𝑚𝑗 𝜌 𝐵 𝑗,𝑘 ∼ 𝐶𝑓𝑠𝑜𝑝𝑣𝑚𝑚𝑗 𝑨 𝑗 𝑘 • Overlapping SBM (OSBM) [Miller et al., 2009] – participation in multiple cochromemmunities. • Latent Feature Relational Model (LFRM), 𝑨 𝑗 ∈ 0,1 𝐿 𝐿 → ∞ 2 ; 𝐵 𝑗,𝑘 ∼ 𝐶𝑓𝑠𝑜𝑝𝑣𝑚𝑚𝑗(𝜏(𝑨 𝑗 𝑈 Λ𝑨 𝑎 ∼ 𝐽𝐶𝑄(𝛽) ; 𝜇 𝑙,𝑙` ∼ 𝒪 0, 𝜏 𝑘 )) 𝜇 • Can handle uncertainty & missing data better. ☺ • Interpretability can be achieved by suitable choice of prior. ☺ • Uses iterative inference methods (MCMC, VB), not easy to scale.  • What about Variational Graph Autoencoder (GVAE) [Kipf & Welling, 2016] ? • Encoder – Graph Convolutional Network (GCN) 𝑈 𝑨 • Decoder – Link prediction: 𝜏 𝑨 𝑗 𝑘 or Node classification: softmax(g(z)) • Fast and scalable ☺ • Generative method + Uses deep NN = Best of both worlds? No • Embeddings are often not interpretable.  • What should be the size of the latent space?  Stochastic blockmodels meet Graph Neural Networks

Deep Generative LFRM • We propose DGLFRM – Deep Generative Model for Graphs • Unification: Interpretability of SBM + fast inference via Graph Neural Network. • Node embedding ( 𝑨 𝑜 ) is the element wise product of two other latent variables: 𝑨 𝑜 = 𝑐 𝑜 ⊙ 𝑠 𝑜 . • 𝑐 𝑜 ∈ 0,1 𝐿 defines the node-community memberships (cluster assignments). This allows the model to infer the “active communities” for a given ( 𝐿 ). 𝑜 ∈ ℝ 𝐿 defines the node-community membership • 𝑠 strength. Stochastic blockmodels meet Graph Neural Networks

Deep Generative LFRM Generative Story • Membership vector ( 𝑐 𝑜 ∈ 0,1 𝐿 ) • Stick-breaking IBP • 𝑤 𝑙 ∼ 𝐶𝑓𝑢𝑏 𝛽, 1 , 𝑙 = 1,2, … , 𝐿 𝑙 • 𝜌 𝑙 = ς 𝑘=1 𝑤 𝑘 , 𝑐 𝑜𝑙 ∼ 𝐶𝑓𝑠𝑜𝑝𝑣𝑚𝑚𝑗(𝜌 𝑙 ) • Membership Strength ( 𝑠 𝑜 ∈ ℝ 𝐿 ) • 𝑠 𝑜 ∼ 𝒪(0,1) • Node embedding: ( 𝑨 𝑜 = 𝑐 𝑜 ⊙ 𝑠 𝑜 ) • 𝑔 𝑜 = 𝑔(𝑨 𝑜 ) , where 𝑔 is a multi-layered perceptron. • 𝑞 𝐵 𝑜𝑛 𝑔 𝑈 𝑔 𝑜 , 𝑔 𝑛 ) = 𝜏(𝑔 𝑛 ) 𝑜 • Posterior: p (𝑤, 𝑐, 𝑠|𝐵, 𝑌) Stochastic blockmodels meet Graph Neural Networks

Deep Generative LFRM Inference Network • Full mean-field approximation: Approximate the true posterior with the variational posterior. 𝐿 𝑂 • 𝑟 𝜚 𝑤, 𝑐, 𝑠 = ς 𝑙=1 ς 𝑜=1 𝑟 𝜚 𝑤 𝑜𝑙 𝑟 𝜚 𝑐 𝑜𝑙 𝑟 𝜚 (𝑠 𝑜𝑙 ) • 𝑟 𝜚 𝑤 𝑜𝑙 = 𝐿𝑣𝑛𝑏𝑠𝑏𝑡𝑥𝑏𝑛𝑧(𝑤 𝑜𝑙 |𝑑 𝑙 , 𝑒 𝑙 ) • 𝑟 𝜚 𝑐 𝑜𝑙 = 𝐶𝑓𝑠𝑜𝑝𝑣𝑚𝑚𝑗 𝑐 𝑜𝑙 𝜌 𝑙 • 𝑟 𝜚 𝑠 2 )) 𝑜𝑙 = 𝒪(𝜈 𝑜 , 𝑒𝑗𝑏𝑕(𝜏 𝑜 • Kumaraswamy can be re-parameterized and act as a reasonable approximation for Beta. For Bernoulli, we use continuous relaxation (Concrete Distribution). Stochastic blockmodels meet Graph Neural Networks

Deep Generative LFRM Learning • Since the vanilla mean-field ignores the posterior dependencies among the latent variables, we 𝐿 𝑂 considered Structured Mean-Field: 𝑟 𝜚 𝑤, 𝑐, 𝑠 = ς 𝑙=1 𝑟 𝜚 𝑤 𝑙 ς 𝑜=1 𝑟 𝜚 𝑐 𝑜𝑙 |𝑤 𝑟 𝜚 (𝑠 𝑜𝑙 ) • The only difference from the Mean-field approximation is that 𝑤 is now a global variable (same for all nodes); b nk |v ∼ 𝐶𝑓𝑠𝑜𝑝𝑣𝑚𝑚𝑗(𝜌 𝑙 ) . • We can maximize the following ELBO: 𝑂 𝑂 𝑂 ෍ ෍ (𝔽[log 𝑞 𝜄 (𝐵 𝑜𝑛 |𝑨 𝑜 , 𝑨 𝑛 )]) + ෍ (𝔽[log 𝑞 𝜄 (𝑌 𝑜 |𝑨 𝑜 )]) 𝑜=1 𝑛=1 𝑜=1 𝑂 − σ 𝑜=1 (𝐿𝑀 𝑟 𝜚 𝑐 𝑜 𝑤 𝑜 𝑞 𝜄 𝑐 𝑜 𝑤 𝑜 + 𝐿𝑀 𝑟 𝜚 𝑠 𝑜 𝑞 𝜄 𝑠 + 𝐿𝑀[𝑟 𝜚 (𝑤 𝑜 )|𝑞(𝑤 𝑜 )]) 𝑜 Stochastic blockmodels meet Graph Neural Networks

Results Train network Sparse latent space Generated network w. Masked edges Performance on Link prediction task on five datasets. Stochastic blockmodels meet Graph Neural Networks

Thank you Please come to our poster @ 06:30PM Pacific Ballroom #180 Stochastic blockmodels meet Graph Neural Networks