Graphite: Iterative Generative Modeling of Graphs Aditya Grover , - - PowerPoint PPT Presentation

Graphite: Iterative Generative Modeling of Graphs

Aditya Grover, Aaron Zweig, Stefano Ermon Computer Science Department Stanford University

Graphs are ubiquitous

Graphite: Iterative Generative Modeling of Graphs

Social, biological, information networks etc. How do we learn representations of nodes in a graph? Useful for several prediction tasks. E.g., friendship links on social networks (link prediction), living status of organisms in ecological networks (node classification)

Latent Variable Model of a Graph

Graphite: Iterative Generative Modeling of Graphs

Z = #$

%

#&

%

#'

%

#(

%

• Graphs are represented as adjacency matrices A ∈ {0,1}0 × 0
• For every node 2, we associate a latent vector representation #3 ∈ ℝ5

A = 1 1 1 1 1 1 1 1

2 3 1 4

Example graph Adjacency matrix Latent feature matrix

Graphite: A VAE for Graphs

Graphite: Iterative Generative Modeling of Graphs

Z A

#$(A |Z)

latent matrix Z ∈ ℝ) × + adjacency matrix A ∈ {0,1}) × ) Decoder: Generate data

Graphite: A VAE for Graphs

Graphite: Iterative Generative Modeling of Graphs

Z A

#$(A |Z) '( Z A)

latent matrix Z ∈ ℝ+ × - adjacency matrix A ∈ {0,1}+ × + Decoder: Generate data Encoder: Infer representations

Graphite: Learning & Inference

Graphite: Iterative Generative Modeling of Graphs

Z A

#$(A |Z) '( Z A)

Given: Dataset of adjacency matrices, )*

Graphite: Learning & Inference

Graphite: Iterative Generative Modeling of Graphs

Z A

#$(A |Z) '( Z A)

Given: Dataset of adjacency matrices, )* Learning objective: max

$,( ELBO(3, 4; D*)

Graphite: Learning & Inference

Graphite: Iterative Generative Modeling of Graphs

Z A

#$(A |Z) '( Z A)

Given: Dataset of adjacency matrices, )* Learning objective: max

$,( ELBO(3, 4; D*)

Test time use cases Generative modeling tasks

• Density estimation, clustering nodes,

compressing graphs etc. Graph tasks

• Link Prediction: Denoise graph
• Semi-supervised node classification: Feed 89 for

labelled nodes to a classifier

Parameterizing Graph Autoencoders

Graphite: Iterative Generative Modeling of Graphs

Z A

#$ Z A)

Encoding #$ Z A): Graph Neural Network (GNN) GNN

Parameterizing Graph Autoencoders

Graphite: Iterative Generative Modeling of Graphs

Z A

#$(A |Z) '( Z A)

Encoding '( Z A): Graph Neural Network (GNN) Decoding #$(A |Z): Challenging to “upsample” graphs given latent representations ? GNN

Decoding Graphs - MLP

Graphite: Iterative Generative Modeling of Graphs

Z A

#$(A |Z)

Option 1: Multi-layer Perceptrons (MLP) '(()* + *,) total parameters for single hidden layer of width *

Z ∈ ℝ0 × 2 A ∈ {0,1}0 × 0

Simonovsky et al., 2018

MLP

Decoding Graphs - RNN

Graphite: Iterative Generative Modeling of Graphs

Option 2: Recurrent Neural Network (RNN) Arbitrary ordering of nodes required for training e.g., BFS, DFS

You et al., 2018

Z A

#$(A |Z)

Z ∈ ℝ) × + A ∈ {0,1}) × )

RNN

Graphite – Decoding Graphs using GNN

Graphite: Iterative Generative Modeling of Graphs

Key idea Learn the low-rank structure of adjacency matrix A in the latent space Z

Z A

#$(A |Z)

Z ∈ ℝ) × + A ∈ {0,1}) × )

GNN

Graphite – Decoding Graphs using GNN

Graphite: Iterative Generative Modeling of Graphs

• For fixed number of iterations:

Step 1 (Low rank matrix reconstruction) Map Z to an intermediate graph ! A via an inner product ! A ≈ ZZ%

Z A

&'(A |Z)

Z ∈ ℝ, × . A ∈ {0,1}, × ,

GNN

Graphite – Decoding Graphs using GNN

Graphite: Iterative Generative Modeling of Graphs

• For fixed number of iterations:

Step 1 (Low rank matrix reconstruction) Map Z to an intermediate graph ! A via an inner product ! A ≈ ZZ% Step 2 (Progressive refinement) Refine Z by message passing over ! A using a GNN Z = GNN)(! A)

Z A

,)(A |Z)

Z ∈ ℝ0 × 2 A ∈ {0,1}0 × 0

GNN

Graphite – Decoding Graphs using GNN

Graphite: Iterative Generative Modeling of Graphs

• For fixed number of iterations:

Step 1 (Low rank matrix reconstruction) Map Z to an intermediate graph ! A via an inner product ! A ≈ ZZ% Step 2 (Progressive refinement) Refine Z by message passing over ! A using a GNN Z = GNN)(! A)

• Output step: Set ,)(A |Z) = Bernoulli(sigmoid(ZZ%))

Z A

,)(A |Z)

Z ∈ ℝ0 × 2 A ∈ {0,1}0 × 0

GNN

Graphite – Decoding Graphs using GNN

Graphite: Iterative Generative Modeling of Graphs

• Unlike MLP, GNN decoder with

single hidden layer of length d has !(dk) parameters

• Unlike RNN, no arbitrary ordering of

input nodes is required

Decoding is also computationally efficient. See paper for details.

Z A

()(A |Z)

Z ∈ ℝ- × / A ∈ {0,1}- × -

GNN

Empirical Results – Density Estimation

Graphite: Iterative Generative Modeling of Graphs 255 260 265 270 275 280

E r d

• s
• …

E g

• R

e g u l a r G e

• m

e … P

• w

e r … B a r a b a s …

Negative log-likelihoods. Lower is better. VGAE Graphite Baseline VGAE [Kipf et al., 2016] GNN Encoder + Non-learned Inner Product Decoder. No iterative refinement.

Empirical Results – Link Prediction

Graphite: Iterative Generative Modeling of Graphs 80 85 90 95 100

SpecCluster DeepWalk node2vec VGAE Graphite

• AUC. Higher is better.

Cora Citeseer Pubmed

Empirical Results – Semi-supervised Node Classification

Graphite: Iterative Generative Modeling of Graphs 55 60 65 70 75 80 85

SemiEmb DeepWalk ICA Planetoid GCN Graphite

Percentage accuracy. Higher is better. Cora Citeseer Pubmed

Summary

Graphite: Iterative Generative Modeling of Graphs

Graphite: A latent variable generative model for graphs where both encoder and decoder are parameterized by graph neural networks.

• Encoder performs message passing on input graph
• Decoder iteratively refines inner product graphs

For more details, please visit us at Poster #7. Code: https://github.com/ermongroup/graphite

Z A

#$(A |Z) '( Z A)

GNN GNN