CMP722 ADVANCED COMPUTER VISION Lecture #9 Graph Networks Aykut - - PowerPoint PPT Presentation

Lecture #9 – Graph Networks

Aykut Erdem // Hacettepe University // Spring 2019

CMP722

ADVANCED COMPUTER VISION

Illustration: Kevin Hong // Quanta Magazine

• image synthesis via generative

models

• conditional generative models
• structured vs unstructured

prediction

• image-to-image translation
• generative adversarial networks
• cycle-consistent adversarial

networks

Previously on CMP722

Illustration: StyleGAN trained on Portrait by Yuli-Ban

Lecture overview

• graph structured data
• graph neural nets (GNNs)
• GNNs for ”classical” network problems
• Disclaimer: Much of the material and slides for this lecture were borrowed from

—Yujia Li and Oriol Vinyals' tutorial on Graph Nets

—Thomas Kipf’s talk on structured deep models: deep Learning on graphs and beyond

3

Deep Learning

Deep neural al nets s that at exp xploit:

• translation equivariance (weight sharing)
• hierarchical compositionality

4

Speech data Natural language processing (NLP) Grid games

Modeling Structured Data

5

• utput

Unstructured Data

sequences visual data

Graph Structured Data Data with Rigid Structure

Modeling Structured Data

6

• utput

Unstructured Data

sequences visual data

Graph Structured Data Data with Rigid Structure

Graph structured data

• A lot of real-world data does not “live” on grids

7

Standard deep learning architectures like CNNs and RNNs don’t work here!

Road maps Protein interaction networks Social networks Citation networks Communication networks Multi-agent systems Molecules Knowledge graphs

Recipe for a good model for graphs

8

• Handle different types of graph prediction problems

Requires: Representations for graphs, nodes and edges

• Handle graphs of varying sizes and structure

Requires: A parametrization independent of graph size and structure

• Handle arbitrary node ordering

Requires: A model invariant to node permutations

• Utilize graph structure

Requires: A mechanism to communicate information on graphs

Graph Neural Networks (GNNs)

• Mai

ain idea: a: Pass messages between pairs of nodes & agglomerate

9

Adjacency matrix Feature matrix

Notation:

Input Output ReLU ReLU Hidden layer Hidden layer

G = (A, X)

A ∈ RN×N

X ∈ RN×F

Recap: Convolutional neural networks (on grids)

10 (Animation by Vincent Dumoulin)

Single CNN layer with 3x3 filter:

Recap: Convolutional neural networks (on grids)

11 (Animation by Vincent Dumoulin)

Single CNN layer with 3x3 filter:

h0

h1

hi

. . .

Recap: Convolutional neural networks (on grids)

12 (Animation by Vincent Dumoulin)

Single CNN layer with 3x3 filter:

h0

h1

hi

. . .

hi ∈ RF

are (hidden layer) activations of a pixel/node

Recap: Convolutional neural networks (on grids)

13 (Animation by Vincent Dumoulin)

Single CNN layer with 3x3 filter:

h0

h1

hi

. . .

hi ∈ RF

are (hidden layer) activations of a pixel/node Up Update fo for a a si single pi pixel:

• Transform messages individually
• Add everything up

Wihi

X

i

Wihi

Recap: Convolutional neural networks (on grids)

14 (Animation by Vincent Dumoulin)

Single CNN layer with 3x3 filter:

h0

h1

hi

. . .

hi ∈ RF

are (hidden layer) activations of a pixel/node Up Update fo for a a si single pi pixel:

• Transform messages individually
• Add everything up

Wihi

X

i

Wihi

Fu Full upda update:

Graph convolutional networks (GCNs)

15

Kipf & Welling (ICLR 2017), related previous works by Duvenaud et al. (NIPS 2015) and Li et al. (ICLR 2016)

Consider this undirected graph:

Graph convolutional networks (GCNs)

16

Kipf & Welling (ICLR 2017), related previous works by Duvenaud et al. (NIPS 2015) and Li et al. (ICLR 2016)

Consider this undirected graph: Consider update for node in red:

Graph convolutional networks (GCNs)

17

Kipf & Welling (ICLR 2017), related previous works by Duvenaud et al. (NIPS 2015) and Li et al. (ICLR 2016)

Consider this undirected graph: Consider update for node in red:

Graph convolutional networks (GCNs)

18

Kipf & Welling (ICLR 2017), related previous works by Duvenaud et al. (NIPS 2015) and Li et al. (ICLR 2016)

Consider this undirected graph: Consider update for node in red:

Up Updat ate e ru rule:

h(l+1)

i

= σ @h(l)

i W(l) 0 +

X

j∈Ni

1 cij h(l)

j W(l) 1

1 A

h(l)

i W(l)

X

j∈Ni

1 cij h(l)

j W(l) 1

Ni :

cij :

neighbor indices

• norm. constant (fixed/trainable)

Graph convolutional networks (GCNs)

19

Kipf & Welling (ICLR 2017), related previous works by Duvenaud et al. (NIPS 2015) and Li et al. (ICLR 2016)

Consider this undirected graph: Consider update for node in red:

Up Updat ate e ru rule:

h(l+1)

i

= σ @h(l)

i W(l) 0 +

X

j∈Ni

1 cij h(l)

j W(l) 1

1 A

h(l)

i W(l)

X

j∈Ni

1 cij h(l)

j W(l) 1

Ni :

cij :

neighbor indices

• norm. constant (fixed/trainable)

Desi sirab able properties: s:

• Weight sharing over all locations
• Invariance to permutations
• Linear complexity O(E)
• Applicable both in transductive

and inductive settings

Graph convolutional networks (GCNs)

20

Kipf & Welling (ICLR 2017), related previous works by Duvenaud et al. (NIPS 2015) and Li et al. (ICLR 2016)

Consider this undirected graph: Consider update for node in red:

Up Updat ate e ru rule:

h(l+1)

i

= σ @h(l)

i W(l) 0 +

X

j∈Ni

1 cij h(l)

j W(l) 1

1 A

h(l)

i W(l)

X

j∈Ni

1 cij h(l)

j W(l) 1

Ni :

cij :

neighbor indices

• norm. constant (fixed/trainable)

Desi sirab able properties: s:

• Weight sharing over all locations
• Invariance to permutations
• Linear complexity O(E)
• Applicable both in transductive

and inductive settings Limitat ations: s:

• Requires gating mechanism /

residual connections for depth

• Only indirect support for edge

features

GCNs with edge embeddings

21

Battaglia et al. (NIPS 2016), Gilmer et al. (ICML 2017), Kipf et al. (ICML 2018)

Fo Formally ally: v → e :hl

(i,j) = f l e

⇥ hl

i, hl j, x(i,j)

⇤ e → v :hl+1

j

= f l

v

⇣hP

i∈Nj hl (i,j), xj

i⌘

GCNs with edge embeddings

22

Battaglia et al. (NIPS 2016), Gilmer et al. (ICML 2017), Kipf et al. (ICML 2018)

Fo Formally ally:

Pros: s:

• Supports edge features
• More expressive than GCN
• As general as it gets (?)
• Supports sparse matrix ops

v → e :hl

(i,j) = f l e

⇥ hl

i, hl j, x(i,j)

⇤ e → v :hl+1

j

= f l

v

⇣hP

i∈Nj hl (i,j), xj

i⌘

GCNs with edge embeddings

23

Battaglia et al. (NIPS 2016), Gilmer et al. (ICML 2017), Kipf et al. (ICML 2018)

Fo Formally ally:

Pros: s:

• Supports edge features
• More expressive than GCN
• As general as it gets (?)
• Supports sparse matrix ops

Cons: s:

• Need to store intermediate

edge-based activations

• Difficult to implement with

subsampling Ø In practice limited to small graphs

v → e :hl

(i,j) = f l e

⇥ hl

i, hl j, x(i,j)

⇤ e → v :hl+1

j

= f l

v

⇣hP

i∈Nj hl (i,j), xj

i⌘

Graph neural networks with atuention

24

Monti et al. (CVPR 2017), Hoshen (NIPS 2017), Veličković et al. (ICLR 2018)

[Figure from Veličković et al. (ICLR 2018)]

Graph neural networks with atuention

25

Monti et al. (CVPR 2017), Hoshen (NIPS 2017), Veličković et al. (ICLR 2018)

[Figure from Veličković et al. (ICLR 2018)]

Graph neural networks with atuention

26

Monti et al. (CVPR 2017), Hoshen (NIPS 2017), Veličković et al. (ICLR 2018)

[Figure from Veličković et al. (ICLR 2018)]

Pros: s:

• No need to store intermediate

edge-based activation vectors (when using dot-product attn.)

• Slower than GCNs but faster than

GNNs with edge embeddings

Graph neural networks with atuention

27

Monti et al. (CVPR 2017), Hoshen (NIPS 2017), Veličković et al. (ICLR 2018)

[Figure from Veličković et al. (ICLR 2018)]

Pros: s:

• No need to store intermediate

edge-based activation vectors (when using dot-product attn.)

• Slower than GCNs but faster than

GNNs with edge embeddings

Cons: s:

• (Most likely) less expressive than

GNNs with edge embeddings

• Can be more difficult to optimize

28

A brief history of graph neural nets

Original GNN


Gori et al. (2005)

GG-NN


Li et al. 
 (ICLR 2016)

Spectral Graph CNN


Bruna et al. (ICLR 2015)

ChebNet


Defferrard et al. (NIPS 2016)

GCN


Kipf & Welling
 (ICLR 2017)

“Spectral methods” “Spatial methods”

GraphSAGE


Hamilton et al. (NIPS 2017)

MoNet


Monti et al.
 (CVPR 2017)

Neural MP


Gilmer et al.
 (ICML 2017)

Relation Nets


Santoro et al. (NIPS 2017)

“DL on graph explosion”

Programs as Graphs


Allamanis et al.
 (ICLR 2018)

NRI

Kipf et al.
 (ICML 2018)

GAT


Veličković et al. (ICLR 2018) Other early work:


• Duvenaud et al. (NIPS 2015)
• Dai et al. (ICML 2016)

• Niepert et al. (ICML 2016)

• Battaglia et al. (NIPS 2016)
• Atwood & Towsley (NIPS 2016)
• Sukhbaatar et al. (NIPS 2016)

… Other ear arly y work: k:

• Duvenaud et al. (NIPS 2015)
• Dai et al. (ICML 2016)
• Niepert et al. (ICML 2016)
• Battaglia et al. (NIPS 2016)
• Atwood & Towsley (NIPS 2016) -

Sukhbaatar et al. (NIPS 2016)

“Spect ctral al methods” s” “Spat atial al methods” s” “DL on grap aph exp xplosi sion”

A brief history of graph neural nets

Applications to “classical” network problems

29

One fits all: Classification and link prediction with GNNs/GCNs

Input Input: Feature matrix , preprocessed adjacency matrix

30

Input Output ReLU ReLU Hidden layer Hidden layer

One fits all: Classification and link prediction with GNNs/GCNs

Input Input: Feature matrix , preprocessed adjacency matrix

31

Input Output ReLU ReLU Hidden layer Hidden layer

No Node cl classi ssifica cation: :

softmax (zn)

e.g. Kipf & Welling (ICLR 2017)

One fits all: Classification and link prediction with GNNs/GCNs

Input Input: Feature matrix , preprocessed adjacency matrix

32

Input Output ReLU ReLU Hidden layer Hidden layer

No Node cl classi ssifica cation: :

softmax (zn)

e.g. Kipf & Welling (ICLR 2017)

Gr Grap aph cl classi ssifica cation: :

softmax (P

n zn)

e.g. Duvenaud et al. (NIPS 2015)

One fits all: Classification and link prediction with GNNs/GCNs

Input Input: Feature matrix , preprocessed adjacency matrix

33

Input Output ReLU ReLU Hidden layer Hidden layer

No Node cl classi ssifica cation: :

softmax (zn)

e.g. Kipf & Welling (ICLR 2017)

Gr Grap aph cl classi ssifica cation: :

softmax (P

n zn)

e.g. Duvenaud et al. (NIPS 2015)

Link k predict ction: :

Kipf & Welling (NIPS BDL 2016) “Grap aph Aut Auto-Enco coders”

p (Aij) = σ

• zT

i zj

• <latexit sha1_base64="wLZF1TLeDRyLIsCNji+W3IeZgIQ=">ACTnicbVFNSysxFM1Ufdaqz6pLN8HyQDdlRoX3NoIfG5c+aKvQqUMmvdPGZj5I7gh1yC90I+78GW5cKA9NPxSfeiBwcs69yc1JmEmh0XvndLM7NyP+fJCZXFp+edKdXWtpdNcWjyVKbqPGQapEigiQIlnGcKWBxKOAsHxyP/7AqUFmnSwGEGnZj1EhEJztBKQRUyX0KEW4dBIeil8ZXo9XF739eiF7OJ5cM+2FUXJugeOPCmIv3TcMY+l3RpXk7L6jW3Lo7Bv1KvCmpkSlOg+qd3015HkOCXDKt256bYadgCgWXYCp+riFjfMB60LY0YTHoTjGOw9BfVunSKFV2JUjH6seOgsVaD+PQVo4m1Z+9kfid184x+tMpRJLlCAmfXBTlkmJKR9nSrlDAUQ4tYVwJOyvlfaYR/sDFRuC9/nJX0lrp+7t1nf+7tUOjqZxlMkG2SRbxCO/yQE5IaekSTi5IQ/kiTw7t86j895mZSWnGnPOvkPpfIrPS+4CA=</latexit>

What do learned representations look like?

Forward pass through untrai ained 3-layer GCN model

34

f( ) =

Parameters initialized randomly

[Zachary’s Karate Club]

Parameters initialized randomly 2-dim output per node

[Zachary’s Karate Club]

Semi-supervised classification on graphs

• Se

Setting: : Some nodes are labeled (black circle) All other nodes are unlabeled

• Task:

ask: Predict node label of unlabeled nodes

35

Semi-supervised classification on graphs

• Se

Setting: : Some nodes are labeled (black circle) All other nodes are unlabeled

• Task:

ask: Predict node label of unlabeled nodes

36

Evaluate loss on labeled nodes only:

set of labeled node indices label matrix GCN output (after softmax)

Toy example (semi-supervised learning)

37

Application: Classification on citation networks

In Input: Citation networks (nodes are papers, edges are citation links, optionally bag-of-words features on nodes) Tar arget: Paper category (e.g. stat.ML, cs.LG, ...) Mo Model: 2-layer GCN

38

Kipf & Welling, Semi-Supervised Classification with Graph Convolutional Networks, ICLR 2017

(Figure from: Bronstein, Bruna, LeCun, Szlam, Vandergheynst, 2016)

Application: Classification on citation networks

In Input: Citation networks (nodes are papers, edges are citation links, optionally bag-of-words features on nodes) Tar arget: Paper category (e.g. stat.ML, cs.LG, ...) Mo Model: 2-layer GCN

39

Classi assificat cation resu sults (accu accuracy acy) )

no input features

(Figure from: Bronstein, Bruna, LeCun, Szlam, Vandergheynst, 2016)

Kipf & Welling, Semi-Supervised Classification with Graph Convolutional Networks, ICLR 2017

Next Lecture: Modeling the Physical World

40