[PPT] - MixHop: Higher-Order Graph Convolutional Architectures via PowerPoint Presentation

SLIDE 1

Abu-El-Haija et al, MixHop, ICML’19

Poster #88 Poster #88

Code: http://github.com/samihaija/mixhop Slides: http://sami.haija.org/icml19

MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixing

Sami Abu-El-Haija1, Bryan Perozzi2, Amol Kapoor2, Nazanin Alipourfard1, Kristina Lerman1, Hrayr Harutyunyan1, Greg Ver Steeg1, Aram Galstyan1 2 1

SLIDE 2

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Agenda

Review Graph Convolutional Networks (GCN)

○ Application Semi-Supervised Node Classification (SSNC) ○ Shortcoming of GCN

MixHop: Higher-Order GCN

○ Sparsification

MixHop Results on SSNC

SLIDE 3

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Agenda

Review Graph Convolutional Networks (GCN)

○ Application Semi-Supervised Node Classification (SSNC) ○ Shortcoming of GCN

MixHop: Higher-Order GCN

○ Sparsification

MixHop Results on SSNC

SLIDE 4

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Graph Convolutional Network (GCN) [1]

[1] Kipf & Welling, ICLR 2017

SLIDE 5

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Graph Convolutional Network (GCN) [1]

x1 x3 x5 x6 x4 x2

[1] Kipf & Welling, ICLR 2017

SLIDE 6

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Graph Convolutional Network (GCN) [1]

x1 x3 x5 x6 Input Features x4 x2

[1] Kipf & Welling, ICLR 2017

SLIDE 7

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Graph Convolutional Network (GCN) [1]

[1] Kipf & Welling, ICLR 2017

GC Layer 1 x1 x3 x5 x6 Input Features x4 x2

SLIDE 8

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Graph Convolutional Network (GCN) [1]

h1

(1)

h6

(1)

Latent Features h2

(1)

h3

(1)

h4

(1)

h5

(1)

[1] Kipf & Welling, ICLR 2017

GC Layer 1 x1 x3 x5 x6 Input Features x4 x2

SLIDE 9

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

…

[1] Kipf & Welling, ICLR 2017

x1 x3 x5 x6 Input Features x4 x2 h1

(1)

h6

(1)

Latent Features h2

(1)

h3

(1)

h4

(1)

h5

(1)

GC Layer 1

Graph Convolutional Network (GCN) [1]

SLIDE 10

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Graph Convolutional Network (GCN) [1]

GC Layer L h1

(L)

h6

(L)

Output Features h2

(L)

h3

(L)

h4

(L)

h5

(L)

[1] Kipf & Welling, ICLR 2017

x1 x3 x5 x6 Input Features x4 x2

…

h1

(1)

h6

(1)

Latent Features h2

(1)

h3

(1)

h4

(1)

h5

(1)

GC Layer 1

SLIDE 11

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Graph Convolutional Network (GCN) [1]

h1

(L)

h6

(L)

Output Features h2

(L)

h3

(L)

h4

(L)

h5

(L)

[1] Kipf & Welling, ICLR 2017

y2 y4

Train on semi-supervised node classification:

measure Loss on labeled nodes (y4, y2)

SLIDE 12

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Graph Convolutional Network (GCN) [1]

h1

(L)

h6

(L)

Output Features h2

(L)

h3

(L)

h4

(L)

h5

(L)

[1] Kipf & Welling, ICLR 2017

y2 y4

Train on semi-supervised node classification:

measure Loss on labeled nodes (y4, y2)
Backprop to learn GC layers.

Loss Loss

SLIDE 13

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Graph Convolutional Network (GCN) [1]

GC Layer L h1

(L)

h6

(L)

Output Features h2

(L)

h3

(L)

h4

(L)

h5

(L)

[1] Kipf & Welling, ICLR 2017

y2 Loss y4 Loss SGD

…

h1

(1)

h6

(1)

Latent Features h2

(1)

h3

(1)

h4

(1)

h5

(1)

x1 x3 x5 x6 Input Features x4 x2 GC Layer 1 update update

SLIDE 14

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Graph Convolutional Network (GCN) [1]

GC Layer L h1

(L)

h6

(L)

h2

(L)

h3

(L)

h4

(L)

h5

(L)

[1] Kipf & Welling, ICLR 2017

y2 y4

…

h1

(1)

h6

(1)

h2

(1)

h3

(1)

h4

(1)

h5

(1)

x1 x3 x5 x6 x4 x2 GC Layer 1

?

SLIDE 15

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

h2

(1)

Graph Convolutional Network (GCN) [1]

[1] Kipf & Welling, ICLR 2017

x1 x3 x5 x6 x4 x2 h1

(1)

h6

(1)

h4

(1)

h5

(1)

h3

(1)

GC Layer 1

SLIDE 16

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

h2

(1)

Graph Convolutional Network (GCN) [1]

[1] Kipf & Welling, ICLR 2017

x1 x5 x6 x4 x2 h1

(1)

h6

(1)

h4

(1)

h5

(1)

h3

(1)

GC Layer 1 Avg fc x3

SLIDE 17

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Graph Convolutional Network (GCN) [1]

[1] Kipf & Welling, ICLR 2017

x1 x5 x6 x4 x2 h1

(1)

h6

(1)

h2

(1)

h3

(1)

h4

(1)

h5

(1)

GC Layer 1 Avg fc x3

Tensor Graph

SLIDE 18

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Graph Convolutional Network (GCN) [1]

[1] Kipf & Welling, ICLR 2017

x1 x5 x6 x4 x2 h1

(1)

h6

(1)

h2

(1)

h3

(1)

h4

(1)

h5

(1)

GC Layer 1 Avg fc x3

Tensor Graph

SLIDE 19

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Graph Convolutional Network (GCN) [1]

[1] Kipf & Welling, ICLR 2017

x1 x5 x6 x4 x2 h1

(1)

h6

(1)

h2

(1)

h3

(1)

h4

(1)

h5

(1)

GC Layer 1 Avg fc x3

Tensor Graph

SLIDE 20

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Graph Convolutional Network (GCN) [1]

[1] Kipf & Welling, ICLR 2017

x1 x5 x6 x4 x2 h1

(1)

h6

(1)

h2

(1)

h3

(1)

h4

(1)

h5

(1)

GC Layer 1 Avg fc x3

Tensor Graph

SLIDE 21

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Graph Convolutional Network (GCN) [1]

[1] Kipf & Welling, ICLR 2017

x1 x5 x6 x4 x2 h1

(1)

h6

(1)

h2

(1)

h3

(1)

h4

(1)

h5

(1)

GC Layer 1 Avg fc x3

Tensor Graph

SLIDE 22

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Shortcoming of Vanilla GCN

[1] Kipf & Welling, ICLR 2017

Vanilla GC Layer

SLIDE 23

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Shortcoming of Vanilla GCN

[1] Kipf & Welling, ICLR 2017

😁 fc is shared ⇒ inductive Vanilla GC Layer

SLIDE 24

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Shortcoming of Vanilla GCN

[1] Kipf & Welling, ICLR 2017

😁 fc is shared ⇒ inductive 😣 Appendix Experiments of [1] shows no gains beyond 2 layers Vanilla GC Layer

SLIDE 25

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Shortcoming of Vanilla GCN

[1] Kipf & Welling, ICLR 2017

😁 fc is shared ⇒ inductive 😣 Appendix Experiments of [1] shows no gains beyond 2 layers 😣 cannot mix neighbors from various distances in arbitrary linear combinations Vanilla GC Layer

SLIDE 26

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Shortcoming of Vanilla GCN

[1] Kipf & Welling, ICLR 2017

😁 fc is shared ⇒ inductive 😣 Appendix Experiments of [1] shows no gains beyond 2 layers 😣 cannot mix neighbors from various distances in arbitrary linear combinations e.g. cannot learn Gabor Filters! Vanilla GC Layer

SLIDE 27

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Shortcoming of Vanilla GCN

[1] Kipf & Welling, ICLR 2017

😁 fc is shared ⇒ inductive 😣 Appendix Experiments of [1] shows no gains beyond 2 layers 😣 cannot mix neighbors from various distances in arbitrary linear combinations e.g. cannot learn Gabor Filters! Vanilla GC Layer ?

SLIDE 28

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Detour: Review Gabor Filters

[2] Daugman, Vision Research,1980 [3] Daugman, Journal of the Optical Society of America, 1985 [4] Honglak Lee et al, ICML, 2009 [5] Alex Krizhevsky et al, NeurIPS 2012

Neuroscientists discover their importance in the primate visual cortex [2, 3]:

SLIDE 29

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Detour: Review Gabor Filters

[2] Daugman, Vision Research,1980 [3] Daugman, Journal of the Optical Society of America, 1985 [4] Honglak Lee et al, ICML, 2009 [5] Alex Krizhevsky et al, NeurIPS 2012

Further, they are automatically recovered by training CNNs on images [4, 5] Neuroscientists discover their importance in the primate visual cortex [2, 3]:

SLIDE 30

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Main Motivation

SLIDE 31

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Extend the class of representations realizable by GCNs e.g. to learn Gabor Filters Main Motivation

SLIDE 32

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Agenda

Review Graph Convolutional Networks (GCN)

○ Application Semi-Supervised Node Classification (SSNC) ○ Shortcoming of GCN

MixHop: Higher-Order GCN

○ Sparsification

MixHop Results on SSNC

SLIDE 33

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Our Model: MixHop

Vanilla GC Layer MixHop GC Layer

SLIDE 34

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Our Model: MixHop

Vanilla GC Layer MixHop GC Layer

Couple of code lines implements concatenation

SLIDE 35

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Our Model: MixHop

Vanilla GC Layer MixHop GC Layer

Couple of code lines implements concatenation

SLIDE 36

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Our Model: MixHop

MixHop GC Layer

SLIDE 37

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Our Model: MixHop

MixHop GC Layer 😁 Inductive

SLIDE 38

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Our Model: MixHop

MixHop GC Layer 😁 Inductive 😁 Can incorporate distant nodes

SLIDE 39

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Our Model: MixHop

MixHop GC Layer 😁 Inductive 😁 Can incorporate distant nodes 😁 Can mix neighbors across distances in arbitrary linear combinations

SLIDE 40

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Our Model: MixHop

MixHop GC Layer 😁 Inductive 😁 Can incorporate distant nodes 😁 Can mix neighbors across distances in arbitrary linear combinations i.e. can learn Gabor Filters!

SLIDE 41

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Sparsification

We add group L2-Lasso Regularization to drop-out columns feature matrices, similar to [6]

[6] Gordon et al, CVPR, 2018

[images are rotated space]

SLIDE 42

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Sparsification

We add group L2-Lasso Regularization to drop-out columns feature matrices, similar to [6] 2nd layer of Cora drops-out zeroth-power completely.

[6] Gordon et al, CVPR, 2018

[images are rotated space]

SLIDE 43

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Agenda

Review Graph Convolutional Networks (GCN)

○ Application Semi-Supervised Node Classification (SSNC) ○ Shortcoming of GCN

MixHop: Higher-Order GCN

○ Sparsification

MixHop Results on SSNC

SLIDE 44

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Results on Citation Datasets

SLIDE 45

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Results on (Synthetic) Homophily Datasets

With less homophily, our performance gap increases

SLIDE 46

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Results on (Synthetic) Homophily Datasets

With less homophily, our performance gap increases With less homophily, our method learns more feature differences (i.e. Gabor-like Filters)

SLIDE 47

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

References

[1] Kipf & Welling, “Semi-Supervised Classification with Graph Convolutional Networks, ICLR”, 2017 [2] Daugman, “Two-dimensional spectral analysis of cortical receptive field profiles”, Vision Research,1980 [3] Daugman, “Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters”, Journal of the Optical Society of America, 1985 [4] Honglak Lee et al, “Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations”, ICML, 2009 [5] Alex Krizhevsky et al, “ImageNet Classification with Deep Convolutional Neural Networks”, NeurIPS 2012 [6] Gordon et al, “Morphnet: Fast & simple resource-constrained structure learning of deep networks” CVPR 2018

SLIDE 48

Abu-El-Haija et al, MixHop, ICML’19

Poster #88

Conclusion

With just a couple of lines, Kipf’s model can be extended to incorporate

powers of (normalized) adjacency matrix

Allowing it to learn general neighborhood mixing, and its special cases:

Gabor-like Filter and Delta Ops

Inspection shows Delta Ops are indeed learned with lower levels of

homophily.