Kipf, T., Welling, M.: Semi-Supervised Classification with Graph - - PowerPoint PPT Presentation

Kipf, T., Welling, M.: Semi-Supervised Classification with Graph Convolutional Networks Radim Špetlík

Czech Technical University in Prague

Overview

2

• Kipf and Welling
• use first order approximation in Fourier-domain to
• btain an efficient linear-time graph-CNNs
• apply the approximation to the semi-supervised graph node

classification problem

Graph Adjacency Matrix 𝑩

3

• symmetric,

square matrix

• 𝐵𝑗𝑘 = 1 iff vertices

𝑤𝑗 and 𝑤𝑘 are incident

• 𝐵𝑗𝑘 = 0 otherwise

http://mathworld.wolfram.com/AdjacencyMatrix.html

Graph Convolutional Network

4

• given a graph 𝐻 = 𝑊, 𝐹 , graph-CNN is a function which:
• takes as input:
• feature description 𝒚𝒋 ∈ ℝ𝐸 for every node 𝑗;

summarized as 𝑌 ∈ ℝ𝑂×𝐸, where 𝑂 is number of nodes, 𝐸 is number of input features

• description of the graph structure in matrix form, typically

an adjacency matrix 𝐵

• produces:
• node-level output 𝑎 ∈ ℝ𝑂×𝐺, where 𝐺 is the number of
• utput features per node

Graph Convolutional Network

5

• is composed of non-linear functions

𝐼(𝑚+1) = 𝑔(𝐼 𝑚 , 𝐵), where 𝐼 0 = 𝑌, and 𝐼(𝑀) = 𝑎, and 𝑀 is the number of layers.

Graph Convolutional Network

6

• graphically:

https://tkipf.github.io/graph-convolutional-networks/

Graph Convolutional Network

7

Let’s start with a simple layer-wise propagation rule 𝑔 𝐼 𝑚 , 𝐵 = 𝜏(𝐵𝐼 𝑚 𝑋 𝑚 ), where 𝑋(𝑚) ∈ ℝ𝐸𝑚×𝐸𝑚+1 is a weight matrix for the 𝑚-th neural network layer, 𝜏(⋅) is a non-linear activation function, 𝐵 ∈ ℝ𝑂×𝑂 is adjacency matrix, 𝑂 is the number of nodes, 𝐼(𝑚) ∈ ℝ𝑂×𝐸𝑚

https://samidavies.wordpress.com/2016/09/20/whats-up-with-the-graph-laplacian/

Graph Convolutional Network

8

multiplication with 𝐵 not enough, we’re missing the node itself 𝑔 𝐼 𝑚 , 𝐵 = 𝜏(𝐵𝐼 𝑚 𝑋 𝑚 ), we fix it by 𝑔 𝐼 𝑚 , 𝐵 = 𝜏( መ 𝐵𝐼 𝑚 𝑋 𝑚 ), where መ 𝐵 = 𝐵 + 𝐽, 𝐽 is the identity matrix

Graph Convolutional Network

9

መ 𝐵 is typically not normalized; this multiplication 𝑔 𝐼 𝑚 , 𝐵 = 𝜏( መ 𝐵𝐼 𝑚 𝑋 𝑚 ), would change the scale of features 𝐼(𝑚) we fix that by symmetric normalization, i.e. ෡ 𝐸−1

2𝐵෡

𝐸−1

2, where ෡

𝐸 is the diagonal node degree matrix of መ 𝐵, ෡ 𝐸𝑗𝑗 = σ𝑘 መ 𝐵𝑗𝑘, producing 𝑔 𝐼 𝑚 , 𝐵 = 𝜏(෡ 𝐸−1

2 መ

𝐵෡ 𝐸−1

2𝐼 𝑚 𝑋 𝑚 ),

Graph Convolutional Network

10

Examining a single layer, single filter 𝜄 ∈ ℝ, and a single node feature vector 𝒚 ∈ ℝ𝐸

Graph Convolutional Network

11

መ 𝐵 = 𝐵 + 𝐽, ෡ 𝐸𝑗𝑗 = σ𝑘 መ 𝐵𝑗𝑘 … renormalization trick

Graph Convolutional Network

12

𝜄 = 𝜄0

′=- 𝜄1 ′

𝜄0

′𝒚 + 𝜄1 ′ 𝑀 − 𝐽 𝒚

Graph Convolutional Network

13

𝜄0

′𝒚 + 𝜄1 ′ 𝑀 − 𝐽 𝒚

Inverse Fourier transform – filtering – Fourier transform

෨ 𝑀 = 𝑑 𝑀 − 𝐽 , 𝑑 ∈ ℝ

𝒉𝜾 ⋆ 𝒚 = 𝑉𝒉𝜾𝑉⊤𝒚

Graph Convolutional Network

14

An efficient graph convolution approximation was performed when the multiplication was interpreted as approximation of convolution in Fourier domain using Chebyshev polynomials. where 𝑂 is number of nodes, E is number of edges, 𝐸𝑚 is number of input channels, 𝐸𝑚+1 is number of output channels.

Overview

15

• Kipf and Welling
• use first order approximation in Fourier-domain to obtain an

efficient linear-time graph-CNNs

• apply the approximation to the semi-supervised graph

node classification problem

▪ given a point set 𝑌 = {𝑦1, … , 𝑦𝑚, 𝑦𝑚+1, … ,𝑦𝑜} ▪ and a label set 𝑀 = {1,… 𝑑}, where

– first 𝑚 points have labels 𝑧1, … , 𝑧𝑚 ∈ 𝑀 – remaining points are unlabeled – 𝑑 is the number of classes

▪ the goal is to

– predict the labels of the unlabeled points

16

Semi-supervised Classification Task

▪ graphically:

17

Semi-supervised Classification Task

https://papers.nips.cc/paper/2506-learning-with-local-and-global-consistency.pdf

▪ example:

– two-layer graph-CNN 𝑎 = 𝑔 𝑌, 𝐵 = softmax መ 𝐵 ReLU መ 𝐵𝑌𝑋 0 𝑋 1 where 𝑋 0 ∈ ℝ𝐷×𝐼 with 𝐷 input channels and 𝐼 features maps, 𝑋 1 ∈ ℝ𝐼×𝐺 with 𝐺 output features per node

18

graph-CNN EXAMPLE

Graph Convolutional Network

19

• graphically:

https://arxiv.org/pdf/1609.02907.pdf

▪ objective function:

– cross-entropy where Y𝑀 is a set of node indices that have labels, 𝑎𝑚𝑔 is the element in the l-th row, f-th column of matrix 𝑎, ground truth: 𝑍

𝑚𝑔 is 1 if instance 𝑚 comes from a class 𝑔.

20

graph-CNN EXAMPLE

▪ weights trained with gradient descent

21

graph-CNN EXAMPLE - RESULTS

▪ different variants of propagation models

22

graph-CNN EXAMPLE - RESULTS

▪ 3-layer GCN, “karate-club” problem, one labeled example per

class:

23

graph-CNN another EXAMPLE

300 training iterations

Limitations

24

• Memory grows linearly with data
• only works with undirected graph
• assumption of locality
• assumption of equal importance of self-connections vs.

edges to neighboring nodes መ 𝐵 = 𝐵 + 𝜇𝐽 where 𝜇 is a learnable parameter.

Summary

25

• Kipf and Welling
• use first order approximation in Fourier-domain to obtain an

efficient linear-time graph-CNNs

• apply the approximation to the semi-supervised graph node

classification problem

26

Thank you very much for your time…

Answers to Questions

27

ሚ 𝐵 = 𝐵 + 𝜇𝐽𝑂

• The lambda parameter would control the influence of

neighbouring edges vs. self-connections.

• How (or why) would the lambda parameter trade-off also

between supervised and unsupervised learning?