Simplifying Graph Convolutional Networks Amauri Holanda Felix Wu* - - PowerPoint PPT Presentation

Cornell University

Simplifying Graph Convolutional Networks

Christopher Fifty Amauri Holanda de Souza Júnior* Kilian Q. Weinberger Tianyi Zhang* Felix Wu* *: Equal contribution Tao Yu

Cornell University

Graph Convolutional Networks Applications

• Social Networks & Citation Networks
• Applied Chemistry
• Natural Language Processing
• Computer Vision
• …

Image source: https://tkipf.github.io/graph-convolutional-networks/images/karate.png

(Kipf and Welling, ICLR 2017)

Cornell University

Node Classification

• 1

+1 Feature Value: Class +1: Class -1: Feature Vector:

Input Graph

Cornell University

Node Classification

• 1

+1 Feature Value: Class +1: Class -1: Feature Vector:

Input Graph Features: X = [x1, …, xn]⊤ x1 x2 x3 x4 x5 x6 x7

Cornell University

Node Classification

• 1

+1 Feature Value: Class +1: Class -1: Feature Vector:

Input Graph Features: X = [x1, …, xn]⊤ x1 x2 x3 x4 x5 x6 x7 (Partially labelled)

Cornell University

Node Classification

• 1

+1 Feature Value: Class +1: Class -1: Feature Vector:

Input Graph Features: X = [x1, …, xn]⊤ x1 x2 x3 x4 x5 x6 x7 (Partially labelled)

Cornell University Input Graph

x1

x2

x3

x4

x5

x6

x7

H(0) = X = [x1, . . . , xn]>

Graph Convolutional Network (GCN)

• 1

+1 Feature Value: Class +1: Class -1: Feature Vector:

Cornell University Input Graph

x1

x2

x3

x4

x5

x6

x7

H(0) = X = [x1, . . . , xn]>

Graph Convolutional Network (GCN)

• 1

+1 Feature Value: Class +1: Class -1: Feature Vector:

¯ H(k) ← SH(k−1)

Feature Propagation

Cornell University Input Graph

x1

x2

x3

x4

x5

x6

x7

H(0) = X = [x1, . . . , xn]>

Graph Convolutional Network (GCN)

• 1

+1 Feature Value: Class +1: Class -1: Feature Vector:

¯ H(k) ← SH(k−1)

Feature Propagation

Linear Transformation

¯ H(k) ← ¯ H(k)Θ(k)

Cornell University Input Graph

x1

x2

x3

x4

x5

x6

x7

H(0) = X = [x1, . . . , xn]>

Graph Convolutional Network (GCN)

• 1

+1 Feature Value: Class +1: Class -1: Feature Vector:

¯ H(k) ← SH(k−1)

Feature Propagation

Linear Transformation ¯ H(k) ← ¯ H(k)Θ(k)

Nonlinearity

H(k) ← ReLU( ¯ H(k))

Cornell University Input Graph

x1

x2

x3

x4

x5

x6

x7

H(0) = X = [x1, . . . , xn]>

Graph Convolutional Network (GCN)

• 1

+1 Feature Value: Class +1: Class -1: Feature Vector:

¯ H(k) ← SH(k−1)

Feature Propagation

Linear Transformation ¯ H(k) ← ¯ H(k)Θ(k)

Nonlinearity

H(k) ← ReLU( ¯ H(k))

× (K − 1)

Cornell University

Graph Convolutional Network (GCN)

• 1

+1 Feature Value: Class +1: Class -1: Feature Vector:

Nonlinearity Linear Transformation

GCN

¯ H(k) ← SH(k−1)

¯ H(k) ← ¯ H(k)Θ(k)

Predictions

ˆ YGCN = softmax(SH(K−1)Θ(K))

×(K − 1)

Feature Propagation

Input Graph

x1

x2

x3

x4

x5

x6

x7

H(0) = X = [x1, . . . , xn]>

H(k) ← ReLU( ¯ H(k))

Cornell University

Graph Convolutional Network (GCN)

• 1

+1 Feature Value: Class +1: Class -1: Feature Vector:

Nonlinearity Linear Transformation

GCN

¯ H(k) ← SH(k−1)

¯ H(k) ← ¯ H(k)Θ(k)

Predictions

ˆ YGCN = softmax(SH(K−1)Θ(K))

×(K − 1)

Feature Propagation

Input Graph

x1

x2

x3

x4

x5

x6

x7

H(0) = X = [x1, . . . , xn]>

H(k) ← ReLU( ¯ H(k))

Is the nonlinearity necessary?

Cornell University

Graph Convolutional Network (GCN)

• 1

+1 Feature Value: Class +1: Class -1: Feature Vector:

Nonlinearity Linear Transformation

GCN

¯ H(k) ← SH(k−1)

¯ H(k) ← ¯ H(k)Θ(k)

Predictions

ˆ YGCN = softmax(SH(K−1)Θ(K))

×(K − 1)

Feature Propagation

Input Graph

x1

x2

x3

x4

x5

x6

x7

H(0) = X = [x1, . . . , xn]>

H(k) ← ReLU( ¯ H(k))

Is the nonlinearity necessary?

̂ Y = softmax (S…SSXΘ(1)Θ(2)…Θ(K))

Cornell University

Graph Convolutional Network (GCN)

• 1

+1 Feature Value: Class +1: Class -1: Feature Vector:

Nonlinearity Linear Transformation

GCN

¯ H(k) ← SH(k−1)

¯ H(k) ← ¯ H(k)Θ(k)

Predictions

ˆ YGCN = softmax(SH(K−1)Θ(K))

×(K − 1)

Feature Propagation

Input Graph

x1

x2

x3

x4

x5

x6

x7

H(0) = X = [x1, . . . , xn]>

H(k) ← ReLU( ¯ H(k))

Is the nonlinearity necessary?

̂ Y = softmax (S…SSXΘ(1)Θ(2)…Θ(K))

Cornell University

Simple Graph Convolution (SGC)

Logistic Regression

ˆ YSGC = softmax ¯ XΘ

• <latexit sha1_base64="(nul)">(nul)</latexit><latexit sha1_base64="(nul)">(nul)</latexit><latexit sha1_base64="(nul)">(nul)</latexit><latexit sha1_base64="(nul)">(nul)</latexit>

Predictions

K-step Feature Propagation ¯ X ← SKX

SGC

Input Graph

x1

x2

x3

x4

x5

x6

x7

X = [x1, . . . , xn]>

• 1

+1 Feature Value: Class +1: Class -1: Feature Vector:

Cornell University

Simple Graph Convolution (SGC)

Logistic Regression

ˆ YSGC = softmax ¯ XΘ

• <latexit sha1_base64="(nul)">(nul)</latexit><latexit sha1_base64="(nul)">(nul)</latexit><latexit sha1_base64="(nul)">(nul)</latexit><latexit sha1_base64="(nul)">(nul)</latexit>

Predictions

K-step Feature Propagation ¯ X ← SKX

SGC

Input Graph

x1

x2

x3

x4

x5

x6

x7

X = [x1, . . . , xn]>

• 1

+1 Feature Value: Class +1: Class -1: Feature Vector:

1.Interpretability 2.Mini-batch training on large graphs 3.Second-order opimization methods

Cornell University

Performance vs. Training Time on Reddit Dataset

100 101 102 Relative Training Time 92 93 94 95 96 97 Test F1 (%) SGC 1x SAGE-mean 29x SAGE-GCN 32x FastGCN 100x SAGE-LSTM 180x DGI GaAN

Reddit

(Hamilton et al., NeurIPS 2017) Reddit dataset contains 233K nodes and 11.6M edges Within 2.7 seconds.

Cornell University

Comparable or Better Results

• SGC performs on par with or better than GCN across 5 tasks including 14 datasets.
• Graph classification on citation and social networks: Cora, Citeseer, Pubmed, and Reddit datasets
• Text classification: 20NG, R8, R52, Ohsumed, MR
• Semi-supervised user geolocation: GEOTEXT, TWITTER-US, TWITTER-WORLD
• Relation extraction: TACRED
• Zero-shot image classification: ImageNet

Cornell University

Comparable or Better Results

• SGC performs on par with or better than GCN across 5 tasks including 14 datasets.
• Graph classification on citation and social networks: Cora, Citeseer, Pubmed, and Reddit datasets
• Text classification: 20NG, R8, R52, Ohsumed, MR
• Semi-supervised user geolocation: GEOTEXT, TWITTER-US, TWITTER-WORLD
• Relation extraction: TACRED
• Zero-shot image classification: ImageNet

But not on graph classification task! (Nonlinearity is still beneficial on this task)

Cornell University

Thank You!

• A linear model (SGC) is sufficient on many graph tasks.
• Our official code is available at: https://github.com/Tiiiger/SGC
• We thank Deep Graph Library, PyTorch Geometric, Spektral, and

StellarGraph for including SGC in their library.

• Please feel free to come by our poster section if you have questions:

Tonight 06:30~09:00 PM @ Pacific Ballroom #267