NetGAN without GAN: From Random Walks to Low-Rank Approximations - - PowerPoint PPT Presentation
NetGAN without GAN: From Random Walks to Low-Rank Approximations Luca Rendsburg, Holger Heidrich, Ulrike von Luxburg ICML 2020 1 3 Input 1 8 1 4 1 1 5 6 9 1 2 1 7 1 9 } 3 1 } NetGAN: Generating Graphs via Random Walks 1 8
Luca Rendsburg, Holger Heidrich, Ulrike von Luxburg ICML 2020
1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 5 1 5 6 1 8 1 9 }
1 4 6 1 7 1 9 1 2 1 3 1 4 1 5 }
Random walks Synthetic random walks Edge probability matrix
NetGAN: Generating Graphs via Random Walks
1 / 14
1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 5 1 5 6 1 8 1 9 }
1 4 6 1 7 1 9 1 2 1 3 1 4 1 5 }
Random walks Synthetic random walks Edge probability matrix 1 / 14
1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 5 1 5 6 1 8 1 9 }
1 4 6 1 7 1 9 1 2 1 3 1 4 1 5 }
Random walks Synthetic random walks Edge probability matrix 1 / 14
1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 5 1 5 6 1 8 1 9 }
1 4 6 1 7 1 9 1 2 1 3 1 4 1 5 }
Random walks Synthetic random walks Edge probability matrix Learn random walk distribution 1 / 14
1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 5 1 5 6 1 8 1 9 }
1 4 6 1 7 1 9 1 2 1 3 1 4 1 5 }
Random walks Synthetic random walks Edge probability matrix Learn random walk distribution 1 / 14
1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 1 2 1 3 1 5 1 5 6 1 8 1 9 }
1 4 6 1 7 1 9 1 2 1 3 1 4 1 5 }
Random walks Synthetic random walks Edge probability matrix Learn random walk distribution 1 / 14
Conceptual analysis 2 / 14
Conceptual analysis § Inductive bias of NetGAN 2 / 14
Conceptual analysis § Inductive bias of NetGAN § Bypass sampling random walks 2 / 14
Conceptual analysis § Inductive bias of NetGAN § Bypass sampling random walks Simplified version (no GAN, no sampling): “Cross-Entropy Low-rank Logits (CELL)” 2 / 14
Conceptual analysis § Inductive bias of NetGAN § Bypass sampling random walks Simplified version (no GAN, no sampling): “Cross-Entropy Low-rank Logits (CELL)” § Higher transparency 2 / 14
Conceptual analysis § Inductive bias of NetGAN § Bypass sampling random walks Simplified version (no GAN, no sampling): “Cross-Entropy Low-rank Logits (CELL)” § Higher transparency § Comparable generalization performance 2 / 14
Conceptual analysis § Inductive bias of NetGAN § Bypass sampling random walks Simplified version (no GAN, no sampling): “Cross-Entropy Low-rank Logits (CELL)” § Higher transparency § Comparable generalization performance § Huge speedup 2 / 14
§ Goal: learn random walk distribution
Generator architecture
C0 h0 C1 h1
vT v2 v1 v1 ⋯ ⋯ ⋯
W down
p1 pN p1 pN Wup Wup
NetGAN architecture Graph Discrimi- nator Dreal Dfake Generator Random walk z ⇠ N (0,I d) z G(z) sample sample (a) Generator architecture (b) NetGAN architecture
Figure from Bojchevski et al. [2018]
3 / 14
§ Goal: learn random walk distribution § Training set: unbiased random walks tpvpiq
0 , . . . , vpiq T qui of length T over input
graph
Generator architecture
C0 h0 C1 h1
vT v2 v1 v1 ⋯ ⋯ ⋯
W down
p1 pN p1 pN Wup Wup
NetGAN architecture Graph Discrimi- nator Dreal Dfake Generator Random walk z ⇠ N (0,I d) z G(z) sample sample (a) Generator architecture (b) NetGAN architecture
Figure from Bojchevski et al. [2018]
3 / 14
§ Goal: learn random walk distribution § Training set: unbiased random walks tpvpiq
0 , . . . , vpiq T qui of length T over input
graph § Generator: generate sequences tpwpiq
0 , . . . , wpiq T qui of “synthetic” random walks
Generator architecture
C0 h0 C1 h1
vT v2 v1 v1 ⋯ ⋯ ⋯
W down
p1 pN p1 pN Wup Wup
NetGAN architecture Graph Discrimi- nator Dreal Dfake Generator Random walk z ⇠ N (0,I d) z G(z) sample sample (a) Generator architecture (b) NetGAN architecture
Figure from Bojchevski et al. [2018]
3 / 14
§ Goal: learn random walk distribution § Training set: unbiased random walks tpvpiq
0 , . . . , vpiq T qui of length T over input
graph § Generator: generate sequences tpwpiq
0 , . . . , wpiq T qui of “synthetic” random walks
§ Discriminator: distinguish synthetic from real random walks
Generator architecture
C0 h0 C1 h1
vT v2 v1 v1 ⋯ ⋯ ⋯
W down
p1 pN p1 pN Wup Wup
NetGAN architecture Graph Discrimi- nator Dreal Dfake Generator Random walk z ⇠ N (0,I d) z G(z) sample sample (a) Generator architecture (b) NetGAN architecture
Figure from Bojchevski et al. [2018]
3 / 14
§ Goal: learn random walk distribution § Training set: unbiased random walks tpvpiq
0 , . . . , vpiq T qui of length T over input
graph § Generator: generate sequences tpwpiq
0 , . . . , wpiq T qui of “synthetic” random walks
§ Discriminator: distinguish synthetic from real random walks § Architecture: LSTMs with Wasserstein loss
Generator architecture
C0 h0 C1 h1
vT v2 v1 v1 ⋯ ⋯ ⋯
W down
p1 pN p1 pN Wup Wup
NetGAN architecture Graph Discrimi- nator Dreal Dfake Generator Random walk z ⇠ N (0,I d) z G(z) sample sample (a) Generator architecture (b) NetGAN architecture
Figure from Bojchevski et al. [2018]
3 / 14
1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 4 6 1 7 1 9 1 2 1 3 1 4 1 5 }
Generate many synthetic random walks Count transitions in score matrix S Transform S into edge probability matrix Sample edges without replacement 4 / 14
1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 4 6 1 7 1 9 1 2 1 3 1 4 1 5 }
Generate many synthetic random walks Count transitions in score matrix S Transform S into edge probability matrix Sample edges without replacement 4 / 14
1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 4 6 1 7 1 9 1 2 1 3 1 4 1 5 }
Generate many synthetic random walks Count transitions in score matrix S Transform S into edge probability matrix Sample edges without replacement 4 / 14
1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9 1 4 6 1 7 1 9 1 2 1 3 1 4 1 5 }
Generate many synthetic random walks Count transitions in score matrix S Transform S into edge probability matrix Sample edges without replacement 4 / 14
1 4 6 1 7 1 9 1 2 1 3 1 4 1 5 }
Generate many synthetic random walks Count transitions in score matrix S Transform S into edge probability matrix Sample edges without replacement
1 1 2 1 3 1 4 1 5 6 1 8 1 7 1 9
4 / 14
Input graph NetGAN: Sample random walks from graph Train GAN with LSTM Sample random walks from generator Count transitions in score matrix Convert score matrix into edge- independent model Sample
5 / 14
Input graph NetGAN: CELL: Sample random walks from graph Train GAN with LSTM Sample random walks from generator Count transitions in score matrix Solve
problem with rank constraint Solve eigenvector problem for score matrix Convert score matrix into edge- independent model Sample
5 / 14
Input graph NetGAN: CELL: Sample random walks from graph Train GAN with LSTM Sample random walks from generator Count transitions in score matrix Solve
problem with rank constraint Solve eigenvector problem for score matrix Convert score matrix into edge- independent model Sample
5 / 14
Input graph NetGAN: CELL: Sample random walks from graph Train GAN with LSTM Sample random walks from generator Count transitions in score matrix Solve
problem with rank constraint Solve eigenvector problem for score matrix Convert score matrix into edge- independent model Sample
5 / 14
Input graph NetGAN: CELL: Sample random walks from graph Train GAN with LSTM Sample random walks from generator Count transitions in score matrix Solve
problem with rank constraint Solve eigenvector problem for score matrix Convert score matrix into edge- independent model Sample
6 / 14
What causes the generalization of NetGAN?
7 / 14
What causes the generalization of NetGAN?
The random walks? 7 / 14
What causes the generalization of NetGAN?
The random walks? ✗ Random walk distribution determines graph 7 / 14
What causes the generalization of NetGAN?
The random walks? ✗ Random walk distribution determines graph The GAN? 7 / 14
What causes the generalization of NetGAN?
The random walks? ✗ Random walk distribution determines graph The GAN? ✗ Perfectly learning random walk distribution simply memorizes graph 7 / 14
What causes the generalization of NetGAN?
The random walks? ✗ Random walk distribution determines graph The GAN? ✗ Perfectly learning random walk distribution simply memorizes graph The LSTM? 7 / 14
What causes the generalization of NetGAN?
The random walks? ✗ Random walk distribution determines graph The GAN? ✗ Perfectly learning random walk distribution simply memorizes graph The LSTM? ✗ No long-term dependencies in random walks 7 / 14
What causes the generalization of NetGAN?
The random walks? ✗ Random walk distribution determines graph The GAN? ✗ Perfectly learning random walk distribution simply memorizes graph The LSTM? ✗ No long-term dependencies in random walks
NetGAN Val-Criterion EO-Criterion
0k 20k 40k 60k 80k 100k
Training iteration
0.00 0.25 0.50 0.75 1.00
Edge overlap
Figure from Bojchevski et al. [2018]
7 / 14
What causes the generalization of NetGAN?
The random walks? ✗ Random walk distribution determines graph The GAN? ✗ Perfectly learning random walk distribution simply memorizes graph The LSTM? ✗ No long-term dependencies in random walks
NetGAN Val-Criterion EO-Criterion
0k 20k 40k 60k 80k 100k
Training iteration
0.00 0.25 0.50 0.75 1.00
Edge overlap
Figure from Bojchevski et al. [2018]
Computational trick! 7 / 14
What causes the generalization of NetGAN?
The random walks? ✗ Random walk distribution determines graph The GAN? ✗ Perfectly learning random walk distribution simply memorizes graph The LSTM? ✗ No long-term dependencies in random walks
NetGAN Val-Criterion EO-Criterion
0k 20k 40k 60k 80k 100k
Training iteration
0.00 0.25 0.50 0.75 1.00
Edge overlap
Figure from Bojchevski et al. [2018]
Computational trick! ✓ Low-dimensional bottleneck 7 / 14
§ Number of nodes N, set of random walks R, low rank H, row-wise softmax σrows 8 / 14
§ Number of nodes N, set of random walks R, low rank H, row-wise softmax σrows § Generator learns random walk distribution by learning random walk transition matrix from parametric family P “ tσrowspWq P RNˆN : W P RNˆN, rankpWq ď Hu . 8 / 14
§ Number of nodes N, set of random walks R, low rank H, row-wise softmax σrows § Generator learns random walk distribution by learning random walk transition matrix from parametric family P “ tσrowspWq P RNˆN : W P RNˆN, rankpWq ď Hu . § Do this directly with maximum likelihood estimation min
WPRNˆN ´
ÿ
pi,jqPR
log σrowspWqi,j ,
rankpWq ď H . 8 / 14
Input graph NetGAN: CELL: Sample random walks from graph Train GAN with LSTM Sample random walks from generator Count transitions in score matrix Solve
problem with rank constraint Solve eigenvector problem for score matrix Convert score matrix into edge- independent model Sample
9 / 14
§ NetGAN samples many random walks from input graph & from generator 10 / 14
§ NetGAN samples many random walks from input graph & from generator § Example: On CORA-ML (2,810/ 7,981), sees every edge „ 14, 000 times on average 10 / 14
§ NetGAN samples many random walks from input graph & from generator § Example: On CORA-ML (2,810/ 7,981), sees every edge „ 14, 000 times on average § Count random walk transitions in score matrix S P RNˆN 10 / 14
§ NetGAN samples many random walks from input graph & from generator § Example: On CORA-ML (2,810/ 7,981), sees every edge „ 14, 000 times on average § Count random walk transitions in score matrix S P RNˆN § Length of random walk T, amount of random walks n. Normalized score matrix converges S nT
Ý Ý Ý Ý Ý Ñ
n,TÑ8 diagpπqP ,
where P is corresponding random walk transition matrix and π its stationary distribution. 10 / 14
§ NetGAN samples many random walks from input graph & from generator § Example: On CORA-ML (2,810/ 7,981), sees every edge „ 14, 000 times on average § Count random walk transitions in score matrix S P RNˆN § Length of random walk T, amount of random walks n. Normalized score matrix converges S nT
Ý Ý Ý Ý Ý Ñ
n,TÑ8 diagpπqP ,
where P is corresponding random walk transition matrix and π its stationary distribution. Replace normalized score matrix with its limit 10 / 14
WPRNˆN ´
pi,jqPR
WPRNˆN ´
k,l
11 / 14
WPRNˆN ´
pi,jqPR
WPRNˆN ´
k,l
11 / 14
WPRNˆN ´
pi,jqPR
WPRNˆN ´
k,l
Bypass sampling in both steps
11 / 14
12 / 14
12 / 14
Graph: CORA-ML citation network (2,810/ 7,981)
3 16 30 43 56 69 Edge overlap (in %) 100 200 300
3 16 30 43 56 69 Edge overlap (in %) 0.10 0.05 0.00 Assortativity 3 16 30 43 56 69 Edge overlap (in %) 1.70 1.75 1.80 1.85 Power law exp. 3 16 30 43 56 69 Edge overlap (in %) 0.94 0.96 0.98
3 16 30 43 56 69 Edge overlap (in %) 0.00 0.02 0.04 0.06 0.08 Clustering coeff. 3 16 30 43 56 69 Edge overlap (in %) 0.3 0.4 Gini coeff. Target (input graph) CELL NetGAN
12 / 14
13 / 14
Table: Training time (in seconds) for NetGAN and CELL on a variety of networks. NetGAN requires a GPU, while CELL runs on a CPU. Data set (Nodes/ Edges) NetGAN CELL CORA-ML (2,810/ 7,981) 7,478 21 Citeseer (2,110/ 3,668) 4,654 10 PolBlogs (1,222/ 16,779) 55,276 15 RT-GOP (4,687/ 5,529) 14,800 23 Web-EDU (3,031/ 6,474) 11,000 16
13 / 14
NetGAN 14 / 14
NetGAN § Successful graph generative model 14 / 14
NetGAN § Successful graph generative model § Complicated and not transparent 14 / 14
NetGAN § Successful graph generative model § Complicated and not transparent Our contribution: conceptual analysis 14 / 14
NetGAN § Successful graph generative model § Complicated and not transparent Our contribution: conceptual analysis § Uncover inductive bias: low-rank assumption 14 / 14
NetGAN § Successful graph generative model § Complicated and not transparent Our contribution: conceptual analysis § Uncover inductive bias: low-rank assumption § Starting point to better understand inductive bias 14 / 14
NetGAN § Successful graph generative model § Complicated and not transparent Our contribution: conceptual analysis § Uncover inductive bias: low-rank assumption § Starting point to better understand inductive bias § Bypass sampling by using a limit argument 14 / 14
NetGAN § Successful graph generative model § Complicated and not transparent Our contribution: conceptual analysis § Uncover inductive bias: low-rank assumption § Starting point to better understand inductive bias § Bypass sampling by using a limit argument § Propose simplified algorithm with comparable generalization performance 14 / 14
NetGAN § Successful graph generative model § Complicated and not transparent Our contribution: conceptual analysis § Uncover inductive bias: low-rank assumption § Starting point to better understand inductive bias § Bypass sampling by using a limit argument § Propose simplified algorithm with comparable generalization performance Future work 14 / 14
NetGAN § Successful graph generative model § Complicated and not transparent Our contribution: conceptual analysis § Uncover inductive bias: low-rank assumption § Starting point to better understand inductive bias § Bypass sampling by using a limit argument § Propose simplified algorithm with comparable generalization performance Future work § Explain contribution of low-rank assumption 14 / 14
NetGAN § Successful graph generative model § Complicated and not transparent Our contribution: conceptual analysis § Uncover inductive bias: low-rank assumption § Starting point to better understand inductive bias § Bypass sampling by using a limit argument § Propose simplified algorithm with comparable generalization performance Future work § Explain contribution of low-rank assumption 14 / 14