of Graph Embeddings Aleksandar Bojchevski Technical University of - PowerPoint PPT Presentation

Uncertainty and Robustness of Graph Embeddings Aleksandar Bojchevski Technical University of Munich, Germany Graph Embedding Day 2018 - Lyon

Neglected aspects of graph embeddings Capturing uncertainty Robustness to noise Robustness to adversarial attacks Uncertainty and Robustness of Graph Embeddings - Bojchevski 2

Neglected aspects of graph embeddings Capturing uncertainty Robustness to noise Robustness to adversarial attacks Uncertainty and Robustness of Graph Embeddings - Bojchevski 3

Nodes are points in a low-dimensional space Uncertainty and Robustness of Graph Embeddings - Bojchevski 4

Nodes are distributions Uncertainty and Robustness of Graph Embeddings - Bojchevski 5

Graph2Gauss - 3 key modeling ideas 1. Uncertainty 2. Personalized ranking 3. Inductiveness 𝑦 𝑗 𝑙 = 2 𝑔 𝜄 (𝑦 𝑗 ) deep 𝑙 = 1 encoder 𝒪( 𝜈 𝑗 , Σ 𝑗 ) Uncertainty and Robustness of Graph Embeddings - Bojchevski 6

Uncertainty Embed nodes as (Gaussian) distributions Sources of uncertainty: • Conflicting structure and attributes • Heterogenous neighborhood • Noise, outliers, anomalies, …. Uncertainty and Robustness of Graph Embeddings - Bojchevski 7

Personalized ranking 𝑙 = 2 For each node 𝑗 : nodes in its (𝑙) -hop neighborhood 𝑙 = 1 should be closer to 𝑗 compared to nodes in its (𝑙 + 1) -hop neighborhood Uncertainty and Robustness of Graph Embeddings - Bojchevski 8

Personalized ranking 𝑙 = 2 For each node 𝑗 : nodes in its (𝑙) -hop neighborhood 𝑙 = 1 should be closer to 𝑗 compared to nodes in its (𝑙 + 1) -hop neighborhood Uncertainty and Robustness of Graph Embeddings - Bojchevski 9

Personalized ranking For each node 𝑗 : nodes in its (𝑙) -hop neighborhood 𝑙 = 2 should be closer to 𝑗 compared to nodes in its (𝑙 + 1) -hop neighborhood 𝑙 = 1 Example: closer in terms of the KL Diveregence KL is asymmetric ⇒ handles directed graphs Uncertainty and Robustness of Graph Embeddings - Bojchevski 10

Personalized ranking 𝑙 = 2 Personalized ranking implies pairwise constraints for node 𝑗 𝑙 = 1 D 𝐿𝑀 (𝒪 𝑘 ||𝒪 𝑗 ) < D 𝐿𝑀 (𝒪 𝑘 ′ ||𝒪 𝑗 ) (𝑙 ′ ) , ∀𝑙 < 𝑙′ (𝑙) , ∀𝑘 ′ ∈ 𝑂 𝑗 ∀𝑘 ∈ 𝑂 𝑗 set of nodes in the 𝑙 -hop neighborhood of node 𝑗 Uncertainty and Robustness of Graph Embeddings - Bojchevski 11

Inductiveness 𝑦 𝑗 𝑔 𝜄 (𝑦 𝑗 ) deep Generalize to unseen nodes by learning encoder a mapping from features to embeddings 𝒪( 𝜈 𝑗 , Σ 𝑗 ) Uncertainty and Robustness of Graph Embeddings - Bojchevski 12

Graph2Gauss - 3 key modeling ideas 1. Uncertainty 2. Personalized ranking 3. Inductiveness 𝑦 𝑗 𝑙 = 2 𝑔 𝜄 (𝑦 𝑗 ) deep 𝑙 = 1 encoder 𝒪( 𝜈 𝑗 , Σ 𝑗 ) Uncertainty and Robustness of Graph Embeddings - Bojchevski 13

Learning with energy-based loss 2 + exp −𝐹 𝑗𝑘′ ) ℒ = σ 𝑗,𝑘,𝑘 ′ (𝐹 𝑗𝑘 𝐹 𝑗𝑘 = D 𝐿𝑀 (𝒪 𝑘 | 𝒪 𝑗 Closer nodes should have lower energy Naively: 𝑃(𝑂 3 ) complexity Node-anchored sampling strategy: • For each node same one another node from every neighborhood • Less than 4.2% triplets seen to match performance • Lower gradient variance Uncertainty and Robustness of Graph Embeddings - Bojchevski 14

Graph2Gauss is parameter/data efficient Uncertainty and Robustness of Graph Embeddings - Bojchevski 15

Graph2Gauss captures uncertainty Uncertainty correlates with diversity Diversity: number of distinct classes in a node’s k -hop neighborhood Uncertainty and Robustness of Graph Embeddings - Bojchevski 16

Graph2Gauss captures uncertainty Uncertainty reveals the intrinsic latent dimensionality of the graph Detected latent dimensions ≈ number ground-truth communities Uncertainty and Robustness of Graph Embeddings - Bojchevski 17

Uncertainty and link prediction Prune dimensions with high uncertainty Maintaining link prediction performance Uncertainty and Robustness of Graph Embeddings - Bojchevski 18

Graph2Gauss is effective for visualization Uncertainty and Robustness of Graph Embeddings - Bojchevski 19

Neglected aspects of graph embeddings Capturing uncertainty Robustness to noise Robustness to adversarial attacks Uncertainty and Robustness of Graph Embeddings - Bojchevski 20

Why spectral embedding https://www.semanticscholar.org Uncertainty and Robustness of Graph Embeddings - Bojchevski 21

What is spectral clustering 𝐸 = 5 4 4 3 3 2 2 1 1 𝑂 = 9 Spectral Similarity 0 0 Embedding Graph -4 -2 0 2 4 -4 -2 0 2 4 -1 -1 -2 -2 -3 -3 -4 -4 -5 -5 Graph clustering • Maximize within-cluster edges • Minimize between cluster edges Uncertainty and Robustness of Graph Embeddings - Bojchevski 22

The minimum cut Partition V into two sets 𝐷 1 and 𝐷 2 , such that the sum of the inter-cluster edge weights cut 𝐷 1 , 𝐷 2 = σ 𝑤 1 ∈𝐷 1 ,𝑤 2 ∈𝐷 2 𝑥(𝑤 1 , 𝑤 2 ) is minimized 1 2 4 2 4 0 5 2 3 1 4 2 3 4 2 Drawbacks: • Tends to cut small vertex sets from the rest of the graph • Considers only inter-cluster edges, no intra-cluster edges Uncertainty and Robustness of Graph Embeddings - Bojchevski 23

The normalized cut 𝑑𝑣𝑢(𝐷 1 ,𝐷 2 ) 𝑑𝑣𝑢(𝐷 2 ,𝐷 1 ) Ratio Cut: Minimize + |𝐷 1 | |𝐷 2 | 𝑑𝑣𝑢(𝐷 1 ,𝐷 2 ) 𝑑𝑣𝑢(𝐷 1 ,𝐷 2 ) Normalized Cut: Minimize vol(𝐷 1 ) + vol(𝐷 2 ) 1 2 4 1 2 4 2 4 2 4 0 5 2 3 0 5 2 3 1 1 4 2 4 2 3 4 3 4 2 2 Uncertainty and Robustness of Graph Embeddings - Bojchevski 24

Multi-way graph partitioning Generalization to 𝑙 ≥ 2 clusters Partition V into disjoint clusters 𝐷 1 , … , 𝐷 𝑙 such that 𝑙 • Cut: min σ 𝑗=1 𝑑𝑣𝑢(𝐷 i , V\𝐷 i ) 1 2 4 C 1 ,…,C k 2 4 𝑑𝑣𝑢(𝐷 i ,V\𝐷 i ) 𝑙 • Ratio Cut: min σ 𝑗=1 0 5 2 3 |𝐷 i | C 1 ,…,C k 1 4 2 𝑑𝑣𝑢(𝐷 i ,V\𝐷 i ) 𝑙 • Normalized Cut: min σ 𝑗=1 3 4 2 vol(𝐷 𝑗 ) C 1 ,…,C k Minimum Cut for 𝑙 = 3 Finding the optimal solution is NP-hard How to compute an approximate solution efficiently? Uncertainty and Robustness of Graph Embeddings - Bojchevski 25

Graph Laplacian Laplacian matrix 𝑀 = 𝐸 − 𝐵 • 𝐵 = (weighted) adjacency matrix, 𝐸 = degree matrix Observation: For any vector 𝑔 we have 𝑔 𝑈 ⋅ 𝑀 ⋅ 𝑔 = 1 𝑤 2 2 ⋅ σ 𝑣,𝑤 ∈ 𝐹 𝑋 𝑣𝑤 𝑔 𝑣 − 𝑔 Normalized Laplacian 𝑀 𝑡𝑧𝑛 = 𝐸 − 1 2 𝑀𝐸 − 1 2 = 𝐽 − 𝐸 − 1 2 𝐵𝐸 − 1 2 Uncertainty and Robustness of Graph Embeddings - Bojchevski 26

Physical interpretation of the Laplacian (I) Let f be a heat distribution over a graph with 𝑔 𝑗 = the heat at node 𝑤 𝑗 The heat transferred between 𝑤 𝑗 and 𝑤 𝑘 is prop. to (𝑔 𝑗 −𝑔 𝑘 ) if 𝑗, 𝑘 ∈ 𝐹 https://en.wikipedia.org/wiki/Laplacian_matrix#/media/ File:Graph_Laplacian_Diffusion_Example.gif Uncertainty and Robustness of Graph Embeddings - Bojchevski 27

Physical interpretation of the Laplacian (I) Graph is viewed as an electrical circuit with edges as wires (resistors) Apply voltage at some nodes and measure induced voltage at other nodes Induced voltages minimizes σ 𝑣,𝑤 ∈ 𝐹 𝑦 𝑣 − 𝑦 𝑤 2 We can find the voltage by minimizing 𝑦 𝑈 𝑀x Uncertainty and Robustness of Graph Embeddings - Bojchevski 28

Properties of the Graph Laplacian L is symmetric and positive semi-definite The number of eigenvectors of 𝑀 with eigenvalue 0 corresponds to the number of connected components Algebraic connectivity of a graph is 𝜇 2 (𝑀) • The magnitude reflects how well connected the graph overall is The spectrum of 𝑀 encodes useful information about the graph • Unfortunately, there exist co-spectral graphs Uncertainty and Robustness of Graph Embeddings - Bojchevski 29

Minimum cut and the graph Laplacian 1 1 2 4 𝑗𝑔 𝑤 𝑗 ∈ 𝐷 𝑙 Define indicator vector: : ℎ 𝐷 𝑙 𝑗 = ቐ 2 4 |C 𝑗 | 0 5 2 3 0 𝑓𝑚𝑡𝑓 1 4 2 Let H = [ℎ 𝐷 1 ; ℎ 𝐷 2 ; … ; ℎ 𝐷 𝑙 ] 3 4 2 Observations: 𝐼 𝑈 𝐼 = 𝐽𝑒 is orthonormal 𝑈 ⋅ 𝑀 ⋅ ℎ 𝑑 𝑗 = 𝑑𝑣𝑢 𝐷 𝑗 ,𝑊\𝐷 𝑗 𝑈 ⋅ 𝑀 ⋅ ℎ 𝑑 𝑗 = (𝐼 𝑈 𝑀𝐼) 𝑗𝑗 ℎ 𝐷 𝑗 and ℎ 𝐷 𝑗 𝐷 𝑗 𝑙 (𝐼 𝑈 𝑀𝐼) 𝑗𝑗 = 𝑢𝑠𝑏𝑑𝑓(𝐼 𝑈 𝑀𝐼) 𝑑𝑣𝑢 𝐷 𝑗 ,𝑊\𝐷 𝑗 𝑙 𝑆𝑏𝑢𝑗𝑝𝐷𝑣𝑢(𝐷 1 , … , 𝐷 𝑙 ) = σ 𝑗=1 = σ 𝑗=1 𝐷 𝑗 NetGAN: Generating Graphs via Random Walks - Bojchevski, Shchur, Zügner, Günnemann. 30

Minimum cut and the graph Laplacian Minimizing ratio-cut (normalized cut with 𝑀 𝑡𝑧𝑛 ) is equivalent to 𝐷 1 ,…,𝐷 𝑙 𝑢𝑠𝑏𝑑𝑓(𝐼 𝑈 𝑀𝐼) subject to 𝐼 𝑈 𝐼 = 𝐽𝑒 min Constraint relaxation: allow arbitrary values for H 𝐼∈𝑆 𝑊×𝐿 𝑢𝑠𝑏𝑑𝑓(𝐼 𝑈 𝑀𝐼) subject to 𝐼 𝑈 𝐼 = 𝐽𝑒 min Standard trace minimization problem Optimal 𝐼 = First 𝐿 smallest eigenvectors of 𝑀 Uncertainty and Robustness of Graph Embeddings - Bojchevski 31