of Graph Embeddings Aleksandar Bojchevski Technical University of - - PowerPoint PPT Presentation

Uncertainty and Robustness

• f Graph Embeddings

Aleksandar Bojchevski Technical University of Munich, Germany Graph Embedding Day 2018 - Lyon

Neglected aspects of graph embeddings

Capturing uncertainty Robustness to noise Robustness to adversarial attacks

Uncertainty and Robustness of Graph Embeddings - Bojchevski 2

Neglected aspects of graph embeddings

Capturing uncertainty Robustness to noise Robustness to adversarial attacks

Uncertainty and Robustness of Graph Embeddings - Bojchevski 3

Nodes are points in a low-dimensional space

Uncertainty and Robustness of Graph Embeddings - Bojchevski 4

Nodes are distributions

Uncertainty and Robustness of Graph Embeddings - Bojchevski 5

Graph2Gauss - 3 key modeling ideas

Uncertainty and Robustness of Graph Embeddings - Bojchevski 6 𝑙 = 1 𝑙 = 2

• 1. Uncertainty
• 2. Personalized ranking
• 3. Inductiveness

𝒪( 𝜈𝑗, Σ𝑗) 𝑦𝑗

𝑔

𝜄(𝑦𝑗)

deep encoder

Uncertainty

Embed nodes as (Gaussian) distributions Sources of uncertainty:

• Conflicting structure and attributes
• Heterogenous neighborhood
• Noise, outliers, anomalies, ….

Uncertainty and Robustness of Graph Embeddings - Bojchevski 7

Personalized ranking

For each node 𝑗: nodes in its (𝑙)-hop neighborhood should be closer to 𝑗 compared to nodes in its (𝑙 + 1)-hop neighborhood

Uncertainty and Robustness of Graph Embeddings - Bojchevski 8

𝑙 = 1 𝑙 = 2

Personalized ranking

For each node 𝑗: nodes in its (𝑙)-hop neighborhood should be closer to 𝑗 compared to nodes in its (𝑙 + 1)-hop neighborhood

Uncertainty and Robustness of Graph Embeddings - Bojchevski 9

𝑙 = 1 𝑙 = 2

Personalized ranking

For each node 𝑗: nodes in its (𝑙)-hop neighborhood should be closer to 𝑗 compared to nodes in its (𝑙 + 1)-hop neighborhood Example: closer in terms of the KL Diveregence KL is asymmetric ⇒ handles directed graphs

Uncertainty and Robustness of Graph Embeddings - Bojchevski 10

𝑙 = 1 𝑙 = 2

Personalized ranking

Personalized ranking implies pairwise constraints for node 𝑗 D𝐿𝑀(𝒪

𝑘||𝒪 𝑗) < D𝐿𝑀 (𝒪𝑘′||𝒪 𝑗)

∀𝑘 ∈ 𝑂𝑗

(𝑙), ∀𝑘′ ∈ 𝑂𝑗 (𝑙′), ∀𝑙 < 𝑙′

Uncertainty and Robustness of Graph Embeddings - Bojchevski 11

𝑙 = 1 𝑙 = 2 set of nodes in the 𝑙-hop neighborhood of node 𝑗

Inductiveness

Generalize to unseen nodes by learning a mapping from features to embeddings

Uncertainty and Robustness of Graph Embeddings - Bojchevski 12

𝒪( 𝜈𝑗, Σ𝑗) 𝑦𝑗

𝑔

𝜄(𝑦𝑗)

deep encoder

Graph2Gauss - 3 key modeling ideas

Uncertainty and Robustness of Graph Embeddings - Bojchevski 13 𝑙 = 1 𝑙 = 2

• 1. Uncertainty
• 2. Personalized ranking
• 3. Inductiveness

𝒪( 𝜈𝑗, Σ𝑗) 𝑦𝑗

𝑔

𝜄(𝑦𝑗)

deep encoder

Learning with energy-based loss

𝐹𝑗𝑘 = D𝐿𝑀(𝒪

𝑘| 𝒪 𝑗

ℒ = σ 𝑗,𝑘,𝑘′ (𝐹𝑗𝑘

2 + exp−𝐹𝑗𝑘′)

Closer nodes should have lower energy Naively: 𝑃(𝑂3) complexity Node-anchored sampling strategy:

• For each node same one another node from every neighborhood
• Less than 4.2% triplets seen to match performance
• Lower gradient variance

Uncertainty and Robustness of Graph Embeddings - Bojchevski 14

Graph2Gauss is parameter/data efficient

15 Uncertainty and Robustness of Graph Embeddings - Bojchevski

Graph2Gauss captures uncertainty

Uncertainty correlates with diversity Diversity: number of distinct classes in a node’s k-hop neighborhood

Uncertainty and Robustness of Graph Embeddings - Bojchevski 16

Graph2Gauss captures uncertainty

Uncertainty reveals the intrinsic latent dimensionality of the graph Detected latent dimensions ≈ number ground-truth communities

Uncertainty and Robustness of Graph Embeddings - Bojchevski 17

Uncertainty and link prediction

Prune dimensions with high uncertainty Maintaining link prediction performance

Uncertainty and Robustness of Graph Embeddings - Bojchevski 18

Graph2Gauss is effective for visualization

Uncertainty and Robustness of Graph Embeddings - Bojchevski 19

Neglected aspects of graph embeddings

Capturing uncertainty Robustness to noise Robustness to adversarial attacks

Uncertainty and Robustness of Graph Embeddings - Bojchevski 20

Why spectral embedding

Uncertainty and Robustness of Graph Embeddings - Bojchevski 21

https://www.semanticscholar.org

What is spectral clustering

Graph clustering

• Maximize within-cluster edges
• Minimize between cluster edges

22

Similarity Graph Spectral Embedding

𝑂 = 9 𝐸 = 5

• 5
• 4
• 3
• 2
• 1

1 2 3 4

• 4
• 2

2 4

• 5
• 4
• 3
• 2
• 1

1 2 3 4

• 4
• 2

2 4 Uncertainty and Robustness of Graph Embeddings - Bojchevski

The minimum cut

Partition V into two sets 𝐷1 and 𝐷2, such that the sum of the inter-cluster edge weights cut 𝐷1, 𝐷2 = σ𝑤1∈𝐷1,𝑤2∈𝐷2 𝑥(𝑤1, 𝑤2) is minimized Drawbacks:

• Tends to cut small vertex sets from the rest of the graph
• Considers only inter-cluster edges, no intra-cluster edges

Uncertainty and Robustness of Graph Embeddings - Bojchevski 23

1 2 5

3 4

2 4 2 4 4 2 2 3

1

The normalized cut

Ratio Cut: Minimize

𝑑𝑣𝑢(𝐷1,𝐷2) |𝐷1|

+

𝑑𝑣𝑢(𝐷2,𝐷1) |𝐷2|

Normalized Cut: Minimize

𝑑𝑣𝑢(𝐷1,𝐷2) vol(𝐷1) + 𝑑𝑣𝑢(𝐷1,𝐷2) vol(𝐷2)

Uncertainty and Robustness of Graph Embeddings - Bojchevski 24

1 2 5

3 4

2 4 2 4 4 2 2 3

1

1 2 5

3 4

2 4 2 4 4 2 2 3

1

Multi-way graph partitioning

Generalization to 𝑙 ≥ 2 clusters Partition V into disjoint clusters 𝐷1, … , 𝐷𝑙 such that

• Cut: min

C1,…,Ck

σ𝑗=1

𝑙

𝑑𝑣𝑢(𝐷i, V\𝐷i)

• Ratio Cut: min

C1,…,Ck

σ𝑗=1

𝑙 𝑑𝑣𝑢(𝐷i,V\𝐷i) |𝐷i|

• Normalized Cut: min

C1,…,Ck

σ𝑗=1

𝑙 𝑑𝑣𝑢(𝐷i,V\𝐷i) vol(𝐷𝑗)

Finding the optimal solution is NP-hard How to compute an approximate solution efficiently?

Uncertainty and Robustness of Graph Embeddings - Bojchevski 25

1 2 5

3 4

2 4 2 4 4 2 2 3

1

Minimum Cut for 𝑙 = 3

Graph Laplacian

Laplacian matrix 𝑀 = 𝐸 − 𝐵

• 𝐵 = (weighted) adjacency matrix, 𝐸 = degree matrix

Observation: For any vector 𝑔 we have 𝑔𝑈 ⋅ 𝑀 ⋅ 𝑔 =

1 2 ⋅ σ 𝑣,𝑤 ∈ 𝐹 𝑋 𝑣𝑤 𝑔 𝑣 − 𝑔 𝑤 2

Normalized Laplacian 𝑀𝑡𝑧𝑛 = 𝐸−1

2𝑀𝐸−1 2 = 𝐽 − 𝐸−1 2𝐵𝐸−1 2

26 Uncertainty and Robustness of Graph Embeddings - Bojchevski

Physical interpretation of the Laplacian (I)

Let f be a heat distribution over a graph with 𝑔

𝑗 = the heat at node 𝑤𝑗

The heat transferred between 𝑤𝑗 and 𝑤𝑘 is prop. to (𝑔

𝑗−𝑔 𝑘) if 𝑗, 𝑘 ∈ 𝐹

27

https://en.wikipedia.org/wiki/Laplacian_matrix#/media/ File:Graph_Laplacian_Diffusion_Example.gif

Uncertainty and Robustness of Graph Embeddings - Bojchevski

Physical interpretation of the Laplacian (I)

Graph is viewed as an electrical circuit with edges as wires (resistors) Apply voltage at some nodes and measure induced voltage at other nodes Induced voltages minimizes σ 𝑣,𝑤 ∈ 𝐹 𝑦𝑣 − 𝑦𝑤 2 We can find the voltage by minimizing 𝑦𝑈𝑀x

28 Uncertainty and Robustness of Graph Embeddings - Bojchevski

Properties of the Graph Laplacian

L is symmetric and positive semi-definite The number of eigenvectors of 𝑀 with eigenvalue 0 corresponds to the number of connected components Algebraic connectivity of a graph is 𝜇2(𝑀)

• The magnitude reflects how well connected the graph overall is

The spectrum of 𝑀 encodes useful information about the graph

• Unfortunately, there exist co-spectral graphs

29 Uncertainty and Robustness of Graph Embeddings - Bojchevski

Minimum cut and the graph Laplacian

Define indicator vector: : ℎ𝐷𝑙 𝑗 = ቐ

1 |C𝑗|

𝑗𝑔 𝑤𝑗 ∈ 𝐷𝑙 𝑓𝑚𝑡𝑓 Let H = [ℎ𝐷1; ℎ𝐷2; … ; ℎ𝐷𝑙] Observations: 𝐼𝑈𝐼 = 𝐽𝑒 is orthonormal ℎ𝐷𝑗

𝑈 ⋅ 𝑀 ⋅ ℎ𝑑𝑗 = 𝑑𝑣𝑢 𝐷𝑗,𝑊\𝐷𝑗 𝐷𝑗

and ℎ𝐷𝑗

𝑈 ⋅ 𝑀 ⋅ ℎ𝑑𝑗 = (𝐼𝑈𝑀𝐼)𝑗𝑗

𝑆𝑏𝑢𝑗𝑝𝐷𝑣𝑢(𝐷1, … , 𝐷𝑙) = σ𝑗=1

𝑙 𝑑𝑣𝑢 𝐷𝑗,𝑊\𝐷𝑗 𝐷𝑗

= σ𝑗=1

𝑙 (𝐼𝑈𝑀𝐼)𝑗𝑗 = 𝑢𝑠𝑏𝑑𝑓(𝐼𝑈𝑀𝐼)

NetGAN: Generating Graphs via Random Walks - Bojchevski, Shchur, Zügner, Günnemann. 30

1 2 5

3 4

2 4 2 4 4 2 2 3

1

Minimum cut and the graph Laplacian

Minimizing ratio-cut (normalized cut with 𝑀𝑡𝑧𝑛) is equivalent to min

𝐷1,…,𝐷𝑙 𝑢𝑠𝑏𝑑𝑓(𝐼𝑈𝑀𝐼) subject to 𝐼𝑈𝐼 = 𝐽𝑒

Constraint relaxation: allow arbitrary values for H min

𝐼∈𝑆𝑊×𝐿 𝑢𝑠𝑏𝑑𝑓(𝐼𝑈𝑀𝐼) subject to 𝐼𝑈𝐼 = 𝐽𝑒

Standard trace minimization problem Optimal 𝐼 = First 𝐿 smallest eigenvectors of 𝑀

31 Uncertainty and Robustness of Graph Embeddings - Bojchevski

Spectral embedding: random walk view

𝑀𝑠𝑥 = 𝐸−1𝑀 = 𝐽 − 𝐸−1𝐵 = 𝐽 − 𝑄 is the the random walk Laplacian

• 𝜇 is an eigenvalue of 𝑀𝑠𝑥 with eigenvector 𝑣 if and only if 𝜇 is an eigenvalue of 𝑀𝑡𝑧𝑛 with

eigenvector 𝑥 = 𝐸1/2𝑣

Let 𝑄 𝐶 𝐵 = 𝑄 𝑌1 ∈ 𝐶 𝑌0 ∈ 𝐵 be the probability of a random walker currently at any node in 𝐵 to transition to any node in 𝐶, for 𝐵 ∩ 𝐶 = ∅ and A, 𝐶 ⊂ 𝑊. Sample 𝑌0 ∼ 𝜌 from the stationary distribution 𝑂𝑑𝑣𝑢(𝐵, ҧ 𝐵) = 𝑄( ҧ 𝐵|𝐵) + 𝑄(𝐵| ҧ 𝐵)

Uncertainty and Robustness of Graph Embeddings - Bojchevski 32

Spectral Embedding

Finding the spectral embedding = Solving an optimization task 𝐼∗ = k-first eigenvectors of 𝑀(𝐵)

33

𝐼∗ = arg min

𝐼∈ℝ𝑜×𝑒 𝑈𝑠𝑏𝑑𝑓 𝐼𝑈 ⋅ 𝑀(𝐵) ⋅ 𝐼

subject to 𝐼𝑈 ⋅ 𝐸𝐵 ⋅ 𝐼 = 𝐽𝑒 𝑀(𝐵) = 𝐸(𝐵) − 𝐵 Input Graph 𝐵 Graph Laplacian 𝑀(𝐵) Trace minimization Ratio/Normalized Cut Output Embedding 𝐼∗

Uncertainty and Robustness of Graph Embeddings - Bojchevski

Problem: sensitive to noisy data

Noisy

34

⇒ ⇒ ⇒ ⇒ ⇒ ⇒

Spurious edges Distorted embedding Wrong clustering Clean

⇒ ⇒ ⇒

Noisy

Uncertainty and Robustness of Graph Embeddings - Bojchevski

Robustness via Latent Decomposition

Jointly learn decomposition & embedding Decomposition steered by the underlying embedding / clustering

35

𝐵 = 𝐵𝑕 + 𝐵𝑑 𝐵𝑑 𝐵𝑕 good graph sparse corruptions

+

𝐼∗ = arg min

𝐼∈ℝ𝑜×𝑒 𝑈𝑠𝑏𝑑𝑓 𝐼𝑈 ⋅ 𝑀(𝐵𝑕) ⋅ 𝐼

subject to 𝐼𝑈 ⋅ 𝐸(𝐵𝑕) ⋅ 𝐼 = 𝐽𝑒 𝐵∗, 𝐼∗ = arg min

𝐼∈ℝ𝑜×𝑒 𝐵𝑕∈ ℝ≥0 𝑜×𝑜

𝑈𝑠𝑏𝑑𝑓 𝐼𝑈 ⋅ 𝑀(𝐵𝑕) ⋅ 𝐼 subject to 𝐼𝑈 ⋅ 𝐸(𝐵𝑕) ⋅ 𝐼 = 𝐽𝑒 𝐵 = 𝐵𝑕 + 𝐵𝑑 𝐵𝑑

0 ≤ 2𝜄

∀𝑗: 𝑏𝑗

𝑑 0 ≤ 𝜕𝑗

global local

Uncertainty and Robustness of Graph Embeddings - Bojchevski

Solution: Alternating optimization

Update 𝐼, Given 𝐵𝑕/𝐵𝑑 → 𝐹𝑏𝑡𝑧

• Trace minimization problem
• Solution for 𝐼 are the 𝑙 first generalized eigenvectors of 𝑀(𝐵𝑕)

36

update 𝐵𝑕 update 𝐼 𝐵∗, 𝐼∗ = arg min

𝐼∈ℝ𝑜×𝑒 𝐵𝑕∈ ℝ≥0 𝑜×𝑜

𝑈𝑠𝑏𝑑𝑓 𝐼𝑈 ⋅ 𝑀(𝐵𝑕) ⋅ 𝐼

Uncertainty and Robustness of Graph Embeddings - Bojchevski

Solution: Alternating optimization

Update 𝐵𝑕/𝐵𝑑, Given 𝐼 → 𝑂𝑄 𝐼𝑏𝑠𝑒

• Express eigenvalues of 𝐵𝑜𝑓𝑥

𝑕

in closed form

• 𝐵𝑜𝑓𝑥

𝑕

that minimizes the trace equivalent to maximizing 𝑔

Robust Spectral Clustering 37 Aleksandar Bojchevski

𝑔 𝑏𝑣𝑤

𝑑 𝑣,𝑤∈𝐹

= ෍

𝑣,𝑤∈𝐹

𝑏𝑣𝑤

𝑑

𝒊𝑣 − 𝒊𝑤 2

2 𝑜𝑝𝑒𝑓𝑡 𝑔𝑏𝑠 𝑏𝑥𝑏𝑧 𝑗𝑜 𝑢ℎ𝑓 𝑓𝑛𝑐𝑓𝑒𝑒𝑗𝑜𝑕 𝑡𝑞𝑏𝑑𝑓

− 𝜇 ∘ 𝒊𝑣 2 − 𝜇 ∘ 𝒊𝑤 2

𝑞𝑠𝑓𝑔𝑓𝑠𝑡 𝑓𝑒𝑕𝑓𝑡 𝑑𝑚𝑝𝑡𝑓 𝑢𝑝 𝑢ℎ𝑓 𝑝𝑠𝑗𝑕𝑗𝑜

subject to ⋅ o constraints

𝐵∗, 𝐼∗ = arg min

𝐼∈ℝ𝑜×𝑒 𝐵𝑕∈ ℝ≥0 𝑜×𝑜

𝑈𝑠𝑏𝑑𝑓 𝐼𝑈 ⋅ 𝑀(𝐵𝑕) ⋅ 𝐼

Solution: Alternating optimization

Equivalent to Multidimensional Knapsack problem

• Greedy approximation
• Best possible approximation ratio of

1 𝑂+1

Efficient solution in O(#edges)

38 Uncertainty and Robustness of Graph Embeddings - Bojchevski

I

39

Conclusion

Spectral embedding is sensitive to noisy data Robustness via latent decomposition

ณ 𝐵

𝑝𝑠𝑗𝑕𝑗𝑜𝑏𝑚 𝑕𝑠𝑏𝑞ℎ

= ด 𝐵𝑕

𝑕𝑝𝑝𝑒 𝑕𝑠𝑏𝑞ℎ

+ ด 𝐵𝑑

𝑡𝑞𝑏𝑠𝑡𝑓 𝑑𝑝𝑠𝑠𝑣𝑞𝑢𝑗𝑝𝑜𝑡

Removed corrupted edges ⇒ increased discrimination

40 Uncertainty and Robustness of Graph Embeddings - Bojchevski

Neglected aspects of graph embeddings

Capturing uncertainty Robustness to noise Robustness to adversarial attacks

Uncertainty and Robustness of Graph Embeddings - Bojchevski 41

Adversarial attacks on graph embeddings

(Spectral) Embeddings are not robust to noise / but we can remedy that

Are graph embeddings robust to adversarial attacks?

In domains where graph embeddings are used (e.g. the Web) adversaries are common and false data is easy to inject

Uncertainty and Robustness of Graph Embeddings - Bojchevski 42

Adversarial attacks in the image domain

Image of a tabby cat correctly classified

• Add imperceptible perturbation
• Model classifies the cat as guacamole

43

Training data Training Model 88% tabby cat

Uncertainty and Robustness of Graph Embeddings - Bojchevski

Perturbation

Adversarial attacks in the image domain

Image of a tabby cat correctly classified Add imperceptible perturbation Model classifies the cat as guacamole

44

Training data Training Model 99% guacamole

Perturbed image

Uncertainty and Robustness of Graph Embeddings - Bojchevski

The relational nature of the data might

Improve Robustness embeddings are computed jointly rather than in isolation Cause Cascading Failures perturbations in one part of the graph can propagate to the rest

45 Uncertainty and Robustness of Graph Embeddings - Bojchevski

Attack possibilities

General attack Goal: decrease the overall quality

• f the embeddings

Actions:

• Add/remove (flip) an edge
• Add/remove a node
• …

Targeted attack Goal: attack a specific node or a specific downstream task Examples:

• Misclassify a target node 𝑢
• Increase/decrease the similarity of

a set of node pairs 𝒰 ⊂ 𝑊 × 𝑊

46 Uncertainty and Robustness of Graph Embeddings - Bojchevski

Attack model formally

መ 𝐵∗ = arg max

෠ 𝐵∈ 0,1 𝑂×𝑂 ℒ( መ

𝐵, 𝑎∗) 𝑎∗ = min

𝑎 ℒ( መ

𝐵, 𝑎) 𝑡𝑣𝑐𝑘. 𝑢𝑝 መ 𝐵 − 𝐵 0 = 2𝑔

Uncertainty and Robustness of Graph Embeddings - Bojchevski 47

Adjacency matrix of the graph after the attacker modified some entries Optimal embedding from the to be optimized graph መ 𝐵 The attacker’s budget

Attack model formally

መ 𝐵∗ = arg max

෠ 𝐵∈ 0,1 𝑂×𝑂 ℒ( መ

𝐵, 𝑎∗) 𝑎∗ = min

𝑎 ℒ( መ

𝐵, 𝑎) 𝑡𝑣𝑐𝑘. 𝑢𝑝 መ 𝐵 − 𝐵 0 = 2𝑔

Uncertainty and Robustness of Graph Embeddings - Bojchevski 48

Adjacency matrix of the graph after the attacker modified some entries Optimal embedding from the to be optimized graph መ 𝐵 The attacker’s budget General attack

Attack model formally

መ 𝐵∗ = arg max

෠ 𝐵∈ 0,1 𝑂×𝑂 ℒ𝑏𝑢𝑑𝑙( መ

𝐵, 𝑎∗) 𝑎∗ = min

𝑎 ℒ( መ

𝐵, 𝑎) 𝑡𝑣𝑐𝑘. 𝑢𝑝 መ 𝐵 − 𝐵 0 = 2𝑔

Uncertainty and Robustness of Graph Embeddings - Bojchevski 49

Adjacency matrix of the graph after the attacker modified some entries Optimal embedding from the to be optimized graph መ 𝐵 The attacker’s budget Targeted attack

Challenges

Discrete and Combinatorial Bi-level optimization problem Transductive learning ⇒ network poisoning setting

Uncertainty and Robustness of Graph Embeddings - Bojchevski 50

frozen model target train

Evasion

train train target model embedding

Poisoning

Random-walk based embeddings

RW-based embeddings solve: 𝑎∗ = min

𝑎 ℒ( 𝑠 1, 𝑠 2, … , 𝑎) with 𝑠 𝑗 = 𝑆𝑋 𝑚(𝐵)

𝑎∗ ∈ ℝ𝑂×𝐿: learned embedding 𝑆𝑋

𝑚: e stochastic procedure that generates RWs of length 𝑚

ℒ: model-specific loss e.g. skip-gram with negative sampling (SGSN) Challenge: RW sampling precludes gradient based optimization

Uncertainty and Robustness of Graph Embeddings - Bojchevski 51

Example: DeepWalk

DeepWalk is equivalent* to factorizing ෩ 𝑁 = log max 𝑁, 1 𝑁 =

𝑤𝑝𝑚 𝐵 𝑈⋅𝑐 𝑇

𝑇 = σ𝑠=1

𝑈

𝑄𝑠 𝐸−1 𝑄 = 𝐸−1𝐵 with 𝑎∗ obtained by the SVD of ෩ 𝑁 = 𝑉Σ𝑊𝑈 using the top K largest singular values/vectors i.e. 𝑎∗ = 𝑉𝐿Σ𝐿

1/2

Uncertainty and Robustness of Graph Embeddings - Bojchevski 52

transition matrix window size 𝑈 𝑐 negative samples Shifted Positive Pointwise Mutual Information (PPMI) Matrix

Example: DeepWalk

Equivalent to min

෩ 𝑁𝐿

|| ෩ 𝑁 − ෩ 𝑁𝐿||𝐺

2

The loss using the optimal embedding is ℒ𝐸𝑋

1 𝐵, 𝑎∗ =

σ𝑞=𝐿+1

|𝑊|

𝜏𝑞

2, where

𝜏1 ≥ 𝜏2 ≥ ⋯ ≥ 𝜏|𝑊| are the singular values of ෩ 𝑁(𝐵) ordered decreasingly Idea: Given a perturbation Δ𝐵, find the change in the singular values of ෩ 𝑁(𝐵 + Δ𝐵)

Uncertainty and Robustness of Graph Embeddings - Bojchevski 53

Example: DeepWalk

෩ 𝑁 = log max 𝑁, 1 𝑁 =

𝑤𝑝𝑚 𝐵 𝑈⋅𝑐 𝑇

𝑇 = σ𝑠=1

𝑈

𝑄𝑠 𝐸−1 Linearization: ignore the log(⋅) and max(⋅, 1) Scalars 𝑤𝑝𝑚 𝐵 , 𝑈, 𝑐 can be also ignore Rewrite ℒ𝐸𝑋

1 𝐵, 𝑎∗ =

σ𝑞=𝐿+1

|𝑊|

|𝜇𝑞|2 Thus, find a change in the spectrum of 𝑇 after the attacker perturbed the graph Δ𝐵

Uncertainty and Robustness of Graph Embeddings - Bojchevski 54

Spectrum of S

Compute the generalized spectrum (generalized eigenvalues/vectors) of 𝐵 i.e. compute and 𝑉, Λ that solve 𝐵𝑣 = 𝜇𝐸𝑣 Rewrite 𝑇 = σ𝑠=1

𝑈

𝑄𝑠 𝐸−1 as 𝑇 = 𝑉 (σ𝑠=1

𝑈

Λ𝑠)𝑉𝑈 The task is now to find the change in generalized eigenvalues 𝜇𝑞 of the adjacency matrix 𝐵 given a perturbation Δ𝐵

Uncertainty and Robustness of Graph Embeddings - Bojchevski 55

simple function of the generalized eigenvalues 𝜇𝑗 of the graph

Eigenvalue perturbation theory

Given 𝑉, Λ that solve 𝐵𝑣 = 𝜇𝐸𝑣′ and a small perturbation Δ𝐵, Δ𝐸 Find 𝑉′, Λ′ that solve 𝐵 + Δ𝐵 𝑣′ = 𝜇′(𝐸 + Δ𝐸)𝑣′ First order approximation: 𝜇′𝑞 = 𝜇𝑞 + 𝑣𝑞

𝑈 Δ𝐵 + 𝜇𝑞Δ𝐸 𝑣𝑞

for small Δ𝐵 and ΔD higher order terms become negligible

Uncertainty and Robustness of Graph Embeddings - Bojchevski 56

For a single edge flip

Δ𝐵 is a matrix with only 2 non-zero elements for a single edge flip (𝑗, 𝑘) namely Δ𝐵𝑗𝑘 = Δ𝐵𝑘𝑗 = 1 − 2A𝑗𝑘 ≔ Δw𝑗𝑘 Similarly, ΔD has only two non-zero elements on the diagonal Then we can approximate the generalized eigenvalues of A + Δ𝐵 in closed-form computable in O(1) time:

𝜇′𝑞 = 𝜇𝑞 + Δw𝑗𝑘 2𝑣𝑞𝑗 ⋅ 𝑣𝑞𝑘 − 𝜇𝑞(𝑣𝑞𝑗

2 + 𝑣𝑞𝑘 2 )

Uncertainty and Robustness of Graph Embeddings - Bojchevski 57

Connecting it all together

• 1. DeepWalk is equivalent to a SVD of ෩

𝑁 = log max

𝑤𝑝𝑚 𝐵 𝑈⋅𝑐 𝑇, 1

• 2. The loss can be computed from the singular values / the spectrum of S
• 3. The spectrum of 𝑇 can be easily computed from the generalized spectrum of A
• 4. For any given edge flip (𝑗, 𝑘) we can compute in O(1) the spectrum of 𝐵 + Δ𝐵

Uncertainty and Robustness of Graph Embeddings - Bojchevski 58

Overall algorithm

However, መ 𝐵∗ = arg max

෠ 𝐵∈ 0,1 𝑂×𝑂 ℒ𝑑𝑚𝑝𝑡𝑓𝑒−𝑔𝑝𝑠𝑛

𝑡𝑣𝑐𝑘. 𝑢𝑝 መ 𝐵 − 𝐵 0 = 2𝑔 is still hard to optimize – 𝑂2 𝑔 ways to choose the flips Greedy solution:

• 1. For each edge (𝑗, 𝑘) calculate its impact on the loss if flipped
• 2. Pick the top f edges

59 Uncertainty and Robustness of Graph Embeddings - Bojchevski

General attack (node classification proxy)

Uncertainty and Robustness of Graph Embeddings - Bojchevski 60

Targeted attack

To target node 𝑤 we need the change in its embedding 𝑎𝑤

∗

That is we need the change in eigenvectors Apply eigenvalue perturbation again to approximate the top 𝐿 eigenvectors For a given edge flip (𝑗, 𝑘) we get: 𝑣𝑞

′ = 𝑣𝑞 − Δw𝑗𝑘 𝐵 − 𝜇𝐸 +

−Δ𝜇𝑞𝑣𝑞 ∘ 𝑒 + 𝐹𝑗 𝑣𝑞𝑘 − 𝜇𝑞𝑣𝑞𝑗 + 𝐹

𝑘 𝑣𝑞𝑗 − 𝜇𝑞𝑣𝑞𝑘

Uncertainty and Robustness of Graph Embeddings - Bojchevski 61

Targeted attack: Link prediction

Uncertainty and Robustness of Graph Embeddings - Bojchevski 62

Targeted attack: Node classification

Uncertainty and Robustness of Graph Embeddings - Bojchevski 63

Before attack Degree attack Random attack Our attack

Analysis of adversarial edges

Uncertainty and Robustness of Graph Embeddings - Bojchevski 64

Transferability

DW (SVD) DW (SGNS) n2v

• Spect. Embd. Label Prop.

GCN 𝑔 = 250 (0.8%)

• 3.59
• 2.37
• 2.04
• 2.11
• 5.78
• 3.34

𝑔 = 500 (1.6%)

• 4.62
• 3.97
• 3.48
• 4.57
• 8.95
• 2.33

𝑔 = 250 (1.7%)

• 7.59
• 5.73
• 6.45
• 3.58
• 4.99
• 2.21

𝑔 = 500 (3.4%)

• 9.68
• 11.47
• 10.24
• 4.57
• 6.27
• 8.61

Uncertainty and Robustness of Graph Embeddings - Bojchevski 65

Conclusion

Node embeddings are vulnerable to adversarial attacks Poisoning has negative effect on the embeddings quality and the downstream tasks Attacks are transferable – they generalize to many models

66 Uncertainty and Robustness of Graph Embeddings - Bojchevski

Important aspects of graph embeddings

Capturing uncertainty Robustness to noise Robustness to adversarial attacks

Uncertainty and Robustness of Graph Embeddings - Bojchevski 67