http://cs224w.stanford.edu ? ? ? ? Machine Learning ? Node - - PowerPoint PPT Presentation

CS224W: Machine Learning with Graphs Jure Leskovec, Stanford University

http://cs224w.stanford.edu

? ? ? ? ?

Machine Learning

Node classification

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 2 10/15/19

Machine Learning

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 4

? ? ?

x

10/15/19

5

Raw Data Structured Data Learning Algorithm Model Downstream task Feature Engineering

Automatically learn the features

¡ (Supervised) Machine Learning Lifecycle

requires feature engineering every single time!

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 10/15/19

Goal: Efficient task-independent feature learning for machine learning with graphs!

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 6

vec node 𝑔: 𝑣 → ℝ& ℝ&

Feature representation, embedding

u

10/15/19

• –

– –

17

¡ Task: We map each node in a network into a

low-dimensional space

§ Distributed representations for nodes § Similarity of embeddings between nodes indicates their network similarity § Encode network information and generate node representation

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 7

¡ 2D embeddings of nodes of the Zachary’s

Karate Club network:

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 8

• Zachary’s Karate Network:

Image from: Perozzi et al. DeepWalk: Online Learning of Social Representations. KDD 2014.

¡ Modern deep learning toolbox is designed for

simple sequences or grids.

§ CNNs for fixed-size images/grids…. § RNNs or word2vec for text/sequences…

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 9

¡ But networks are far more complex!

§ Complex topographical structure (i.e., no spatial locality like grids) § No fixed node ordering or reference point (i.e., the isomorphism problem) § Often dynamic and have multimodal features.

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 10

¡ Assume we have a graph G:

§ V is the vertex set. § A is the adjacency matrix (assume binary). § No node features or extra information is used!

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 12 10/15/19

¡ Goal is to encode nodes so that similarity in

the embedding space (e.g., dot product) approximates similarity in the original network

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 13 10/15/19

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 14

similarity(u, v) ≈ z>

v zu

Go Goal: Ne Need t to d define!

10/15/19

in the original network Similarity of the embedding

1.

Define an encoder (i.e., a mapping from nodes to embeddings)

2.

Define a node similarity function (i.e., a measure of similarity in the original network)

3.

Optimize the parameters of the encoder so that:

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 15

similarity(u, v) ≈ z>

v zu

10/15/19

in the original network Similarity of the embedding

¡ Encoder: maps each node to a low-

dimensional vector

¡ Similarity function: specifies how the

relationships in vector space map to the relationships in the original network

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 16

enc(v) = zv

node in the input graph d-dimensional embedding Similarity of u and v in the original network dot product between node embeddings

similarity(u, v) ≈ z>

v zu

10/15/19

¡ Simplest encoding approach: encoder is just

an embedding-lookup

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 17

matrix, each column is a node embedding [w [what w we l learn!] !] indicator vector, all zeroes except a one in column indicating node v

enc(v) = Zv

Z ∈ Rd×|V| v ∈ I|V|

10/15/19

¡ Simplest encoding approach: encoder is just

an embedding-lookup

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 18

Z =

Dimension/size

• f embeddings
• ne column per node

embedding matrix embedding vector for a specific node

10/15/19

Simplest encoding approach: encoder is just an embedding-lookup Each node is assigned to a unique embedding vector Many methods: DeepWalk, node2vec, TransE

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 19

¡ Key choice of methods is how they define

node similarity.

¡ E.g., should two nodes have similar

embeddings if they….

§ are connected? § share neighbors? § have similar “structural roles”? § …?

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 20

Material based on:

• Perozzi et al. 2014. DeepWalk: Online Learning of Social Representations. KDD.
• Grover et al. 2016. node2vec: Scalable Feature Learning for Networks. KDD.

1 4 3 2 5 6 7 9 10 8 11 12

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 22

Given a graph and a starting point, we select a neighbor of it at random, and move to this neighbor; then we select a neighbor of this point at random, and move to it, etc. The (random) sequence of points selected this way is a random walk on the graph.

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 23

probability that u and v co-occur on a random walk over the network

z>

u zv ≈

10/15/19

1.

Estimate probability of visiting node 𝒘 on a random walk starting from node 𝒗 using some random walk strategy 𝑺

2.

Optimize embeddings to encode these random walk statistics:

Similarity (here: dot product=cos(𝜄)) encodes random walk “similarity”

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 24 10/15/19

1.

Expressivity: Flexible stochastic definition of node similarity that incorporates both local and higher-order neighborhood information

2.

Efficiency: Do not need to consider all node pairs when training; only need to consider pairs that co-occur on random walks

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 25 10/15/19

¡ Intuition: Find embedding of nodes to

d-dimensions that preserves similarity

¡ Idea: Learn node embedding such that nearby

nodes are close together in the network

¡ Given a node 𝑣, how do we define nearby

nodes?

§ 𝑂7 𝑣 … neighbourhood of 𝑣 obtained by some strategy 𝑆

26 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 10/15/19

¡ Given 𝐻 = (𝑊, 𝐹), ¡ Our goal is to learn a mapping 𝑨: 𝑣 → ℝ&. ¡ Log-likelihood objective:

max

B

C

D ∈F

log P(𝑂J(𝑣)| 𝑨D)

§ where 𝑂7(𝑣) is neighborhood of node 𝑣 by strategy 𝑆

¡ Given node 𝑣, we want to learn feature

representations that are predictive of the nodes in its neighborhood 𝑂J(𝑣)

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 27

1.

Run short fixed-length random walks starting from each node on the graph using some strategy R

2.

For each node 𝑣 collect 𝑂7(𝑣), the multiset*

• f nodes visited on random walks starting

from u

3.

Optimize embeddings according to: Given node 𝑣, predict its neighbors 𝑂J(𝑣) max

B

C

D ∈F

log P(𝑂J(𝑣)| 𝑨D)

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 28

*𝑂7(𝑣) can have repeat elements since nodes can be visited multiple times on random walks

10/15/19

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 30

• Intuition: Optimize embeddings to maximize

likelihood of random walk co-occurrences

• Parameterize 𝑄(𝑤|𝒜𝑣) using softmax:

L = X

u∈V

X

v∈NR(u)

− log(P(v|zu))

P(v|zu) = exp(z>

u zv)

P

n2V exp(z> u zn)

10/15/19

Why softmax? We want node 𝑤 to be most similar to node 𝑣 (out of all nodes 𝑜). Intuition: ∑R exp 𝑦R ≈ max

R

exp(𝑦R)

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 31

Putting it all together:

sum over all nodes 𝑣 sum over nodes 𝑤 seen on random walks starting from 𝑣 predicted probability of 𝑣 and 𝑤 co-occuring on random walk

Optimizing random walk embeddings = Finding embeddings zu that minimize L

L = X

u2V

X

v2NR(u)

− log ✓ exp(z>

u zv)

P

n2V exp(z> u zn)

◆

10/15/19

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 32

But doing this naively is too expensive!!

Nested sum over nodes gives O(|V|2) complexity!

L = X

u2V

X

v2NR(u)

− log ✓ exp(z>

u zv)

P

n2V exp(z> u zn)

◆

10/15/19

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 33

But doing this naively is too expensive!! The normalization term from the softmax is the culprit… can we approximate it?

L = X

u2V

X

v2NR(u)

− log ✓ exp(z>

u zv)

P

n2V exp(z> u zn)

◆

10/15/19

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 34

sigmoid function

(makes each term a “probability” between 0 and 1)

random distribution over all nodes

log ✓ exp(z>

u zv)

P

n2V exp(z> u zn)

◆ ≈ log(σ(z>

u zv)) − k

X

i=1

log(σ(z>

u zni)), ni ∼ PV

10/15/19

¡ Solution: Negative sampling

Instead of normalizing w.r.t. all nodes, just normalize against 𝑙 random “negative samples” 𝑜R

Why is the approximation valid? Technically, this is a different objective. But Negative Sampling is a form of Noise Contrastive Estimation (NCE) which approx. maximizes the log probability of softmax. New formulation corresponds to using a logistic regression (sigmoid func.) to distinguish the target node 𝑤 from nodes 𝑜R sampled from background distribution 𝑄

\.

More at https://arxiv.org/pdf/1402.3722.pdf

log ✓ exp(z>

u zv)

P

n2V exp(z> u zn)

◆ ≈ log(σ(z>

u zv)) − k

X

i=1

log(σ(z>

u zni)), ni ∼ PV

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 35

random distribution

• ver all nodes

§ Sample 𝑙 negative nodes proportional to degree § Two considerations for 𝑙 (# negative samples):

• 1. Higher 𝑙 gives more robust estimates
• 2. Higher 𝑙 corresponds to higher bias on negative events

In practice 𝑙 =5-20

10/15/19

1.

Run short fixed-length random walks starting from each node on the graph using some strategy R.

2.

For each node u collect NR(u), the multiset of nodes visited on random walks starting from u

3.

Optimize embeddings using Stochastic Gradient Descent:

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 36

We We can efficiently approximate this using ne negative sampling ng!

L = X

u∈V

X

v∈NR(u)

− log(P(v|zu))

10/15/19

¡ So far we have described how to optimize

embeddings given random walk statistics

¡ What strategies should we use to run these

random walks?

§ Simplest idea: Just run fixed-length, unbiased random walks starting from each node (i.e., DeepWalk from Perozzi et al., 2013).

§ The issue is that such notion of similarity is too constrained

§ How can we generalize this?

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 37 10/15/19

¡ Goal: Embed nodes with similar network

neighborhoods close in the feature space

¡ We frame this goal as a maximum likelihood

• ptimization problem, independent to the

downstream prediction task

¡ Key observation: Flexible notion of network

neighborhood 𝑂7(𝑣) of node 𝑣 leads to rich node embeddings

¡ Develop biased 2nd order random walk 𝑆 to

generate network neighborhood 𝑂7(𝑣) of node 𝑣

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 38

Idea: use flexible, biased random walks that can trade off between local and global views of the network (Grover and Leskovec, 2016).

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 39

u s3 s2

s1

s4 s8 s9 s6 s7 s5

BFS DFS

10/15/19

Two classic strategies to define a neighborhood 𝑂7 𝑣 of a given node 𝑣: Walk of length 3 (𝑂7 𝑣 of size 3):

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 40

𝑂^_` 𝑣 = { 𝑡c, 𝑡d, 𝑡e} 𝑂g_` 𝑣 = { 𝑡h, 𝑡i, 𝑡j} Local microscopic view Global macroscopic view

u s3 s2

s1

s4 s8 s9 s6 s7 s5

BFS DFS

10/15/19

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 41

BFS: Micro-view of neighbourhood

u

DFS: Macro-view of neighbourhood

10/15/19

Biased fixed-length random walk 𝑺 that given a node 𝒗 generates neighborhood 𝑶𝑺 𝒗

¡ Two parameters:

§ Return parameter 𝒒:

§ Return back to the previous node

§ In-out parameter 𝒓:

§ Moving outwards (DFS) vs. inwards (BFS) § Intuitively, 𝑟 is the “ratio” of BFS vs. DFS

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 42 10/15/19

Biased 2nd-order random walks explore network neighborhoods:

§ Rnd. walk just traversed edge (𝑡c, 𝑥) and is now at 𝑥 § Insight: Neighbors of 𝑥 can only be: Idea: Remember where that walk came from

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 43

s1 s2 w s3 u

Back k to 𝒕𝟐 Sam Same e distan ance ce to 𝒕𝟐 Fa Farthe her fr from 𝒕𝟐

10/15/19

¡ Walker came over edge (sc, w) and is at w.

Where to go next?

¡ 𝑞, 𝑟 model transition probabilities

§ 𝑞 … return parameter § 𝑟 … ”walk away” parameter

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 44

1 1/𝑟 1/𝑞

1/𝑞, 1/𝑟, 1 are

unnormalized probabilities

s1 s2 w s3 u

10/15/19

s4

1/𝑟

¡ Walker came over edge (sc, w) and is at w.

Where to go next?

§ BFS-like walk: Low value of 𝑞 § DFS-like walk: Low value of 𝑟

𝑂7(𝑣) are the nodes visited by the biased walk

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 45

w →

s1 s2 s3 s4 1/𝑞 1 1/𝑟 1/𝑟

Unnormalized transition prob. segmented based

• n distance from 𝑡!

10/15/19

• Dist. (𝒕𝟐, 𝒖)

1 2 2 1 1/𝑟 1/𝑞

s1 s2 w s3 u s4

1/𝑟

Target 𝒖 Prob.

¡ 1) Compute random walk probabilities ¡ 2) Simulate 𝑠 random walks of length 𝑚 starting

from each node 𝑣

¡ 3) Optimize the node2vec objective using

Stochastic Gradient Descent Linear-time complexity All 3 steps are individually parallelizable

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 46 10/15/19

¡ How to use embeddings 𝒜𝒋 of nodes:

§ Clustering/community detection: Cluster points 𝑨R § Node classification: Predict label 𝑔(𝑨R) of node 𝑗 based on 𝑨R § Link prediction: Predict edge (𝑗, 𝑘) based on 𝑔(𝑨R, 𝑨

})

§ Where we can: concatenate, avg, product, or take a difference between the embeddings:

§ Concatenate: 𝑔(𝑨R, 𝑨

})= 𝑕([𝑨R, 𝑨 }])

§ Hadamard: 𝑔(𝑨R, 𝑨

})= 𝑕(𝑨R ∗ 𝑨 }) (per coordinate product)

§ Sum/Avg: 𝑔(𝑨R, 𝑨

})= 𝑕(𝑨R + 𝑨 })

§ Distance: 𝑔(𝑨R, 𝑨

})= 𝑕(||𝑨R − 𝑨 }||d)

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 50

¡ Basic idea: Embed nodes so that distances in

embedding space reflect node similarities in the original network.

¡ Different notions of node similarity:

§ Adjacency-based (i.e., similar if connected) § Multi-hop similarity definitions § Random walk approaches (covered today)

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 51 10/15/19

¡ So what method should I use..? ¡ No one method wins in all cases….

§ E.g., node2vec performs better on node classification while multi-hop methods performs better on link prediction (Goyal and Ferrara, 2017 survey)

¡ Random walk approaches are generally more

efficient

¡ In general: Must choose definition of node

similarity that matches your application!

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 52 10/15/19

An Application of Embeddings to the Knowledge Graph:

Bordes, Usunier, Garcia-Duran. NeurIPS 2013.

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 54 Pierre-Yves Vandenbussche

Nodes are referred to as en enti titi ties es, edges as re relations

A kn knowledge graph is composed of facts/statements about inter-related entities In KGs, edges can be of many types!

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 55

missing relation KG incompleteness can substantially affect the efficiency

• f systems relying on it!

IN INTUIT ITION ION: : we want a link prediction model that learns from local and global connectivity patterns in the KG, taking into account entities and relationships

• f different types at the same time.

DOWNSTREAM TASK: relation predictions are performed by using the learned patterns to generalize observed relationships between an entity of interest and all the other entities.

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 56

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 57

¡ In TransE, relationships between entities are

represented as triplets

§ 𝒊 (head entity), 𝒎 (relation), 𝒖 (tail entity) => (ℎ, 𝑚, 𝑢)

¡ Entities are first embedded in an entity space 𝑆ˆ

§ similarly to the previous methods

¡ Relations are represented as translations

§ ℎ + 𝑚 ≈ 𝑢 if the given fact is true § else, ℎ + 𝑚 ≠ 𝑢

𝑚

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 58

Entities and relations are initialized uniformly, and normalized

Negative sampling with triplet that does not appear in the KG Co Comparative loss: favors lower distance values for valid triplets, high distance values for corrupted ones

positive sample negative sample

¡ Goal: Want to embed an entire graph 𝐻 ¡ Tasks:

§ Classifying toxic vs. non-toxic molecules § Identifying anomalous graphs

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 60

𝒜Š

Simple idea:

¡ Run a standard graph embedding

technique on the (sub)graph 𝐻

¡ Then just sum (or average) the node

embeddings in the (sub)graph 𝐻

¡ Used by Duvenaud et al., 2016 to classify

molecules based on their graph structure

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 61

𝑨Š = C

\∈Š

𝑨\

10/15/19

¡ Idea: Introduce a “virtual node” to represent

the (sub)graph and run a standard graph embedding technique

¡ Proposed by Li et al., 2016 as a general

technique for subgraph embedding

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 62 10/15/19

States in anonymous walk correspond to the index of the first time we visited the node in a random walk

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 63

Anonymous Walk Embeddings, ICML 2018 https://arxiv.org/pdf/1805.11921.pdf

Number of anonymous walks grows exponentially:

§ There are 5 anon. walks 𝑏R of length 3: 𝑏c=111, 𝑏d=112, 𝑏e= 121, 𝑏h= 122, 𝑏i= 123

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 64

¡ Enumerate all possible anonymous walks 𝑏R of 𝑚

steps and record their counts

¡ Represent the graph as a probability distribution

• ver these walks

¡ For example:

§ Set 𝑚 = 3 § Then we can represent the graph as a 5-dim vector

§ Since there are 5 anonymous walks 𝑏R of length 3: 111, 112, 121, 122, 123

§ 𝑎Š[𝑗] = probability of anonymous walk 𝑏R in 𝐻

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 65

¡ Complete counting of all anonymous walks in a

large graph may be infeasible

¡ Sampling approach to approximating the true

distribution: Generate independently a set of 𝑛 random walks and calculate its corresponding empirical distribution of anonymous walks

¡ How many random walks 𝑛 do we need?

§ We want the distribution to have error of more than 𝜁 with prob. less than 𝜀:

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 66

where: 𝜃 is the total number of anon. walks of length 𝑚.

For example: There are 𝜃 = 877 anonymous walks of length 𝑚 = 7. If we set 𝜁 = 0.1 and 𝜀 = 0.01 then we need to generate 𝑛=122500 random walks

Learn embedding 𝒜𝒋 of every anonymous walk 𝒃𝒋

¡ The embedding of a graph 𝐻 is then

sum/avg/concatenation of walk embeddings zR How to embed walks?

¡ Idea: Embed walks s.t.

next walk can be predicted

§ Set zR s.t. we maximize 𝑄 𝑥—

D 𝑥—˜™ D

, … , 𝑥—

D = 𝑔(𝑨)

§ Where 𝑥—

D is a 𝑢-th random

walk starting at node 𝑣

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 67

¡ Run 𝑼 different random walks from 𝒗 each of length 𝒎:

𝑂7 𝑣 = 𝑥c

D, 𝑥d D … 𝑥œ D

§ Let 𝑏R be its anonymous version of walk 𝑥R

¡ Learn to predict walks that co-occur in 𝚬-size window ¡ Estimate embedding 𝑨R of anonymous walk 𝑏R of 𝑥R:

max 1 𝑈 C

—Ÿ™ œ

log 𝑄(𝑥—|𝑥—˜™, … , 𝑥—˜c)

where: Δ… context window size § 𝑄 𝑥— 𝑥—˜™, … , 𝑥—˜c =

¡¢£(¤ ¥¦ ) ∑§

¨ ¡¢£(¤(¥§))

§ 𝑧 𝑥— = 𝑐 + 𝑉 ⋅

c ™ ∑RŸc ™

𝑨R

§ where 𝑐 ∈ ℝ, 𝑉 ∈ ℝg, 𝑨R is the embedding of 𝑏R (anonymized version of walk 𝑥R)

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 68

Anonymous Walk Embeddings, ICML 2018 https://arxiv.org/pdf/1805.11921.pdf

We discussed 3 ideas to graph embeddings

¡ Approach 1: Embed nodes and sum/avg them ¡ Approach 2: Create super-node that spans the

(sub) graph and then embed that node

¡ Approach 3: Anonymous Walk Embeddings

§ Idea 1: Represent the graph via the distribution over all the anonymous walks § Idea 2: Sample the walks to approximate the distribution § Idea 3: Embed anonymous walks

10/15/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 69