CSE 6240: Web Search and Text Mining. Spring 2020 Temporal Graph Representation Learning Rakshit Trivedi School of Computational Science and Engineering rstrivedi@gatech.edu 1 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
Today’s Lecture • GraphSAGE • Dynamic Graphs and its Applications • Representation Learning with: – Discrete-Time Approaches – Continuous-Time Approaches 2 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
[Hamilton et al., NIPS 2017] GraphSAGE Idea • In GCN, we aggregated the neighbors’ messages as the (weighted) average of all neighbors . How can we generalize this? A ? C TARGET NODE B B A A C B ? ? C A E F D F E ? D A INPUT GRAPH 3 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
GraphSAGE Idea Any differentiable function that maps set of vectors in 𝑂(𝑣) to a single vector A C TARGET NODE B B A A C B C A E F D F E D A INPUT GRAPH h k A k · agg ( { h k − 1 , ∀ u ∈ N ( v ) } ) , B k h k − 1 �⇥ ⇤� v = σ u v 4 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
Neighborhood Aggregation • Simple neighborhood aggregation: 0 1 h k − 1 X | N ( v ) | + B k h k − 1 u h k v = σ @ W k A v u ∈ N ( v ) Concatenate neighbor embedding • GraphSAGE: and self embedding h k { h k − 1 , B k h k − 1 �⇥ � � ⇤� v = σ W k · agg , ∀ u ∈ N ( v ) } u v Generalized aggregation 5 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
Neighbor Aggregation: Variants • Mean: Take a weighted average of neighbors h k − 1 X u agg = | N ( v ) | u ∈ N ( v ) • Pool: Transform neighbor vectors and apply symmetric vector function Element-wise mean/max { Qh k − 1 � � agg = γ , ∀ u ∈ N ( v ) } u • LSTM: Apply LSTM to reshuffled of neighbors [ h k − 1 � � agg = LSTM , ∀ u ∈ π ( N ( v ))] u 6 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
Experiments: Dataset • Dynamic datasets: – Citation Network: Predict paper category • Data from 2000-2005 • 302,424 nodes • Train: data till 2004, test: 2005 data – Reddit Post Network: Predict subreddit of post • Nodes = posts • Edges between posts if common users comment on the post • 232,965 posts • Train: 20 days of data, test: next 10 days of data 7 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
Experiments: Results 8 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
Summary: GCN and GraphSAGE • Key idea: Generate node embeddings based on local neighborhoods – Nodes aggregate “messages” from their neighbors using neural networks • Graph convolutional networks: – Basic variant: Average neighborhood information and stack neural networks • GraphSAGE: – Generalized neighborhood aggregation 9 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
Today’s Lecture • GraphSAGE • Dynamic Graphs and its Applications • Representation Learning with: – Discrete-Time Approaches – Continuous-Time Approaches 10 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
Temporally Evolving Graphs Social media E-commerce Education Web Finance IoT 11 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
Temporally Evolving Graphs (i) How to model dynamics over graphs? (ii) How leverage such a dynamic graph model to encode evolving graph information into low-dimensional representations? 12 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
Application: Social Networks 13 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
Application: Recommendation Systems 14 . . . … … . . . Features Time …... …... …... Users Products Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
Application: Anomaly Detection [Image from NetWalk presentation, Yu et. al. KDD 2018] 15 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
How Do We Model Dynamics? 1. Snapshot-Based Observation: – Network Evolution observed as a collection of snapshots of the graph at different time steps – Possibly significant changes in graph structure observed between the two-time steps – Time information may or may not be explicitly available – Demand Discrete-time modeling 2. Event Based Observation: – Network Evolution observed as time-stamped edges (each edge represent an event) – Time information is fine-grained and explicitly available – Demand Continuous-time modeling 16 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
Today’s Lecture • GraphSAGE • Dynamic Graphs and its Applications • Representation Learning with: – Discrete-Time Approaches – Continuous-Time Approaches 17 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
[Morris 2000] Snapshot based Evolution of Graphs Ti Time t1 Ti Time t2 Ti Time t3 • Let denote the graph at time t 𝑯 𝒖 = (𝑾 𝒖 , 𝑭 𝒖 ) • Let 𝑩 𝒖 be the corresponding adjacency matrix at time t • Dynamic graph 𝑯 = {𝑯 𝟐 , 𝑯 𝟑 , … , 𝑯 𝑼 } is the series of graph snapshots recorded at T different time steps 18 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
[Morris 2000] Snapshot based Evolution of Graphs Embe Em beddi dding ngs RNN Em Embe beddi dding ngs RNN Em Embe beddi dding ngs GE GE GE Ti Time t1 Time t2 Ti Time t3 Ti • One Approach: Use a single graph encoder at each time step to extract node features • Use RNN based model over these node features to model dynamics What problems could this potentially have? 19 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
[Morris 2000] Snapshot based Evolution of Graphs Em Embe beddi dding ngs RNN Embe Em beddi dding ngs RNN Embe Em beddi dding ngs GE GE GE Ti Time t1 Time t2 Ti Ti Time t3 • Number of Nodes and edges vary with time step • Above approach would require complete knowledge of nodes • Doesn’t perform well in practice 20 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
General Model Em Embe beddi dding ngs Embe Em beddi dding ngs Embe Em beddi dding ngs GE(t2) GE(t3) GE(t1) Capture Ca Ca Capture Dy Dynamics? Dy Dynamics? Time t1 Ti Ti Time t2 Ti Time t3 Alternative Approach: Use graph-specific encoder at each time step 21 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
General Model Embe Em beddi dding ngs Em Embe beddi dding ngs Embe Em beddi dding ngs GE(t2) GE(t3) GE(t1) Ca Capture Ca Capture Dy Dynamics? Dynamics? Dy Ti Time t1 Ti Time t2 Time t3 Ti Alternative Approach: Use graph-specific encoder at each time step • Adapt the architecture based on changes in graph properties • Adapt Encoder parameters to model dynamics • Train using unsupervised or semi-supervised loss as before e.g. cross-entropy loss 22 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
Variant I: Dynamic Autoencoder Architecture DynGEM: Deep Embedding Method for Dynamic Graphs [Slides for DynGEM adapted from author’s original slides, Goyal et. al. 2018] 23 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
DynGEM: Model 24 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
DynGEM: Adaptive Architecture Embedding Stability 25 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
DynGEM: Data Setup 26 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
DynGEM: Visualization 27 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
DynGEM: Link Prediction 28 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
DynGEM: Anomaly Detection 29 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
Variant II: GCN Weight Evolution EvolveGCN: Evolving Graph Convolutional Networks for Dynamic Graphs 30 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
EvolveGCN: Model EvolveGCN: Model 31 33 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
EvolveGCN: Weight Evolution • GCN Reminder: 32 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
EvolveGCN: Weight Evolution • GCN Reminder: • Weight Evolution I: (only structural properties) • Weight Evolution II: (for attributed graphs) 33 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
EvolveGCN: Weight Evolution • GCN Reminder: • Weight Evolution I: (only structural properties) • Weight Evolution II: (for attributed graphs) 34 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
EvolveGCN: Summarization What is the challenge? 35 Srijan Kumar, Georgia Tech, CSE6240 Spring 2020: Web Search and Text Mining
Recommend
More recommend
