Predicting Dynamic Embedding Trajectory in Temporal Interaction - - PowerPoint PPT Presentation

▶

Mar 01, 2023 558 likes •836 views

Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks Xikun Zhang Jure Leskovec Srijan Kumar Stanford University Stanford University UIUC Georgia Institute of Technology Code and Data: https://snap.stanford.edu/jodie 1

SLIDE 1

Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks

Srijan Kumar

Stanford University Georgia Institute of Technology

Jure Leskovec

Stanford University

Xikun Zhang

UIUC

Code and Data: https://snap.stanford.edu/jodie

SLIDE 2

Temporal Interaction Networks

Time [KDD’19]

Flexible way to represent time-evolving relations

Users Items

Feature interaction user item time features

Represented as a sequence of interactions, sorted by time:

SLIDE 3

Temporal Interaction Networks

[KDD’19] E-commerce Social media Finance Web Education IoT

Application domains Accounts Posts

…...

SLIDE 4

Temporal Interaction Networks

[KDD’19] E-commerce Social media Finance Web

Students Courses

Education IoT

Application domains

…...

SLIDE 5

Problem Setup

Given a temporal interaction network where generate an embedding trajectory of every user and an embedding trajectory of every item

[KDD’19] interaction user item time features

SLIDE 6

Goal: Generate Dynamic Trajectory

Output: Dynamic trajectory in embedding space Input: Temporal interaction network

[KDD’19] 1 2 4 3 5 6

SLIDE 7

Challenges

Challenges in modeling:

C1: How to learn inter-dependent user and item

embeddings?

C2: How to generate embedding for every point

in time?

Challenges in scalability:

C3: How to scalably train models on temporal

networks?

[KDD’19]

SLIDE 8

Existing Methods

Deep recommender systems

Time-LSTM (IJCAI 2017)
Recurrent Recommender Networks (WSDM

2017)

Latent Cross (WSDM 2018)

Dynamic co-evolution

Deep Coevolve (DLRS, 2016)

Temporal network embedding

CTDNE (BigNet, 2018)

Our model: JODIE

[KDD’19]

C1

Co- influence

C2

Embed any time

C3

Train in batches

SLIDE 9

Our Model: JODIE

JODIE: Joint Dynamic Interaction Embedding

Mutually-recursive recurrent neural network framework

[KDD’19]

Projection Operator

Project Component

User RNN Item RNN

Update Component

SLIDE 10

JODIE: Update Component

[KDD’19]

User RNN Item RNN f = Weight matrices W are trainable

All users share the User-RNN parameters. Similar

for items.

SLIDE 11

JODIE: Project Component

How can we predict the next item?

Rank items using distance in the embedding space

[KDD’19]

Projected embedding Projection operator

Time Δ

Projected embedding f = User RNN Item RNN

SLIDE 12

Summary: JODIE Formulation

Update embeddings:

[KDD’19]

Loss:

Predicted next item is close to the real item embedding Smoothness in evolving embeddings

Project user embedding: Predict next item:

SLIDE 13

Challenges in Dynamic Trajectories

Challenges in learning:

C1: How to learn inter-dependent user and item

embeddings? Solution: Update component

C2: How to generate embedding for every point in

time? Solution: Project component

Challenges in scalability:

C3: How to scalably train models on temporal

networks?

[KDD’19]

SLIDE 14

Standard Training Processes: N/A

Training must maintain temporal order

[KDD’19] (1) (2) (3) (4) . . . . . .

User 1 User 2 User 3

Split by user (or item): not allowed Sequential processing: not scalable

1 2 3 4 4 3 2 1 5 6

Batch 1

Temporal inconsistency

SLIDE 15

T-batch: Temporal data batching algorithm

Main idea: create each batch as an

independent edge set

Create a sequence of batches

– Interactions in each batch are processed in parallel – Process the batches in sequence to maintain temporal ordering

[KDD’19]

T-batch: Batching for Scalability

SLIDE 16

T-batch: Batching for Scalability

Batch 2 Batch 1 Batch 3

[KDD’19] 1 2 3 4 5 6 2 1 4 3 5 6

Iteratively select the maximal independent edge set.

SLIDE 17

Challenges in Dynamic Trajectories

Challenges in learning:

C1: How to learn inter-dependent user and item

embeddings? Solution: Update component

C2: How to generate embedding for every point in

time? Solution: Project component

Challenges in scalability:

C3: How to scalably train models on temporal

networks? Solution: T-batch Algorithm

[KDD’19]

SLIDE 18

Experiments: Prediction Tasks

Temporal Link Prediction:

– Which item i ∈ 𝐽 will user u interact with at time t?

Temporal Node Classification:

– Does a user u become anomalous after an interaction?

Settings:

– Temporal Splits: 80%, 10%, 10% – Metrics: Mean reciprocal rank, Recall@10, AUROC

[KDD’19]

Code and Data: https://snap.stanford.edu/jodie

SLIDE 19

Datasets

[KDD’19]

Dataset Users Items Interactions Temporal Anomalies Reddit 10,000 984 672,447 366 Wikipedia 8,227 1,000 157,474 217 LastFM 980 1,000 1,293,103

MOOC

7,047 97 411,749 4,066

NEW! NEW!

Code and Data: https://snap.stanford.edu/jodie

SLIDE 20

Experiment 1: Link Prediction

JODIE outperforms baselines by > 20%

Mean Reciprocal Rank

0.0 1.0 Latent Cross 0.42 0.18 Time- LSTM 0.60 RRN 0.73 0.39 0.17 CTDNE Deep Coevolve JODIE 0.2 0.4 0.6 0.8

[KDD’19]

SLIDE 21

Experiment 2: Node Classification

JODIE outperforms all baselines by >12%

AUROC

0.5 1.0 Latent Cross 0.63 0.58 Time- LSTM 0.65 RRN 0.73 0.65 0.64 CTDNE Deep Coevolve JODIE 0.6 0.7 0.8 0.9

[KDD’19]

SLIDE 22

Experiment 3: T-batch Speed-up

T-batch leads to 8.5x speed-up in training

5.1 minutes 44 minutes

JODIE without T-batch JODIE with T-batch

Running Time

50 10 20 30 40

8.5x speed-up

[KDD’19]

SLIDE 23

Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks

Srijan Kumar, Xikun Zhang, Jure Leskovec

Code and Data: https://snap.stanford.edu/jodie

JODIE generates and projects embedding trajectories

JODIE: a mutually-recursive RNN framework
T-batch: 8.5x training speed-up
Efficient in temporal link prediction and node classification
Extendible to > 2 entity types

SLIDE 24

Open Positions @ Georgia Tech

Hiring multiple Ph.D. students
Research areas:

– Machine Learning for Networks – Safety, Integrity, and Anti-Abuse – Computational Social Science

Collaborations

Contact: srijan@cs.stanford.edu