Predicting Temporal Sets with Deep Neural Networks Le Yu, Leilei - - PowerPoint PPT Presentation

▶

Aug 31, 2023 243 likes •441 views

Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 20), August 23 27, 2020, Virtual Event, CA, USA. Predicting Temporal Sets with Deep Neural Networks Le Yu, Leilei Sun*, Bowen Du, Chuanren Liu, Hui

SLIDE 1

Predicting Temporal Sets with Deep Neural Networks

Le Yu, Leilei Sun*, Bowen Du, Chuanren Liu, Hui Xiong, Weifeng Lv Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’20), August 23–27, 2020, Virtual Event, CA, USA.

SLIDE 2

Background

Formalization

Motivation CONTENTS

Experiments

Methodology

SLIDE 3

Background

SLIDE 4

Baskets in shopping

Background

Drugs in healthcare Places in traveling Courses in schools

SLIDE 5

Information loss

Related Work

Existing methods usually follow a two-stage strategy (Yu et al. 2016, Hu and He, 2019): (1) set embedding (2) sequential behaviors learning The two-stage approach often leads to information loss.

SLIDE 6

A unique perspective

How would the model perform if we make prediction of temporal sets from a different perspective? Set Embedding Sequential Behaviors Learning

+

Existing methods: Lead to information loss

Learning the relationship of elements

Leverage information in elements relationship as much as possible This work:

SLIDE 7

Formalization

⚫ Let 𝕍 = 𝑣1, 𝑣2, ⋯ , 𝑣𝑜 , 𝕎 = {𝑤1, 𝑤2, ⋯ , 𝑤𝑛} denote the set of 𝑜 users and 𝑛 elements, a set 𝑇 ⊂ 𝕎 denotes the collection of elements. ⚫ Given：a sequence of sets 𝕋𝑗 = 𝑇𝑗

1, 𝑇𝑗 2, ⋯ , 𝑇𝑗 𝑈 that records the

historical behaviors of user 𝑣𝑗 ∈ 𝕍 ⚫ Goal：predict the next-period set of 𝑣𝑗, መ 𝑇𝑗

𝑈+1 = 𝑔 𝑇𝑗 1, 𝑇𝑗 2, ⋯ , 𝑇𝑗 𝑈, 𝑿 ,

where 𝑿 is the trainable parameter.

SLIDE 8

Learn temporal dependencies of elements in the sequence of sets Learn set-level element relationship Fuse static and dynamic information by a gated updating mechanism

Our Method: DNNTSP

SLIDE 9

2) Weighted Convolutions on Dynamic Graphs

𝒅𝑗,𝑘

𝑢,𝑚+1 = 𝜏

𝒄𝑚 + ෍

𝑙∈𝒪

𝑗,𝑘 𝑢 ∪{𝑘}

𝐵𝑗

𝑢 𝑘, 𝑙 · 𝑿𝑚𝒅𝑗,𝑙 𝑢,𝑚

1) Weighted Graphs Construction

Element Relationship Learning

… … Learn element relationship in the same set.

trainable convolutional parameters

𝑤𝑗,1, 𝑤𝑗,2 , 𝑤𝑗,1, 𝑤𝑗,3 , 𝑤𝑗,2, 𝑤𝑗,1 , 𝑤𝑗,2, 𝑤𝑗,3 , 𝑤𝑗,3, 𝑤𝑗,1 , (𝑤𝑗,3, 𝑤𝑗,2) 𝑤𝑗,1, 𝑤𝑗,3 , 𝑤𝑗,1, 𝑤𝑗,4 , 𝑤𝑗,3, 𝑤𝑗,1 , 𝑤𝑗,3, 𝑤𝑗,4 , 𝑤𝑗,4, 𝑤𝑗,1 , (𝑤𝑗,4, 𝑤𝑗,3) 𝑤𝑗,1, 𝑤𝑗,2, 2 , 𝑤𝑗,1, 𝑤𝑗,3, 3 , 𝑤𝑗,1, 𝑤𝑗,4, 1 , 𝑤𝑗,2, 𝑤𝑗,1, 2 , … 𝑤𝑗,1, 𝑤𝑗,1, 1 , 𝑤𝑗,2, 𝑤𝑗,2, 1 , 𝑤𝑗,3, 𝑤𝑗,3, 1 , 𝑤𝑗,4, 𝑤𝑗,4, 1

…

𝑤𝑗,1, 𝑤𝑗,2, 0.67 , 𝑤𝑗,1, 𝑤𝑗,3, 1.0 , 𝑤𝑗,1, 𝑤𝑗,4, 0.33 , 𝑤𝑗,2, 𝑤𝑗,1, 0.67 , … 𝑤𝑗,1, 𝑤𝑗,1, 0.33 , 𝑤𝑗,2, 𝑤𝑗,2, 0.33 , 𝑤𝑗,3, 𝑤𝑗,3, 0.33 , 𝑤𝑗,4, 𝑤𝑗,4, 0.33

(a) (b) (c) (d)

SLIDE 10

𝒂𝑗,𝑘 = 𝑡𝑝𝑔𝑢𝑛𝑏𝑦

𝑫𝑗,𝑘𝑿𝑟 𝑫𝒋,𝑘𝑿𝑙

𝐺′′

+ 𝑵𝑗 ⋅ (𝑫𝑗,𝑘𝑿𝑤)

Attention-based Temporal Dependency Learning

Learn element temporal dependency among the sequence of sets.

𝒜𝑗,𝑘 = 𝒂𝑗,𝑘 · 𝒙𝑏𝑕𝑕

T · 𝒂𝑗,𝑘 T

a trainable parameter to learn the importance of different timestamps 𝑁𝑗

𝑢,𝑢′ = ቊ 0, 𝑗𝑔 𝑢 ≤ 𝑢′

−∞, 𝑝𝑢ℎ𝑓𝑠𝑥𝑗𝑡𝑓 is a masked matrix.

SLIDE 11

Gated Information Fusing

𝑭𝑗,𝐽 𝑘

𝑣𝑞𝑒𝑏𝑢𝑓 = 1 − 𝛾𝑗,𝐽 𝑘 · 𝛿𝐽 𝑘

· 𝑭𝑗,𝐽 𝑘 + (𝛾𝑗,𝐽 𝑘 · 𝛿𝐽 𝑘 ) · 𝒜𝑗,𝑘

Mine shared patterns and fuse static and dynamic representations of elements.

an indicator vector controls the importance of static and dynamic representations

SLIDE 12

The Learning Process

Prediction:

ෝ 𝒛𝑗 = 𝑡𝑗𝑕𝑛𝑝𝑗𝑒 𝑭𝑗

𝑣𝑞𝑒𝑏𝑢𝑓𝒙𝑝 + 𝒄𝑝

Loss function:

𝑀 = − 1 𝑂 ෍

𝑗 𝑂 1

𝑛 ෍

𝑘 𝑛

𝑧𝑗,𝑘 log ො 𝑧𝑗,𝑘 + (1 − 𝑧𝑗,𝑘) log 1 − ො 𝑧𝑗,𝑘 + 𝝁 𝑿

𝟑

SLIDE 13

Datasets and Baselines

Four datasets are used for evaluation, i.e. TaFeng, DC, TaoBao and TMS. Both classical and state-of-the-art methods: TOP, PersonalTOP, ElementTransfer, DREAM and Sets2Sets Three evaluation metrics: Recall, NDCG and PHR Statistics of the datasets:

SLIDE 14

Our model outperforms existing methods with a significant margin.

Experimental Results

SLIDE 15

Our model is applicable to scenarios with sparse data.

Experimental Results

Our model is better than Sets2Sets- when no empirical information is added.
Our method outperforms Sets2Sets when incorporating the component for modelling

repeated behaviors.

SLIDE 16

next-basket recommendation travel-package recommendation

… 𝑈 + 1

customer

time

? ? ?

automatic treatment

… 𝑈 + 1

patient

time

? ? ?

… 𝑈 + 1

tourist

time

? ? ?

courses planning

… 𝑈 + 1 time

? ? ?

student

Applications

SLIDE 17

Website: https://www.brilliantasus.com/

Our Team

SLIDE 18

Predicting Temporal Sets with Deep Neural Networks

Le Yu, Leilei Sun*, Bowen Du, Chuanren Liu, Hui Xiong, Weifeng Lv Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’20), August 23–27, 2020, Virtual Event, CA, USA.

Background

Formalization

Motivation CONTENTS

Experiments

Methodology

Background

Baskets in shopping

Background

Drugs in healthcare Places in traveling Courses in schools

Related Work

Existing methods usually follow a two-stage strategy (Yu et al. 2016, Hu and He, 2019): (1) set embedding (2) sequential behaviors learning The two-stage approach often leads to information loss.

A unique perspective

How would the model perform if we make prediction of temporal sets from a different perspective? Set Embedding Sequential Behaviors Learning

+

Learning the relationship of elements

Formalization

⚫ Let 𝕍 = 𝑣1, 𝑣2, ⋯ , 𝑣𝑜 , 𝕎 = {𝑤1, 𝑤2, ⋯ , 𝑤𝑛} denote the set of 𝑜 users and 𝑛 elements, a set 𝑇 ⊂ 𝕎 denotes the collection of elements. ⚫ Given：a sequence of sets 𝕋𝑗 = 𝑇𝑗

historical behaviors of user 𝑣𝑗 ∈ 𝕍 ⚫ Goal：predict the next-period set of 𝑣𝑗, መ 𝑇𝑗

where 𝑿 is the trainable parameter.

Learn temporal dependencies of elements in the sequence of sets Learn set-level element relationship Fuse static and dynamic information by a gated updating mechanism

Our Method: DNNTSP

2) Weighted Convolutions on Dynamic Graphs

1) Weighted Graphs Construction

Element Relationship Learning

… … Learn element relationship in the same set.

trainable convolutional parameters

…

𝒂𝑗,𝑘 = 𝑡𝑝𝑔𝑢𝑛𝑏𝑦

+ 𝑵𝑗 ⋅ (𝑫𝑗,𝑘𝑿𝑤)

Attention-based Temporal Dependency Learning

Learn element temporal dependency among the sequence of sets.

𝒜𝑗,𝑘 = 𝒂𝑗,𝑘 · 𝒙𝑏𝑕𝑕

a trainable parameter to learn the importance of different timestamps 𝑁𝑗

−∞, 𝑝𝑢ℎ𝑓𝑠𝑥𝑗𝑡𝑓 is a masked matrix.

Gated Information Fusing

𝑭𝑗,𝐽 𝑘

· 𝑭𝑗,𝐽 𝑘 + (𝛾𝑗,𝐽 𝑘 · 𝛿𝐽 𝑘 ) · 𝒜𝑗,𝑘

Mine shared patterns and fuse static and dynamic representations of elements.

an indicator vector controls the importance of static and dynamic representations

The Learning Process

Prediction:

Loss function:

Datasets and Baselines

Four datasets are used for evaluation, i.e. TaFeng, DC, TaoBao and TMS. Both classical and state-of-the-art methods: TOP, PersonalTOP, ElementTransfer, DREAM and Sets2Sets Three evaluation metrics: Recall, NDCG and PHR Statistics of the datasets:

Our model outperforms existing methods with a significant margin.

Experimental Results

Our model is applicable to scenarios with sparse data.

Experimental Results

repeated behaviors.

next-basket recommendation travel-package recommendation

… 𝑈 + 1

customer

time

? ? ?

automatic treatment

… 𝑈 + 1

patient

time

? ? ?

… 𝑈 + 1

tourist

time

? ? ?

courses planning

… 𝑈 + 1 time

? ? ?

student

Applications

Website: https://www.brilliantasus.com/

Our Team

Thanks!

Contact

Le Yu yule@buaa.edu.cn

Reference

Le Yu, Leilei Sun*, Bowen Du, Chuanren Liu, Hui Xiong, Weifeng Lv, Predicting Temporal Sets with Deep Neural Networks, KDD 2020

Code and data

https://github.com/yule-BUAA/DNNTSP