BERT4Rec : Sequential Recommendation with Bidirectional Encoder - - PowerPoint PPT Presentation

▶

Sep 10, 2023 3.28k likes •3.63k views

BERT4Rec : Sequential Recommendation with Bidirectional Encoder Representations from Transformer Advisor: Jia-Ling Koh Presenter: You-Xiang Chen Source: CIKM19 Data: 2020/04/20 INTRODUCTION Introduction target item Sequential

SLIDE 1

BERT4Rec : Sequential Recommendation with Bidirectional Encoder Representations from Transformer

Advisor: Jia-Ling Koh Presenter: You-Xiang Chen Source: CIKM’19 Data: 2020/04/20

SLIDE 2

INTRODUCTION

SLIDE 3

Introduction

Sequential Recommendation

historical subsequence target item Recommender system

SLIDE 4

Motivation & Goal

▪ Unidirectional models often assume a rigidly ordered sequence

ver data which is not always true for user behaviors in real-world

applications. Proposing bidirectional self-attention network - BERT4Rec

SLIDE 5

Motivation & Goal

▪ Conventional bidirectional models encode each historical subsequence to predict the target item. ▪ This approach is very time and resources consuming since we need to create a new sample for each position in the sequence and predict them separately. Introducing the Cloze task to produce more samples to train a more powerful model.

target item historical subsequence

SLIDE 6

METHOD

SLIDE 7

Problem Statement

Interaction sequence Sets of user & item Output

SLIDE 8

Framework

SLIDE 9

Embedding Layer

Input representation

d-dim. 𝒊𝟐

𝟏

position embedding matrix item embedding matrix

SLIDE 10

Transformer Layer

Multi-Head Self-Attention Scaled Dot-Product Attention

𝑰𝒎 = [𝒊𝟐

𝒎 , 𝒊𝟑 𝒎 , … , 𝒊𝒖 𝒎]

𝑜 𝑜 projects 𝑰𝒎 into 𝒐 subspaces

SLIDE 11

Transformer

SLIDE 12

Multi-Head Attention

SLIDE 13

Transformer Layer

Position-wise Feed-Forward Network

Gaussian Error Linear Unit (GELU) activation function separately and identically at each position

SLIDE 14

Gaussian Error Linear Units

https://arxiv.org/pdf/1606.08415.pdf

SLIDE 15

Transformer Layer

Stacking Transformer Layer

LN(·) : layer normalization function

https://arxiv.org/pdf/1607.06450.pdf

SLIDE 16

Output Layer

𝑿𝑸: 𝑴𝒇𝒃𝒐𝒃𝒄𝒎𝒇 𝒒𝒔𝒑𝒌𝒇𝒅𝒖𝒋𝒑𝒐 𝒏𝒃𝒖𝒔𝒋𝒚 𝑭: 𝑭𝒏𝒄𝒇𝒆𝒆𝒋𝒐𝒉 𝑵𝒃𝒖𝒔𝒋𝒚 𝒑𝒈 𝒋𝒖𝒇𝒏𝒕

SLIDE 17

Model Learning

𝒏𝒃𝒕𝒍𝒇𝒆 𝒘𝒇𝒔𝒕𝒋𝒑𝒐 𝒈𝒑𝒔 𝒗𝒕𝒇𝒔 𝒄𝒇𝒊𝒃𝒘𝒋𝒑𝒔 𝒖𝒊𝒇 𝒏𝒃𝒕𝒍𝒇𝒆 𝒋𝒖𝒇𝒏𝒕

SLIDE 18

EXPERIMENT

SLIDE 19

Datasets

SLIDE 20

Baselines

▪ POP ▪ BPR-MF ▪ NCF ▪ FPMC ▪ GRU4Rec + ▪ Caser ▪ SASRec

markov chain 𝒐𝒑𝒐 − 𝒕𝒇𝒓𝒗𝒇𝒐𝒖𝒋𝒃𝒎 𝒕𝒇𝒓𝒗𝒇𝒐𝒖𝒋𝒃𝒎

SLIDE 21

Evaluation metrics

𝐼𝑆@𝐿 = 𝑂𝑣𝑛𝑐𝑓𝑠 𝑝𝑔 𝐼𝑗𝑢𝑡 @ 𝐿 𝐻𝑈

Hit Ratio

𝐸𝐷𝐻𝑙 = ෍

𝑗=1 𝑙

2𝑠𝑓𝑚𝑗 − 1 log2(𝑗 + 1) 𝑂𝐸𝐷𝐻@𝐿 = 𝐸𝐷𝐻@𝐿 𝐽𝐸𝐷𝐻

Normalized Discounted cumulative gain Mean Reciprocal Rank

𝑁𝑆𝑆 = 1 𝑅 ෍

𝑗=1 𝑅

1 𝑠𝑏𝑜𝑙𝑗

SLIDE 22

Performance

𝒙𝒑𝒔𝒕𝒖 𝒐𝒑𝒐 − 𝒕𝒇𝒓𝒗𝒇𝒐𝒖𝒋𝒃𝒎 𝒕𝒇𝒓𝒗𝒇𝒐𝒖𝒋𝒃𝒎 𝑼𝒔𝒃𝒐𝒕𝒈𝒑𝒔𝒏𝒇𝒔

SLIDE 23

Analysis on Bidirection and Cloze

SLIDE 24

CONCLUSION

▪ We introduce a deep bidirectional sequential model called BERT4Rec for sequential recommendation. ▪ For model training, we introduce the Cloze task which predicts the masked items using both left and right context. ▪ Extensive experimental results on four real-world datasets show that our model

utperforms state-of-the-art baselines.