Relational Stacked Denoising Autoencoder for Tag Recommendation Hao - - PowerPoint PPT Presentation
Relational Stacked Denoising Autoencoder for Tag Recommendation Hao Wang Dept. of Computer Science and Engineering Hong Kong University of Science and Technology Joint work with Xingjian Shi and Dit-Yan Yeung To appear in AAAI 2015 Hao Wang
Hong Kong University of Science and Technology Joint work with Xingjian Shi and Dit-Yan Yeung To appear in AAAI 2015
Hao Wang Relational SDAE 1 / 36
1
Background and Related Work
2
Generalized Probabilistic SDAE
3
Relational SDAE
4
Performance Evaluation
5
Case study
6
Conclusion
Hao Wang Relational SDAE 2 / 36
1
Background and Related Work
2
Generalized Probabilistic SDAE
3
Relational SDAE
4
Performance Evaluation
5
Case study
6
Conclusion
Hao Wang Relational SDAE 3 / 36
https://www.flickr.com
Hao Wang Relational SDAE 3 / 36
http://www.citeulike.org
Hao Wang Relational SDAE 4 / 36
Hao Wang Relational SDAE 5 / 36
1 Chen et al., 2008 2 Chen et al., 2010 3 Shen and Fan, 2010
1 Garg and Weber, 2008 2 Weinberger et al., 2008 3 Rendle and Schmidt-Thieme, 2010
1 Wu et al., 2009 2 Wang and Blei, 2011 3 Yang et al., 2013 Hao Wang Relational SDAE 6 / 36
1 Chen et al., 2008 2 Chen et al., 2010 3 Shen and Fan, 2010 4 . . .
1 Tag independence 2 Interpretability 3 No New-item problem
1 Need domain knowledge Hao Wang Relational SDAE 7 / 36
1 Garg and Weber, 2008 2 Weinberger et al., 2008 3 Rendle and Schmidt-Thieme, 2010 4 . . .
1 No domain knowledge needed
1 Requires some form of rating feedback (co-occurrence matrix) 2 New-tag problem and new-item problem Hao Wang Relational SDAE 8 / 36
1 Wu et al., 2009 2 Wang and Blei, 2011 3 Yang et al., 2013 4 . . .
Hao Wang Relational SDAE 9 / 36
Hao Wang Relational SDAE 10 / 36
LDA: sparse, relatively high dimension MF: low rank, low dimension
Hao Wang Relational SDAE 11 / 36
1 Can SDAE learn effective representation for recommendation? 2 How to incorporate relational information into SDAE? 3 How is the performance? Hao Wang Relational SDAE 12 / 36
1
Background and Related Work
2
Generalized Probabilistic SDAE
3
Relational SDAE
4
Performance Evaluation
5
Case study
6
Conclusion
Hao Wang Relational SDAE 13 / 36
X0 X0 X1 X1 X2 X2 X3 X3 X4 X4 Xc Xc
min
{Wl},{bl} Xc − XL2 F + λ
Wl2
F ,
where λ is a regularization parameter and · F denotes the Frobenius norm.
Hao Wang Relational SDAE 13 / 36
1 For each layer l of the SDAE network, 1 For each column n of the weight matrix Wl, draw
Wl,∗n ∼ N(0, λ−1
w IKl).
2 Draw the bias vector bl ∼ N(0, λ−1
w IKl).
3 For each row j of Xl, draw
Xl,j∗ ∼ N(σ(Xl−1,j∗Wl + bl), λ−1
s IKl).
2 For each item j, draw a clean input
Xc,j∗ ∼ N(XL,j∗, λ−1
n IB).
Hao Wang Relational SDAE 14 / 36
1
Background and Related Work
2
Generalized Probabilistic SDAE
3
Relational SDAE
4
Performance Evaluation
5
Case study
6
Conclusion
Hao Wang Relational SDAE 15 / 36
1 Draw the relational latent matrix S from a matrix variate normal
distribution: S ∼ NK,J(0, IK ⊗ (λlLa)−1).
2 For layer l of the SDAE where l = 1, 2, . . . , L
2 − 1,
1 For each column n of the weight matrix Wl, draw
Wl,∗n ∼ N(0, λ−1
w IKl).
2 Draw the bias vector bl ∼ N(0, λ−1
w IKl).
3 For each row j of Xl, draw
Xl,j∗ ∼ N(σ(Xl−1,j∗Wl + bl), λ−1
s IKl).
3 For layer L
2 of the SDAE network, draw the representation vector
for item j from the product of two Gaussians (PoG): X L
2 ,j∗ ∼ PoG(σ(X L 2 −1,j∗Wl + bl), sT
j , λ−1 s IK, λ−1 r IK).
Hao Wang Relational SDAE 15 / 36
1 For layer l of the SDAE network where l = L
2 + 1, L 2 + 2, . . . , L,
1 For each column n of the weight matrix Wl, draw
Wl,∗n ∼ N(0, λ−1
w IKl).
2 Draw the bias vector bl ∼ N(0, λ−1
w IKl).
3 For each row j of Xl, draw
Xl,j∗ ∼ N(σ(Xl−1,j∗Wl + bl), λ−1
s IKl).
2 For each item j, draw a clean input
Xc,j∗ ∼ N(XL,j∗, λ−1
n IB).
Hao Wang Relational SDAE 16 / 36
Hao Wang Relational SDAE 17 / 36
Q
Hao Wang Relational SDAE 18 / 36
The log-likelihood: L = − λl 2 tr(SLaST ) − λr 2
(sT
j − X L
2 ,j∗)2
2
− λw 2
(Wl2
F + bl2 2)
− λn 2
XL,j∗ − Xc,j∗2
2
− λs 2
σ(Xl−1,j∗Wl + bl) − Xl,j∗2
2,
where Xl,j∗ = σ(Xl−1,j∗Wl + bl). Similar to the generalized SDAE, taking λs to infinity, the last term of the joint log-likelihood will vanish.
Hao Wang Relational SDAE 19 / 36
For S: Sk∗(t + 1) ← Sk∗(t) + δ(t)r(t) r(t) ← λrXT
L 2 ,∗k − (λlLa + λrIJ)Sk∗(t)
δ(t) ← r(t)T r(t) r(t)T (λlLa + λrIJ)r(t). For X, W, and b: Use Back Propagation.
Hao Wang Relational SDAE 20 / 36
Objective function: L = − λu 2
ui2
2 − λv
2
vj − XT
L 2 ,j∗2
2
−
cij 2 (Rij − uT
i vj)2,
where λu and λv are hyperparameters. cij is set to 1 for the existing ratings and 0.01 for the missing entries.
Hao Wang Relational SDAE 21 / 36
repeat Update S using the updating rules Update X, W, and b until convergence Get resulting representation X L
2 ,j∗
Optimize the objective function L
Rij = uT
i vj
Rank R1j,R2j,. . . ,RIj Recommend tags with largest Rij to item j
Hao Wang Relational SDAE 22 / 36
1 Can SDAE learn effective representation for recommendation? 2 How to incorporate relational information into SDAE? 3 How is the performance? Hao Wang Relational SDAE 23 / 36
1
Background and Related Work
2
Generalized Probabilistic SDAE
3
Relational SDAE
4
Performance Evaluation
5
Case study
6
Conclusion
Hao Wang Relational SDAE 24 / 36
Description of datasets
citeulike-a citeulike-t movielens-plot #items 16980 25975 7261 #tags 7386 8311 2988 #tag-item paris 204987 134860 51301 #relations 44709 32665 543621
Hao Wang Relational SDAE 24 / 36
50 100 150 200 250 300 0.15 0.2 0.25 0.3 0.35
M Recall RSDAE SDAE CTR−SR CTR
Hao Wang Relational SDAE 25 / 36
50 100 150 200 250 300 0.3 0.35 0.4 0.45 0.5 0.55 0.6
M Recall RSDAE SDAE CTR−SR CTR
Hao Wang Relational SDAE 26 / 36
50 100 150 200 250 300 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 0.22 0.24
M Recall RSDAE SDAE CTR−SR CTR
Hao Wang Relational SDAE 27 / 36
50 100 150 200 250 300 0.2 0.25 0.3 0.35 0.4 0.45
M Recall RSDAE SDAE CTR−SR CTR
Hao Wang Relational SDAE 28 / 36
1
Background and Related Work
2
Generalized Probabilistic SDAE
3
Relational SDAE
4
Performance Evaluation
5
Case study
6
Conclusion
Hao Wang Relational SDAE 29 / 36
An example article with recommended tags
Example Article Title: Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of Product Reviews Top topic 1: language, text, mining, representation, semantic, concepts, words, relations, processing, categories Top 10 tags SDAE True? RSDAE True?
no
no
yes
no
no
yes
no
no
no
yes
no
yes
no
yes
no
no
no
yes
no
yes
Hao Wang Relational SDAE 29 / 36
An example movie with recommended tags
Example Movie Title: E.T. the Extra-Terrestrial Top topic 1: crew, must, on, earth, human, save, ship, rescue, by, find, scientist, planet Top 10 recommended tags SDAE True tag?
yes
no
no
yes
no
no
no
no
yes
no
Hao Wang Relational SDAE 30 / 36
An example movie with recommended tags
Example Movie Title: E.T. the Extra-Terrestrial Top topic 1: crew, must, on, earth, human, save, ship, rescue, by, find, scientist, planet Top 10 recommended tags RSDAE True tag?
yes
yes
yes
no
no
no
yes
yes
no
yes
Hao Wang Relational SDAE 31 / 36
1
Background and Related Work
2
Generalized Probabilistic SDAE
3
Relational SDAE
4
Performance Evaluation
5
Case study
6
Conclusion
Hao Wang Relational SDAE 32 / 36
Contribution:
1 Adapt SDAE for tag recommendation 2 A probabilistic relational model for relational deep learning 3 State-of-the-art performance
Take-home Message:
1 Deep models significantly boost recommendation accuracy 2 Probabilistic formulation facilitates relational deep learning 3 Incorporating relational information further boosts accuracy Hao Wang Relational SDAE 32 / 36
1 Applications other than tag recommendation 2 Adaption for other deep learning models 3 Integrated model instead of separate ones 4 Fully Bayesian methods Hao Wang Relational SDAE 33 / 36
1 Collaborative CNN for recommender systems 2 Relational deep learning for link prediction Hao Wang Relational SDAE 34 / 36
50 100 150 200 250 300 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 0.22 M Recall
CCNN CTR CMF SVDFeature
Hao Wang Relational SDAE 35 / 36
Hao Wang Relational SDAE 36 / 36