Recommender Systems Alejandro Bellogn Supervised by Pablo Castells - PowerPoint PPT Presentation

Predicting Performance in Recommender Systems Alejandro Bellogín Supervised by Pablo Castells and Iván Cantador Escuela Politécnica Superior Universidad Autónoma de Madrid @abellogin alejandro.bellogin@uam.es IRG ACM Conference on Recommender Systems 2011 – Doctoral Symposium October 23, Chicago, USA IR Group @ UAM

Motivation Is it possible to predict the accuracy of a recommendation? 2 IRG ACM Conference on Recommender Systems 2011 – Doctoral Symposium October 23, Chicago, USA IR Group @ UAM

Hypothesis Data that are commonly available to a Recommender System could contain signals that enable an a priori estimation of the success of the recommendation 3 IRG ACM Conference on Recommender Systems 2011 – Doctoral Symposium October 23, Chicago, USA IR Group @ UAM

Research Questions Is it possible to define a performance prediction theory for 1. recommender systems in a sound, formal way? Is it possible to adapt query performance techniques (from 2. IR) to the recommendation task? What kind of evaluation should be performed? Is IR 3. evaluation still valid in our problem? What kind of recommendation problems can these models 4. be applied to? 4 IRG ACM Conference on Recommender Systems 2011 – Doctoral Symposium October 23, Chicago, USA IR Group @ UAM

Predicting Performance in Recommender Systems RQ1 . Is it possible to define a performance prediction theory for recommender systems in a sound, formal way? a) Define a predictor of performance  =  (u, i, r, … ) b) Agree on a performance metric  =  (u, i, r, … ) c) Check predictive power by measuring correlation corr([  (x 1 ), … ,  (x n )], [  (x 1 ), … ,  (x n )]) d) Evaluate final performance: dynamic vs static 5 IRG ACM Conference on Recommender Systems 2011 – Doctoral Symposium October 23, Chicago, USA IR Group @ UAM

Predicting Performance in Recommender Systems RQ2 . Is it possible to adapt query performance techniques (from IR) to the recommendation task? In IR: “Estimation of the system’s performance in response  to a specific query” Several predictors proposed  We focus on query clarity  user clarity  6 IRG ACM Conference on Recommender Systems 2011 – Doctoral Symposium October 23, Chicago, USA IR Group @ UAM

User clarity  It captures uncertainty in user’s data • Distance between the user’s and the system’s probability model     user’s model | p x u         clarity | lo g u p x u     p x    system’s model x X c • X may be: users, items, ratings, or a combination 7 IRG ACM Conference on Recommender Systems 2011 – Doctoral Symposium October 23, Chicago, USA IR Group @ UAM

User clarity  Three user clarity formulations: Background Name Vocabulary User model model     Rating-based Ratings | p r u p r c     p i Item-based Items | p i u c     | p r i Item-and-rating-based Items rated by the user | , p r i u m l     | user model p x u         clarity | lo g u p x u     p x    x X background model c 8 IRG ACM Conference on Recommender Systems 2011 – Doctoral Symposium October 23, Chicago, USA IR Group @ UAM

User clarity  Seven user clarity models implemented: Name Formulation User model Background model       RatUser Rating-based | , ; | p r i u p i u p r U U R c       | , ; | p r i u p i u RatItem Rating-based p r I U R c     p i ItemSimple Item-based | p i u c R     p i | p i u ItemUser Item-based c U R     IRUser Item-and-rating-based | p r i | , p r i u m l U     | | , p r i IRItem Item-and-rating-based p r i u m l I     | p r i | , p r i u IRUserItem Item-and-rating-based m l U I 9 IRG ACM Conference on Recommender Systems 2011 – Doctoral Symposium October 23, Chicago, USA IR Group @ UAM

User clarity  Predictor that captures uncertainty in user’s data  Different formulations capture different nuances  More dimensions in RS than in IR: user, items, ratings, features, … RatItem 0.7 p_c(x) p(x|u1) 0.6 p(x|u2) 0.5 0.4 0.3 0.2 0.1 0.0 4 3 5 2 1 Ratings 4 3 5 2 1 10 IRG ACM Conference on Recommender Systems 2011 – Doctoral Symposium October 23, Chicago, USA IR Group @ UAM

Predicting Performance in Recommender Systems RQ3 . What kind of evaluation should be performed? Is IR evaluation still valid in our problem?  In IR: Mean Average Precision + correlation  50 points (queries) vs 1000+ points (users) Performance metric is not clear: error-based, precision-based?  What is performance?  r ~ 0.57 It may depend on the final application  Possible bias   E.g., towards users or items with larger profiles 11 IRG ACM Conference on Recommender Systems 2011 – Doctoral Symposium October 23, Chicago, USA IR Group @ UAM

Predicting Performance in Recommender Systems RQ3 . What kind of evaluation should be performed? Is IR evaluation still valid in our problem?  In IR: Mean Average Precision + correlation  50 points (queries) vs 1000+ points (users) Performance metric is not clear: error-based, precision-based?  What is performance?  It may depend on the final application  Possible bias   E.g., towards users or items with larger profiles 12 IRG ACM Conference on Recommender Systems 2011 – Doctoral Symposium October 23, Chicago, USA IR Group @ UAM

Predicting Performance in Recommender Systems RQ4 . What kind of recommendation problems can these models be applied to? Whenever a combination of strategies is available  Example 1: dynamic neighbor weighting  Example 2: dynamic ensemble recommendation  13 IRG ACM Conference on Recommender Systems 2011 – Doctoral Symposium October 23, Chicago, USA IR Group @ UAM

Dynamic neighbor weighting  The user’s neighbors are weighted according to their similarity  Can we take into account the uncertainty in neighbor’s data?  User neighbor weighting [1]        • Static:   , sim , , g u i C u v ra t v i  [ ] v N u             γ sim , , • Dynamic: , g u i C v u v ra t v i  v N [ u ] 14 IRG ACM Conference on Recommender Systems 2011 – Doctoral Symposium October 23, Chicago, USA IR Group @ UAM

Dynamic hybrid recommendation  Weight is the same for every item and user (learnt from training)  What about boosting those users predicted to perform better for some recommender?  Hybrid recommendation [3]                • Static: , , 1 , g u i g u i g u i R 1 R 2                  γ , 1 γ , • Dynamic: , g u i u g u i u g u i R 1 R 2 15 IRG ACM Conference on Recommender Systems 2011 – Doctoral Symposium October 23, Chicago, USA IR Group @ UAM

Results – Neighbor weighting  Correlation analysis [1] • With respect to Neighbor Goodness metric : “ how good a neighbor is to her vicinity ”  Performance [1] (MAE = Mean Average Error, the lower the better) 0,98 Standard CF 0,88 Standard CF 0,96 Clarity-enhanced CF 0,87 Clarity-enhanced CF 0,94 0,86 0,92 0,85 0,90 MAE MAE 0,84 0,88 0,83 0,86 0,82 0,84 0,81 0,82 0,80 0,80 10 20 30 40 50 60 70 80 90 100 150 200 250 300 350 400 450 500 16 % of ratings for training Neighbourhood size IRG ACM Conference on Recommender Systems 2011 – Doctoral Symposium October 23, Chicago, USA IR Group @ UAM

Results – Neighbour weighting  Correlation analysis [1] • With respect to Neighbour Goodness metric : “ how good a neighbour is to her vicinity ” Positive, although not very strong correlations  Performance [1] (MAE = Mean Average Error, the lower the better) 0,98 Standard CF 0,88 Standard CF 0,96 Clarity-enhanced CF 0,87 Clarity-enhanced CF 0,94 0,86 0,92 Improvement of over 5% wrt. the baseline 0,85 0,90 MAE MAE 0,84 Plus, it does not degrade performance 0,88 0,83 0,86 0,82 0,84 0,81 0,82 0,80 0,80 10 20 30 40 50 60 70 80 90 100 150 200 250 300 350 400 450 500 17 % of ratings for training Neighbourhood size IRG ACM Conference on Recommender Systems 2011 – Doctoral Symposium October 23, Chicago, USA IR Group @ UAM

Results – Hybrid recommendation  Correlation analysis [2] • With respect to nDCG@50 (nDCG, normalized Discount Cumulative Gain)  Performance [3] nDCG@50 0,2 0,15 0,1 0,05 0 H1 H2 H3 H4 18 Adaptive Static IRG ACM Conference on Recommender Systems 2011 – Doctoral Symposium October 23, Chicago, USA IR Group @ UAM

Results – Hybrid recommendation  Correlation analysis [2] • With respect to nDCG@50 (nDCG, normalized Discount Cumulative Gain) In average, most of the predictors obtain positive, strong correlations  Performance [3] nDCG@50 0,2 0,15 Dynamic strategy outperforms static for 0,1 different combination of recommenders 0,05 0 H1 H2 H3 H4 19 Adaptive Static IRG ACM Conference on Recommender Systems 2011 – Doctoral Symposium October 23, Chicago, USA IR Group @ UAM