Time-Aware Novelty Metrics for Recommender Systems Pablo S anchez - PowerPoint PPT Presentation

Time-Aware Novelty Metrics for Recommender Systems Pablo S´ anchez Alejandro Bellog´ ın Universidad Aut´ onoma de Madrid Escuela Polit´ ecnica Superior Departamento de Ingenier´ ıa Inform´ atica European Conference on Information Retrieval, 2018 1 / 83

Outline Recommender Systems 1 Time-Aware Novelty Metrics for Recommender Systems 2 Experiments 3 Conclusions and future work 4 2 / 83

Outline Recommender Systems 1 Time-Aware Novelty Metrics for Recommender Systems 2 Experiments 3 Conclusions and future work 4 3 / 83

Recommender Systems ... ... ... ... Suggest new items to users based on their tastes and needs 4 / 83

Recommender Systems ... ... ... ... Suggest new items to users based on their tastes and needs Measure the quality of recommendations. How? 5 / 83

Recommender Systems ... ... ... ... Suggest new items to users based on their tastes and needs Measure the quality of recommendations. How? Several evaluation dimensions: Error, Ranking, Novelty / Diversity 6 / 83

Recommender Systems ... ... ... ... Suggest new items to users based on their tastes and needs Measure the quality of recommendations. How? Several evaluation dimensions: Error, Ranking, Novelty / Diversity We will focus on the temporal dimension 7 / 83

Different notions of quality 8 / 83

Different notions of quality Best in Relevance? 9 / 83

Different notions of quality Best in Relevance? R 2 > R 1 > R 3 10 / 83

Different notions of quality Best in Relevance? R 2 > R 1 > R 3 Best in Novelty? 11 / 83

Different notions of quality Best in Relevance? R 2 > R 1 > R 3 Best in Novelty? R 1 > R 3 > R 2 12 / 83

Different notions of quality Best in Relevance? R 2 > R 1 > R 3 Best in Novelty? R 1 > R 3 > R 2 Best in Freshness ? 13 / 83

Different notions of quality Best in Relevance? R 2 > R 1 > R 3 Best in Novelty? R 1 > R 3 > R 2 Best in Freshness ? R 3 > R 1 > R 2 14 / 83

Types of data splitting time time items items Random split Temporal split 15 / 83

Types of data splitting time time items items Random split Temporal split Random splitting has been the most extended way to test recommender systems 16 / 83

Types of data splitting time time items items Random split Temporal split Random splitting has been the most extended way to test recommender systems Temporal splitting is becoming more important 17 / 83

Types of data splitting time time items items Random split Temporal split Random splitting has been the most extended way to test recommender systems Temporal splitting is becoming more important Hence, time should also be incorporated in evaluation metrics 18 / 83

Outline Recommender Systems 1 Time-Aware Novelty Metrics for Recommender Systems 2 Experiments 3 Conclusions and future work 4 19 / 83

Preliminaries Framework proposed in Vargas and Castells (2011) � m ( R u | θ ) = C disc( n ) p ( rel | i n , u )nov( i n | θ ) (1) i n ∈ R u 20 / 83

Preliminaries Framework proposed in Vargas and Castells (2011) � m ( R u | θ ) = C disc( n ) p ( rel | i n , u )nov( i n | θ ) (1) i n ∈ R u Where: R u items recommended to user u θ contextual variable (e.g., the user profile) disc( n ) is a discount model (e.g. NDCG) p ( rel | i n , u ) relevance component nov( i n | θ ) novelty model 21 / 83

Preliminaries Framework proposed in Vargas and Castells (2011) � m ( R u | θ ) = C disc( n ) p ( rel | i n , u )nov( i n | θ ) (1) i n ∈ R u When using nov( i n | θ ) = (1 − p (seen | i )) we obtain the expected popularity complement (EPC) metric 22 / 83

Preliminaries Framework proposed in Vargas and Castells (2011) � m ( R u | θ ) = C disc( n ) p ( rel | i n , u )nov( i n | θ ) (1) i n ∈ R u When using nov( i n | θ ) = (1 − p (seen | i )) we obtain the expected popularity complement (EPC) metric However, all the metrics derived from this framework are time-agnostic 23 / 83

Preliminaries Framework proposed in Vargas and Castells (2011) � m ( R u | θ t ) = C disc( n ) p ( rel | i n , u ) nov( i n | θ t ) (1) i n ∈ R u When using nov( i n | θ ) = (1 − p (seen | i )) we obtain the expected popularity complement (EPC) metric However, all the metrics derived from this framework are time-agnostic We propose to replace the novelty component defining new time-aware novelty models 24 / 83

Time-Aware Novelty Metrics Classic metrics do not provide any information about the evolution of the items: we can recommend relevant but well-known (old) items 25 / 83

Time-Aware Novelty Metrics Classic metrics do not provide any information about the evolution of the items: we can recommend relevant but well-known (old) items Every item in the system can be modeled with a temporal representation: θ t = { θ t ( i ) } = { ( i , � t 1 ( i ) , · · · , t n ( i ) � ) } (2) 26 / 83

Time-Aware Novelty Metrics Classic metrics do not provide any information about the evolution of the items: we can recommend relevant but well-known (old) items Every item in the system can be modeled with a temporal representation: θ t = { θ t ( i ) } = { ( i , � t 1 ( i ) , · · · , t n ( i ) � ) } (2) Two different sources for the timestamps: 27 / 83

Time-Aware Novelty Metrics Classic metrics do not provide any information about the evolution of the items: we can recommend relevant but well-known (old) items Every item in the system can be modeled with a temporal representation: θ t = { θ t ( i ) } = { ( i , � t 1 ( i ) , · · · , t n ( i ) � ) } (2) Two different sources for the timestamps: Metadata information: release date (movies or songs), creation time, etc. 28 / 83

Time-Aware Novelty Metrics Classic metrics do not provide any information about the evolution of the items: we can recommend relevant but well-known (old) items Every item in the system can be modeled with a temporal representation: θ t = { θ t ( i ) } = { ( i , � t 1 ( i ) , · · · , t n ( i ) � ) } (2) Two different sources for the timestamps: Metadata information: release date (movies or songs), creation time, etc. Rating history of the items 29 / 83

Time-Aware Novelty Metrics ... ... 30 / 83

Modeling time profiles for items How can we aggregate the temporal representation? 31 / 83

Modeling time profiles for items How can we aggregate the temporal representation? We explored four possibilities: 32 / 83

Modeling time profiles for items How can we aggregate the temporal representation? We explored four possibilities: Take the first interaction (FIN) 33 / 83

Modeling time profiles for items How can we aggregate the temporal representation? We explored four possibilities: Take the first interaction (FIN) Take the last interaction (LIN) 34 / 83

Modeling time profiles for items How can we aggregate the temporal representation? We explored four possibilities: Take the first interaction (FIN) Take the last interaction (LIN) Take the average of the ratings times (AIN) 35 / 83

Modeling time profiles for items How can we aggregate the temporal representation? We explored four possibilities: Take the first interaction (FIN) Take the last interaction (LIN) Take the average of the ratings times (AIN) Take the median of the ratings times (MIN) 36 / 83

Modeling time profiles for items How can we aggregate the temporal representation? We explored four possibilities: Take the first interaction (FIN) Take the last interaction (LIN) Take the average of the ratings times (AIN) Take the median of the ratings times (MIN) Each case defines a function f ( θ t ( i )) 37 / 83

Modeling time profiles for items: an example ... ... 38 / 83

Modeling time profiles for items: an example Which model represents better the freshness of the items? FIN? ... ... 39 / 83

Modeling time profiles for items: an example Which model represents better the freshness of the items? FIN? i 2 > i 10 > i 9 > i 1 ... ... 40 / 83

Modeling time profiles for items: an example Which model represents better the freshness of the items? FIN? i 2 > i 10 > i 9 > i 1 LIN? ... ... 41 / 83

Modeling time profiles for items: an example Which model represents better the freshness of the items? FIN? i 2 > i 10 > i 9 > i 1 LIN? i 9 > i 1 > i 10 > i 2 ... ... 42 / 83

Modeling time profiles for items: an example Which model represents better the freshness of the items? FIN? i 2 > i 10 > i 9 > i 1 LIN? i 9 > i 1 > i 10 > i 2 ... MIN? ... 43 / 83

Modeling time profiles for items: an example Which model represents better the freshness of the items? FIN? i 2 > i 10 > i 9 > i 1 LIN? i 9 > i 1 > i 10 > i 2 ... MIN? i 10 > i 2 > i 9 > i 1 ... 44 / 83

Modeling time profiles for items: an example Which model represents better the freshness of the items? FIN? i 2 > i 10 > i 9 > i 1 LIN? i 9 > i 1 > i 10 > i 2 ... MIN? i 10 > i 2 > i 9 > i 1 AIN? ... 45 / 83