Unbiased Offline Recommender Evaluation for Missing-Not-At-Random - - PowerPoint PPT Presentation

Unbiased Offline Recommender Evaluation for Missing-Not-At-Random Implicit Feedback

1

Funders: Lo Longqi Yang Yin Cui Yuan Xuan Chenyang Wang Serge Belongie Deborah Estrin

2

Offline Evaluation of Recommendation Algorithm

( , ) ( , ) ( , ) …

…

user-item interactions

R

recommendation algorithms rewards

…

…

R

recommendation algorithms rewards

…

3

Offline Evaluation of Recommendation Algorithm

( , ) ( , ) ( , ) …

user interaction history

Pr Pros:

• Cost effective.
• Efficient.
• Iterate faster.
• Experiment before deployment.

…

R

recommendation algorithms rewards

…

4

Offline Evaluation of Recommendation Algorithm

( , ) ( , ) ( , ) …

user interaction history

Pr Pros:

• Cost effective.
• Efficient.
• Iterate faster.
• Experiment before deployment.

Co Cons:

• The data is Missing-Not-At-Random (MNAR)

5

Of Offline E e Evaluation

• n procedure

user ! item " user ! interacted with item "

6

Of Offline E e Evaluation

• n procedure

train/test

7

Of Offline E e Evaluation

• n procedure
• 1. Train and validate a

recommendation model

• 2. Averaged performance over held-
• ut (user, item) interaction pairs

(Average-Over-All)

8

Of Offline E e Evaluation

• n procedure
• 1. Train and validate a

recommendation model (policy) !

• 2. Averaged performance over held-
• ut (user, item) interaction pairs

(Average-Over-All)

Rating-based recommendation systems Implicit feedback-based recommendation systems

9

Previous work: Av Average-Ov Over er-Al All is is bia biased fo for r ra rating ting-ba based d re recommenda ndatio tion n systems, be becaus use ra rating tings are re MN MNAR

[Marlin et al. 09], [Schnabel et al. 16], [Steck 10], [Steck 11], and [Steck 13]

10

Previous work: Av Average-Ov Over er-Al All is is bia biased fo for r ra rating ting-ba based d re recommenda ndatio tion n systems, be becaus use ra rating tings are re MN MNAR

[Marlin et al. 09], [Schnabel et al. 16], [Steck 10], [Steck 11], and [Steck 13]

Previous work: Av Average-Ov Over er-Al All is is unb unbiased fo for r im implic plicit it fe feedba dback-ba based d re recommenda datio ion systems, be because im implic plicit it fe feedba dback is is mi missing uniforml mly at random.

[Lim 15]

11

This work: Av Average-Ov Over er-Al All is is bia biased fo for r im implic plicit it fe feedba dback-ba based d re recommenda datio ion systems, be because im impl plic icit it fe feedbac dback k is is NO NOT mi missing uniforml mly at random.

12

This work: Av Average-Ov Over er-Al All is is bia biased fo for r im implic plicit it fe feedba dback-ba based d re recommenda datio ion systems, be because im impl plic icit it fe feedbac dback k is is NO NOT mi missing uniforml mly at random.

Popularity bias (Users are more likely to be exposed to popular items)

tr trending re recommendation

13

A Hypothetical Example

Popular Items Long-tail Items # of liked items (over all items) # of liked items (over observations)

1 10 10 1 : :

Algorithm 1 Performance Algorithm 2 Performance

0.8 0.75 0.75

14

A Hypothetical Example

Popular Items Long-tail Items # of liked items (over all items) # of liked items (over observations)

1 10 10 1 : :

Algorithm 1 Performance Algorithm 2 Performance

0.8 0.75 0.75

15

A Hypothetical Example

Popular Items Long-tail Items # of liked items (over all items) # of liked items (over observations)

1 10 10 1 : :

Algorithm 1 Performance Algorithm 2 Performance

0.8 0.75 0.75

16

A Hypothetical Example

Popular Items Long-tail Items # of liked items (over all items) # of liked items (over observations)

1 10 10 1 : :

Algorithm 1 Performance Algorithm 2 Performance

0.8 0.75 0.75

17

A Hypothetical Example

Popular Items Long-tail Items # of liked items (over all items) # of liked items (over observations)

1 10 10 1 : :

Algorithm 1 Performance Algorithm 2 Performance

0.8 0.75 0.75

18

A Hypothetical Example

Popular Items Long-tail Items # of liked items (over all items) # of liked items (over observations)

1 10 10 1 : :

Algorithm 1 Performance Algorithm 2 Performance

0.8 0.75 0.75

Any sensible evaluation

19

A Hypothetical Example

Popular Items Long-tail Items # of liked items (over all items) # of liked items (over observations)

1 10 10 1 : :

Algorithm 1 Performance Algorithm 2 Performance

0.8 0.75 0.75

Average- Over-All

20

Formalize Reward !

R( ˆ Z) = 1 |U| X

u∈U

1 |Su| X

i∈Su

c( ˆ Zu,i)

Ideal evaluation:

Item rankings predicted by an algorithm

21

Formalize Reward !

R( ˆ Z) = 1 |U| X

u∈U

1 |Su| X

i∈Su

c( ˆ Zu,i)

Ideal evaluation:

Predicted ranking of item i for user u Items liked by user u among the entire item set Reward for (u, i) pair scoring metric

Item rankings predicted by an algorithm

22

Formalize Reward !

R( ˆ Z) = 1 |U| X

u∈U

1 |Su| X

i∈Su

c( ˆ Zu,i)

Ideal evaluation:

Predicted ranking of item i for user u Items liked by user u among the entire item set Reward for user u scoring metric

Item rankings predicted by an algorithm

23

Formalize Reward !

R( ˆ Z) = 1 |U| X

u∈U

1 |Su| X

i∈Su

c( ˆ Zu,i)

Ideal evaluation:

Predicted ranking of item i for user u Items liked by user u among the entire item set scoring metric

Item rankings predicted by an algorithm

Reward for the algorithm

24

Formalize Reward !

Average-Over-All: ˆ

RAOA( ˆ Z) = 1 |U| X

u∈U

1 |S∗

u|

X

i∈S∗

u

c( ˆ Zu,i)

Items liked by user u (observed)

25

Formalize Bias

Ou,i = 1 if (u, i) is observed, and Ou,i = 0 otherwise EO h ˆ RAOA( ˆ Z) i 6= R( ˆ Z) Ou,i ∼ B(1, Pu,i)

26

ˆ RIPS( ˆ Z|P) = 1 |U| X

u∈U

1 |Su| X

i∈S∗

u

c( ˆ Zu,i) Pu,i ˆ RAOA( ˆ Z) = 1 |U| X

u∈U

1 |S∗

u|

X

i∈S∗

u

c( ˆ Zu,i)

Inverse-Propensity-Scoring (IPS)

27

ˆ RIPS( ˆ Z|P) = 1 |U| X

u∈U

1 |Su| X

i∈S∗

u

c( ˆ Zu,i) Pu,i ˆ RAOA( ˆ Z) = 1 |U| X

u∈U

1 |S∗

u|

X

i∈S∗

u

c( ˆ Zu,i) EO h ˆ RIPS( ˆ Z|P) i = R( ˆ Z)

Inverse-Propensity-Scoring (IPS)

28

ˆ RIPS( ˆ Z|P) = 1 |U| X

u∈U

1 |Su| X

i∈S∗

u

c( ˆ Zu,i) Pu,i

Self-Normalized Inverse-Propensity-Scoring (SNIPS)

[Swaminathan et al.15]

ˆ RSNIPS( ˆ Z|P) = 1 |U| X

u∈U

1 P

i∈S∗

u

1 Pu,i

X

i∈S∗

u

c( ˆ Zu,i) Pu,i

29

Estimating Propensity Scores

Fa Factor: Popularity bias (Users are more likely to be exposed to popular items) As Assumptions:

• User-independence assumption
• Two-steps assumption
• User preference is not affected by item presentation

Pu,i = P(Ou,i = 1) = P(O∗,i = 1) = P∗,i P∗,i = P select

∗,i

· P interact|select

∗,i

P interact|select

∗,i

= P interact

∗,i

30

Estimating Propensity Scores

Po Popularity bias model [St Steck 11] 11]:

ˆ P select

∗,i

∝ (n∗

i )γ

Observed item popularity

31

Estimating Propensity Scores

Po Popularity bias model [St Steck 11] 11]:

ˆ P select

∗,i

∝ (n∗

i )γ

ˆ P∗,i ∝ (n∗

i )( γ+1

2 )

Estimated from known online content serving policy

32

Measuring bias in recommender evaluation (Yahoo! music rating dataset)

Model Average- Over-All !SNIPS (# = 1.5) !SNIPS (# = 2.0) !SNIPS (# = 2.5) !SNIPS (# = 3.0) U-CML 0.401 0.270 0.260 0.253 0.248 A-CML 0.399 0.274 0.264 0.258 0.253 BPR 0.380 0.275 0.268 0.262 0.258 PMF 0.386 0.267 0.259 0.252 0.248 Mean Absol

• lute Error
• r (MAE), Reca

call !SNIPS produces significantly lower MAE

33

Measuring bias in recommender evaluation (Yahoo! music rating dataset)

Model Average- Over-All !SNIPS (# = 1.5) !SNIPS (# = 2.0) !SNIPS (# = 2.5) !SNIPS (# = 3.0) U-CML 0.401 0.270 0.260 0.253 0.248 A-CML 0.399 0.274 0.264 0.258 0.253 BPR 0.380 0.275 0.268 0.262 0.258 PMF 0.386 0.267 0.259 0.252 0.248 Mean Absol

• lute Error
• r (MAE), Reca

call !SNIPS produces significantly lower MAE

The accuracy of recommending popular items is a significant ov

• verestimation of

the true recommendation performance

34

Please come to our poster or refer to our paper for:

• Proofs
• Experimental details.
• More experiments.
• Deeper analysis of the unbiased evaluator.

35

Conclusions and Future Work

ˆ RSNIPS( ˆ Z|P) = 1 |U| X

u∈U

1 P

i∈S∗

u

1 Pu,i

X

i∈S∗

u

c( ˆ Zu,i) Pu,i EO h ˆ RIPS( ˆ Z|P) i = R( ˆ Z)

• Understanding variance of evaluators.
• Propensity estimation (e.g., incorporate auxiliary

user and item information).

• Debias training of recommendation systems (e.g.,

[Liang et al. 16]).

36

http://www.openrec.ai

Github link, documents, and tutorials

Lo Longqi Yang

Ph.D. candidate Computer Science, Cornell Tech, Cornell University Email: ylongqi@cs.cornell.edu Web: bit.ly/longqi Twitter: @ylongqi Connected Experiences Lab http://cx.jacobs.cornell.edu/ Small Data Lab http://smalldata.io/

Funders: