Understanding a and R Recommending Po Podcast Content Longqi - - PowerPoint PPT Presentation

Understanding a and R Recommending Po Podcast Content

1

Funders:

Longqi Yang

Computer Science Ph.D. Candidate ylongqi@cs.cornell.edu Twitter: @ylongqi

2

Col Collabor aborator ators

3

Wh Why Podc dcast ast

Eme Emerging ng Int nterfa faces for Podcast Co Conte tent t Co Consump mpti tion

2

5

6

Wh What’ at’s s spec special al abo about t po podc dcasts asts (c (conten tent) t)

… the architecture of the podcast is the precise antidote for the flaws of the present. It is de deep where now is shallow. It is insulated fr from ads where now is completely vulnerable. It is a chance for th thinking and refl flection; it has an attention span an

• rder of magnitude greater than the Tweet. It is an opportunity

for serious (and playful) engagement. It is healthy eating for a brain-scape that now gorges on fast food. …

• -- Lawrence Lessig (Professor of Law at Harvard Law School)

7

… It turns out, certain things humans can only do well if they do it sl

• slowly. Eating, cooking, reflecting, thinking, loving: These

are the things we need to pace and pause… We should all spread the idea that every healthy mind spends time every week in slow thinking …

• -- Lawrence Lessig (Professor of Law at Harvard Law School)

Wh What’ at’s s spec special al abo about t po podc dcasts asts (c (conten tent) t)

8

Past Fu Future

(W (What at you as aspire re to to liste ten in in the future, user in intentio ions and aspir iratio ions) (What you listened before)

Wh What’ at’s s spec special al abo about t po podc dcasts asts (u (user) ser)

9

People listened to episodes from subscribed channels (subscription-based consumption)

Wh What’ at’s s spec special al abo about t po podc dcasts asts (u (user) ser)

10

Compu Computati tation

• nal

al Su Suppor pport f t for P

• r Podcasts
• dcasts

Aa Aa

articles posts … music Past rec. search

11

Compu Computati tation

• nal

al Su Suppor pport f t for P

• r Podcasts
• dcasts

Aa Aa

articles posts … music Past rec. search Podcast

12

Compu Computati tation

• nal

al Su Suppor pport f t for P

• r Podcasts
• dcasts

Aa Aa

articles posts … music Past rec. search Podcast Podcast

13

Age Agend nda

More than Just Words (WSDM’19) Intention Informed Recommendations (Under Review) Debias Offline Recommendation Evaluation (Recsys’18)

14

Con Conten tent == t == Words

• rds

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

15

Po Podcast Content == Words

(i (iTunes Podcas ast t dire recto tory)

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

16

Po Podcast Content > Words

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

Conversational Paralinguistic Musical

17

Po Podcast Content > Words

https://podcastfasttrack.com/podcast-editing/

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

Conversational Paralinguistic Musical

18

Our Goal: Our Goal: Mod Modeling eling Non Non-textu textual al Ch Char aracter acteristi stics of cs of P Podcasts

• dcasts

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

feature representation

19

A Naïv A Naïve Solution Solution

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

MFCC IS09 IS13 …

20

A Naïv A Naïve Solution Solution

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

MFCC IS09 IS13 …

Expected to be sub-optimal

21

Our ap Our approach: roach: Unsup Unsuper ervised vised Rep Representation Lear resentation Learning ning

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

large unlabeled podcast corpus

22

Our ap Our approach: roach: Unsup Unsuper ervised vised Rep Representation Lear resentation Learning ning

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

23

Our ap Our approach: roach: Unsup Unsuper ervised vised Rep Representation Lear resentation Learning ning

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

Fine-grained variations

24

Ad Adversar arial Le ial Lear arning ning-based based Podc dcast ast Represen epresentati tation (A (ALPR PR)

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

Generator

vectors sampled from a uniform distribution

Discriminator CE

features (ALPR)

25

Ad Adversar arial Le ial Lear arning ning-based based Podc dcast ast Represen epresentati tation (A (ALPR PR)

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

Generator Discriminator CE

Label=1 (real)

vectors sampled from a uniform distribution features (ALPR)

Train the generator

26

Ad Adversar arial Le ial Lear arning ning-based based Podc dcast ast Represen epresentati tation (A (ALPR PR)

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

Train the discriminator and the classifier

CE CE

Label=1 (real) Label=0 (generated)

Generator Discriminator

spectrograms of real podcast audio

27

Ad Adversar arial Le ial Lear arning ning-based based Podc dcast ast Represen epresentati tation (A (ALPR PR)

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

The generator

8 32 64 32 128 64 256 512 256 128 64

512 128 1

16 deconv, 5x5 stride 2 fully connected deconv, 5x5 stride 2 deconv, 5x5 stride 2 deconv, 5x5 stride 2 z x

28

Ad Adversar arial Le ial Lear arning ning-based based Podc dcast ast Represen epresentati tation (A (ALPR PR)

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

The discriminator

128 512 256 64 128 32 64 16 32 8 64 128 256 512 conv, 5x5 stride 2 x D(x) fully connected global average pooling 1 conv, 5x5 stride 2 conv, 5x5 stride 2 conv, 5x5 stride 2

29

Ad Adversar arial Le ial Lear arning ning-based based Podc dcast ast Represen epresentati tation (A (ALPR PR)

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

Corpus:

88, 88,728 728 episodes (18, 18,433 433 channels)

Training:

42, 42,370 370 episodes

Evaluation:

46, 46,358 358 episodes

30

Ev Evalua luations ions

Attr Attrib ibute utes Clas Classif ification ication (binar inary) y) Calm vs. Energetic Humorous vs. Serious Po Popularity prediction (binary) Top channels on iTunes vs. Others

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

31

Attr Attrib ibute utes Clas Classif ification ication

How calm or energetic is the audio presentation? calm energetic How humorous or serious is the audio presentation? humorous serious Does the above audio presentation contain men’s or women’s voices? Men’s Women’s Both Other

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

32

Attr Attrib ibute utes Clas Classif ification ication

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

250 500

serious (10) humorous (0) calm (0) energetic (10)

negative positive positive negative

Count

33

Basel Baselines es

Discriminator D Generator G

D x z

sample

σ µ

AE VAE

MFCC IS09 IS13 Autoencoder (AE) and Variational Autoencoder (VAE):

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

34

Attr Attrib ibute utes Clas Classif ification P ication Perfor

• rmance

mance

Se Serious iousne ness Ene Energy

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

35

Wh Why do does es ALPR PR outperf tperform AE an and d VAE?

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

AE VAE Adv. Real

36

Po Popularity Prediction Pe Performance

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

37

Po Popular vs. unpopular channels (Energy score)

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM), 2019.

38

Age Agend nda

More than Just Words (WSDM’19) Intention Informed Recommendations (Under Review) Debias Offline Recommendation Evaluation (Recsys’18)

39

Of Offline e Evaluation

• n of
• f Recom

Recommen endation

• n Al

Algor

• rithm

( , ) ( , ) ( , ) …

…

user-item interactions

R

recommendation algorithms rewards

…

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

40

Of Offline e Evaluation

• n of
• f Recom

Recommen endation

• n Al

Algor

• rithm

( , ) ( , ) ( , ) …

…

user-item interactions

R

recommendation algorithms rewards

…

Pros:

• Cost effective.
• Efficient.
• Iterate faster.
• Experiment before deployment.

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

41

Of Offline e Evaluation

• n of
• f Recom

Recommen endation

• n Al

Algor

• rithm

( , ) ( , ) ( , ) …

…

user-item interactions

R

recommendation algorithms rewards

…

Pros:

• Cost effective.
• Efficient.
• Iterate faster.
• Experiment before deployment.

Cons:

• The data is Missing-Not-At-Random (MNAR)

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

42

Of Offline e Evaluation

• n proced
• cedure

user i item j user i interacted with item j

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

43

Of Offline e Evaluation

• n proced
• cedure

train/test

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

44

Of Offline e Evaluation

• n proced
• cedure
• 1. Train and validate a

recommendation model

• 2. Averaged performance over held-out

(user, item) interaction pairs (Average-Over-All)

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

45

Of Offline e Evaluation

• n proced
• cedure
• 1. Train and validate a

recommendation model

• 2. Averaged performance over held-out

(user, item) interaction pairs (Average-Over-All)

Rating-based recommendation systems Implicit feedback-based recommendation systems

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

46

Previous work: Av Average-Ov Over er-Al All is bi biased sed fo for rating-ba based sed recommendatio ion systems, because ratin ings are MN MNAR

[Marlin et al. 09], [Schnabel et al. 16], [Steck 10], [Steck 11], and [Steck 13]

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

47

Previous work: Av Average-Ov Over er-Al All is bi biased sed fo for rating-ba based sed recommendatio ion systems, because ratin ings are MN MNAR

[Marlin et al. 09], [Schnabel et al. 16], [Steck 10], [Steck 11], and [Steck 13]

Previous work: Av Average-Ov Over er-Al All is unb unbiased fo for implicit fe feedback-ba based sed rec ecommen enda dation sy syst stem ems, s, bec because se impl plicit feedback is is mi missing ng uni uniforml mly at rand ndom.

[Lim 15]

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

48

This work: Av Average-Ov Over er-Al All is bi biased sed fo for implicit feedback- ba based sed rec ecommen enda dation sy syst stem ems, s, bec because se impl plicit feedba eedback is s NO NOT mi missing ng uni uniforml mly at rand ndom.

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

49

This work: Av Average-Ov Over er-Al All is bi biased sed fo for implicit feedback- ba based sed rec ecommen enda dation sy syst stem ems, s, bec because se impl plicit feedba eedback is s NO NOT mi missing ng uni uniforml mly at rand ndom.

Popularity bias (Users are more likely to be exposed to popular items)

trending recommendation

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

50

A Hy Hypothetical Example

Popular Items Long-tail Items # of liked items (over all items) # of liked items (over observations)

1 10 10 1 : :

Algorithm 1 Performance Algorithm 2 Performance

0.8 0.75 0.75

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

51

A Hy Hypothetical Example

Popular Items Long-tail Items # of liked items (over all items) # of liked items (over observations)

1 10 10 1 : :

Algorithm 1 Performance Algorithm 2 Performance

0.8 0.75 0.75

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

52

Popular Items Long-tail Items # of liked items (over all items) # of liked items (over observations)

1 10 10 1 : :

Algorithm 1 Performance Algorithm 2 Performance

0.8 0.75 0.75 A Hy Hypothetical Example

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

53

Popular Items Long-tail Items # of liked items (over all items) # of liked items (over observations)

1 10 10 1 : :

Algorithm 1 Performance Algorithm 2 Performance

0.8 0.75 0.75 A Hy Hypothetical Example

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

54

Popular Items Long-tail Items # of liked items (over all items) # of liked items (over observations)

1 10 10 1 : :

Algorithm 1 Performance Algorithm 2 Performance

0.8 0.75 0.75 A Hy Hypothetical Example

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

55

Popular Items Long-tail Items # of liked items (over all items) # of liked items (over observations)

1 10 10 1 : :

Algorithm 1 Performance Algorithm 2 Performance

0.8 0.75 0.75

Any sensible evaluation

A Hy Hypothetical Example

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

56

Popular Items Long-tail Items # of liked items (over all items) # of liked items (over observations)

1 10 10 1 : :

Algorithm 1 Performance Algorithm 2 Performance

0.8 0.75 0.75

Average-Over- All

A Hy Hypothetical Example

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

57

Fo Formaliz lize R Reward !

R( ˆ Z) = 1 |U| X

u∈U

1 |Su| X

i∈Su

c( ˆ Zu,i)

Ideal evaluation:

Item rankings predicted by an algorithm

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

58

Fo Formaliz lize R Reward !

R( ˆ Z) = 1 |U| X

u∈U

1 |Su| X

i∈Su

c( ˆ Zu,i)

Predicted ranking of item i for user u Items liked by user u among the entire item set Reward for (u, i) pair scoring metric

Item rankings predicted by an algorithm

Ideal evaluation:

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

59

Fo Formaliz lize R Reward !

R( ˆ Z) = 1 |U| X

u∈U

1 |Su| X

i∈Su

c( ˆ Zu,i)

Predicted ranking of item i for user u Items liked by user u among the entire item set Reward for user u scoring metric

Item rankings predicted by an algorithm

Ideal evaluation:

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

60

Fo Formaliz lize R Reward !

R( ˆ Z) = 1 |U| X

u∈U

1 |Su| X

i∈Su

c( ˆ Zu,i)

Predicted ranking of item i for user u Items liked by user u among the entire item set scoring metric

Item rankings predicted by an algorithm

Reward for the algorithm

Ideal evaluation:

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

61

Fo Formaliz lize R Reward !

Average-Over-All:

ˆ RAOA( ˆ Z) = 1 |U| X

u∈U

1 |S∗

u|

X

i∈S∗

u

c( ˆ Zu,i)

Items liked by user u (observed)

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

62

Fo Formaliz lize B Bia ias

Ou,i = 1 if (u, i) is observed, and Ou,i = 0 otherwise EO h ˆ RAOA( ˆ Z) i 6= R( ˆ Z) Ou,i ∼ B(1, Pu,i)

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

63

ˆ RIPS( ˆ Z|P) = 1 |U| X

u∈U

1 |Su| X

i∈S∗

u

c( ˆ Zu,i) Pu,i ˆ RAOA( ˆ Z) = 1 |U| X

u∈U

1 |S∗

u|

X

i∈S∗

u

c( ˆ Zu,i)

In Inverse-Pr Propen pensi sity-Sc Scorin ing ( (IPS) S)

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

64

ˆ RIPS( ˆ Z|P) = 1 |U| X

u∈U

1 |Su| X

i∈S∗

u

c( ˆ Zu,i) Pu,i ˆ RAOA( ˆ Z) = 1 |U| X

u∈U

1 |S∗

u|

X

i∈S∗

u

c( ˆ Zu,i) EO h ˆ RIPS( ˆ Z|P) i = R( ˆ Z)

In Inverse-Pr Propen pensi sity-Sc Scorin ing ( (IPS) S)

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

65

ˆ RIPS( ˆ Z|P) = 1 |U| X

u∈U

1 |Su| X

i∈S∗

u

c( ˆ Zu,i) Pu,i

Se Self lf-No Norma malized Inverse-Pr Propen pensi sity-Sc Scorin ing ( (SN SNIPS) S)

[Swaminathan et al.15] 15]

ˆ RSNIPS( ˆ Z|P) = 1 |U| X

u∈U

1 P

i∈S∗

u

1 Pu,i

X

i∈S∗

u

c( ˆ Zu,i) Pu,i

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

66

Es Estima mating ng Propens nsity Scores

Factor: Popularity bias (Users are more likely to be exposed to popular items) Assumptions:

• User-independence assumption
• Two-steps assumption
• User preference is not affected by item presentation

Pu,i = P(Ou,i = 1) = P(O∗,i = 1) = P∗,i P∗,i = P select

∗,i

· P interact|select

∗,i

P interact|select

∗,i

= P interact

∗,i

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

67

Popularity bias model [Steck 11]:

ˆ P select

∗,i

∝ (n∗

i )γ

Observed item popularity

Es Estima mating ng Propens nsity Scores

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

68

Popularity bias model [Steck 11]:

ˆ P select

∗,i

∝ (n∗

i )γ

ˆ P∗,i ∝ (n∗

i )( γ+1

2 )

Estimated from known

• nline content serving

policy

Es Estima mating ng Propens nsity Scores

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

69

Me Measuring bias in recommender evaluation (Yahoo! music ic ratin ing dataset)

Model Average- Over-All !SNIPS (# = 1.5) !SNIPS (# = 2.0) !SNIPS (# = 2.5) !SNIPS (# = 3.0) U-CML 0.401 0.270 0.260 0.253 0.248 A-CML 0.399 0.274 0.264 0.258 0.253 BPR 0.380 0.275 0.268 0.262 0.258 PMF 0.386 0.267 0.259 0.252 0.248 Mean Absolute Error (MAE), Recall RSNIPS produces significantly lower MAE

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

70

Me Measuring bias in recommender evaluation (Yahoo! music ic ratin ing dataset)

Model Average- Over-All !SNIPS (# = 1.5) !SNIPS (# = 2.0) !SNIPS (# = 2.5) !SNIPS (# = 3.0) U-CML 0.401 0.270 0.260 0.253 0.248 A-CML 0.399 0.274 0.264 0.258 0.253 BPR 0.380 0.275 0.268 0.262 0.258 PMF 0.386 0.267 0.259 0.252 0.248 Mean Absolute Error (MAE), Recall RSNIPS produces significantly lower MAE

The accuracy of recommending popular items is a significant overestimation of the true recommendation performance

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys), 2018.

71

Age Agend nda

More than Just Words (WSDM’19) Intention Informed Recommendations (Under Review) Debias Offline Recommendation Evaluation (Recsys’18)

72

2-by by-2 2 Randomized Controlled Trial RS: How do in intentio ion-in informed recommendations modulate users po podc dcast st conten ent choices es?

Longqi Yang, Michael Sobolev, Yu Wang, Jenny Chen, Drew Dunne, Christina Tsangouri, Nicola Dell, Mor Naaman, and Deborah Estrin. How intention informed recommendations modulate choices: A field study of spoken word content. Under Review, 2019.

73

2-by by-2 2 Randomized Controlled Trial

Longqi Yang, Michael Sobolev, Yu Wang, Jenny Chen, Drew Dunne, Christina Tsangouri, Nicola Dell, Mor Naaman, and Deborah Estrin. How intention informed recommendations modulate choices: A field study of spoken word content. Under Review, 2019.

RS: How do in intentio ion-in informed recommendations modulate users po podc dcast st conten ent choices es?

Intended topics-informed Rec. Intended channels-informed Rec. No Rec. (Trending chart) No Rec. (Subscription-only)

X

74

Ma Main Findings

Longqi Yang, Michael Sobolev, Yu Wang, Jenny Chen, Drew Dunne, Christina Tsangouri, Nicola Dell, Mor Naaman, and Deborah Estrin. How intention informed recommendations modulate choices: A field study of spoken word content. Under Review, 2019.

Intended topics-informed Rec. 72% related onboarding subs. 36% related field subs. 24% related field listening Intended channels-informed Rec. 127% listening exploration

75

Ma Main Findings

Longqi Yang, Michael Sobolev, Yu Wang, Jenny Chen, Drew Dunne, Christina Tsangouri, Nicola Dell, Mor Naaman, and Deborah Estrin. How intention informed recommendations modulate choices: A field study of spoken word content. Under Review, 2019.

Intended topics-informed Rec. Intended channels-informed Rec.

X

Intended channels-informed Rec.

X

No Rec. (Trending chart) User satisfaction User satisfaction

76

http://www.openrec.ai

Github link, documents, and tutorials

Longq Longqi i Yang ang

Ph.D. candidate Computer Science, Cornell Tech, Cornell University Email: ylongqi@cs.cornell.edu Web: bit.ly/longqi Twitter: @ylongqi Connected Experiences Lab http://cx.jacobs.cornell.edu/ Small Data Lab http://smalldata.io/

Funders: