Buildi ding ng Reco commen mmende ders rs and Searc rch h - PowerPoint PPT Presentation

Buildi ding ng Reco commen mmende ders rs and Searc rch h Engines es by Re-usin sing g User r Feedback ck Adit ith Sw Swaminathan adswamin@microsoft.com ad Join Joint work with Thorst sten n Joa Joachims s an and d Tobia ias Sch Schnabel (Co (Cornell Uni niversit ity) Ack Ack: NS NSF F Gr Grants

Bi Bio Counterfactual Evaluation MSR - DLTC and Learning 2

Summary mmary “Use logs collected from interactive systems to evaluate/train new interaction policies” “Randomize “Pay attention to feedback effects, cleverly to break Now: Simple/pragmatic confounding/feed and dis-entangle techniques to tackle back” -- Yisong them” -- David biased user feedback 3

Wald’s insight: What’s missing? • Where re to add armor? or? Cover er bullet et-holes? holes? (Survivor rvivor bias!) s!) • Beware: are: Confound founding ing due to missi sing ng info 4

Overview verview • “Use user ratings for collaborative filtering” – Project: t: MNAR (Schnabel et al, ICML 2016) • “Use user clicks for search ranking” – Project: t: ULTR (Joachims et al, WSDM 2017) 5

Movie vie Recommen commendation ation O Horro ror Romance Drama ma Observe served Y/N 5 5 1 3 5 1 3 Lovers rs ror 5 5 Horro 5 5 1 3 5 5 5 3 5 5 1 3 3 Data a is Missi sing ng Not At Random om (MNAR) AR) 1 1 5 5 3 3 1 5 3 Romance 5 5 Lovers rs Y 5 True Rati ting 5 5 5 3 1 5 5 3 Example adapted from (Steck et al, 2010) 6

Se Select ection ion Bi Bias as in n Rec ecommend mmendati tion ons • User-induced (e.g. browsing) • System-induced (e.g. advertising) Question: What if we ignore these biases? 7

Evaluatin Ev aluating g rec ecommend mmendations ations un under der Se Select ection ion Bi Bias as O Horro ror Romance Drama ma Observe served Y/N 5 5 5 5 1 1 3 3 5 1 3 rs Lovers ror 5 5 5 5 Horro ෡ 𝒁 5 5 5 5 1 1 3 3 Reco commend 5 5 5 5 5 5 3 3 5 5 5 5 1 1 3 3 3 3 Observed erved ratings ngs are misleadi eading ng 1 1 1 1 5 5 5 5 3 3 3 3 1 5 3 Romance rs Lovers 5 5 5 5 Y 5 5 3 True Rati ting 5 5 5 5 5 5 3 3 1 1 5 5 5 5 3 3 8

Ev Evaluatin aluating g rating ating pr predictions edictions un under der Se Select ection ion Bi Bias as Horro ror Romance Drama ma Horro ror Romance Drama ma 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 3 3 3 3 3 3 3 3 3 3 5 5 5 5 5 5 5 5 5 5 1 1 1 1 1 1 1 1 1 1 5 5 5 5 5 5 5 5 5 5 Lovers rs ror 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 3 3 3 3 3 3 3 3 3 3 5 5 5 5 5 1 1 1 1 1 5 5 5 5 5 Horro 5 5 5 5 5 1 1 1 1 1 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 3 3 3 3 3 3 3 3 3 3 5 5 5 5 5 5 5 5 5 5 1 1 1 1 1 1 1 1 1 1 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 3 3 3 3 3 3 3 3 3 3 5 5 5 5 5 5 5 5 5 5 1 1 1 1 1 1 1 1 1 1 5 5 5 5 5 5 5 5 5 5 Observed erved losses es are misleadi eading ng 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 3 3 3 3 3 3 3 3 3 3 5 5 5 5 5 1 1 1 1 1 5 5 5 5 5 5 5 5 5 5 1 1 1 1 1 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 3 3 3 3 3 3 3 3 3 3 1 1 1 1 1 1 1 1 1 1 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 Romance rs Lovers 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 3 3 3 3 3 3 3 3 3 3 1 1 1 1 1 1 1 1 1 1 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 3 3 3 3 3 3 3 3 3 3 1 1 1 1 1 1 1 1 1 1 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 3 3 3 3 3 3 3 3 3 3 1 1 1 1 1 1 1 1 1 1 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 3 3 3 3 3 3 3 3 3 3 1 1 1 1 1 1 1 1 1 1 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 ෠ ෠ 𝑍 𝑍 1 2 Pred Ratings (worse) Pred Ratings (better) 9

Rec ecommend mmendati tion ons s as as Treat eatments ments Fix select ction on bias s  potentia ential l outcomes comes frame amework work Counterfactual Outcomes 𝑍 Factual Outcomes ෨ 𝑍 treatme ments Items ms 5 5 1 3 5 1 3 Users rs 5 5 5 5 1 3 5 5 5 3 5 5 1 3 3 1 1 5 5 3 3 tients 1 5 3 patien 5 5 5 5 5 5 3 1 5 5 3 ⇒ Understand erstand assign ignme ment nt mechani hanism sm (Imbens & Ruben, 2015) 10

As Assi signm gnment ent Mec echanism anism for or Rec ecommend mmendati tion on 𝑄 𝑣,𝑗 = 𝑄 𝑃 𝑣,𝑗 = 1 Propensiti pensities es P Inverse Propensity Scoring Drama Horror or Roman ance ce (IPS) is unbiased if 𝑄 𝑣,𝑗 > 0 : 𝑞 𝑞/10 𝑞/2 2 1 𝟚{𝑃 𝑣𝑗 =1} ෠ 𝑣,𝑗 − ෠ 𝑆 𝐽𝑄𝑇 = 𝑉⋅𝐽 ෍ 𝑍 𝑍 𝑣,𝑗 𝑄 𝑣,𝑗 𝑣,𝑗 𝑞/10 𝑞 𝑞/2 (Horvitz & Thompson, 1952; Rosenbaum & Rubin, 1983; ...) 11

Debiasing ebiasing Ev Evalua aluation tion Seve verity rity of of Sele lecti ction Bias Seve verity rity of of Sele lecti ction Bias IPS S is robust ust to selection ction bias 12

Ex Exper perime menta ntal l vs. . Obs bser erva vation tional al • Control trolled led Experim eriments ents – We control ntrol assign ignme ment nt mechan hanis ism m (e.g. .g. ad place acemen ment) t) – Prop open ensiti ities es 𝑄 𝑣,𝑗 = 𝑄 𝑃 𝑣,𝑗 = 1 kno nown wn [ Just t log g prop open ensiti ities es! ] – Requ quireme irement: nt: 𝑄 𝑣,𝑗 > 0 (prob. b. assign ignmen ment) t) • Observa ervational onal Study dy – Assign ignmen ment mecha hanis nism m not t under der our cont ontrol ol (e.g. .g. revie iews ws/rating /ratings) – Use e featu atures 𝑎 ; ; ෠ [ [ Estima timate te prope opens nsity ity ] 𝑄 𝑣,𝑗 = 𝑄 𝑃 𝑣,𝑗 = 1| 𝑎 – Requ quireme irement: nt: 𝑃 𝑣,𝑗 ⊥ 𝑍 (unc ncon onfou found nded) ed) 𝑣,𝑗 | 𝑎 13

Pr Prope opens nsity ity Es Estimatio imation • Supervi ervise sed d Regress ession ion Probl blem em ෠ 𝑄 𝑣,𝑗 = 𝑄 𝑃 𝑣,𝑗 = 1| 𝑎 Observa ervations ons O Horr rror Romance ce Drama 1 0 1 0 0 1 0 0 0 0 0 0 1 0 0 Off-the he-sh shelf elf ML, e.g., ., • 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 0 0 0 0 – Logis gistic ic regre gression ion 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 1 0 0 0 0 1 0 1 0 0 0 1 – Naïv ïve e Bayes es 1 0 0 0 1 1 0 0 1 0 0 1 0 0 1 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 – Bernou noulli lli Matrix trix Factor toriz izati ation on 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 1 1 0 0 0 0 – … 1 0 0 0 0 0 1 1 0 0 0 0 0 1 0 IPS S is robust ust to inaccura curate te propen pensiti sities es 14

Debiased ebiased Col ollabo labora rative tive Filter tering ing 1 2 + 𝜇 𝑍 𝐹𝑆𝑁 = argmin 2 + 𝑋 𝐺 ෠ 2 ෍ 𝑍 𝑣,𝑗 − 𝑊 𝑣 𝑋 𝑊 𝐺 𝑗 𝑄 𝑣,𝑗 𝑊,𝑋 𝑃 𝑣,𝑗 =1 Latent variables Obse serva rvati tions s O Prop open ensity Featu tures res Z estimat mation on MF MF Comple lete te Missin sing Obse serve rved Data ta Model Data ta Model s ෩ ratin ings 𝒁 discriminative generative (Marlin et al, 2007; Steck, 2011; ...) 15

Col ollabo labora rative tive Filtering tering Results esults • Two real-worl world d MNAR R datasets asets – YAHO HOO: Song ng rating ings (154 5400 00 users; ers; Marlin & Zemel, 2009 ) – COAT: T: Shopp oppin ing g ratin ings gs (300 00 users; ers; new ew Schnabel et al, 2016 ) • Report rt performa formance nce on MAR datase asets ts http://www.cs.cornell.edu/~schnabts/mnar/ 16

Overview verview • “Use user ratings for collaborative filtering” – Project: t: MNAR (Schnabel et al, ICML 2016) • “Use user clicks for search ranking” – Project: t: ULTR (Joachims et al, WSDM 2017) 17

ҧ ҧ Learning-to-Rank from Clicks Query Distribution Presented 𝒛 𝟐 𝑦 𝑗 ∼ 𝑸(𝒀) Presented 𝒛 𝟐 Presented 𝒛 𝟐 Deployed Ranker Presented 𝒛 𝟐 Click Presented 𝒛 𝟐 A 𝑧 𝑗 = ത 𝑇(𝑦 𝑗 ) Presented 𝒛 𝟐 A Presented ഥ 𝒛 𝒐 A Click A A Click B A Click B A B Learning New Ranker B B C B 𝑇(𝑦) Click Algorithm C B Click C C C D C D C Click D D Should perform D E D better than E D E Click E 𝑇(𝑦) E F E F E F F F G F G F G G G G G

ҧ Evaluating Rankings Deployed Ranker New Ranker to Evaluate 𝑧 = ത 𝑇("𝑻𝑾𝑵") 𝑧 = 𝑻("𝑻𝑾𝑵") Presented ഥ Presented ഥ 𝒛 𝒐 𝒛 New 𝒛 New 𝒛 New 𝒛 Presented ഥ 𝒛 A A F F F A 1 Manually Labeled B B B G G G 2 C C C D D D 3 Click C D D D C C 4 E E E E E E F F A A A F 6 G G G B B B 7