SLIDE 1
A Feedback Shift Correction in Predicting Conversion Rates under Delayed Feedback
Shota Yasui, Gota Morishita, Komei Fujita, Masashi Shibata The Web Conference 2020
Introduction and Problem Setting
2
A Feedback Shift Correction in Predicting Conversion Rates under - - PowerPoint PPT Presentation
A Feedback Shift Correction in Predicting Conversion Rates under - - PowerPoint PPT Presentation
A Feedback Shift Correction in Predicting Conversion Rates under Delayed Feedback Shota Yasui, Gota Morishita, Komei Fujita, Masashi Shibata The Web Conference 2020 Introduction and Problem Setting 2
SLIDE 2
SLIDE 3
Conversion Prediction
Predict Conversion-Rate(CVR) for each request.
DSP
bid
Predicting CVR is important to decide the bid price
User use Apps AD Auction
request
3
SLIDE 4
Ideal loss function
The following loss should be minimized. The ideal parameters are as follow This is not possible! Because we do not observe c due to the delayed feedback.
features Conversion model
4
SLIDE 5
Delayed Feedback
5
SLIDE 6
Delayed Feedback
6
timestamp of Click timestamp of CV time
timestamp of click and cv for certain user
- user takes sometimes to purchase items after clicked the ad.
delay
SLIDE 7
The problem of Delayed Feedback
7
- we can not observe CV for this user
- this sample is recognized as negative label! (mislabeled)
timestamp of Click timestamp of CV time training begins
timestamp of click and cv for certain user
included in training data
Unobserved
SLIDE 8
The relation between Y and C
8
C=1 C=0 Y=1 Y=0 mislabeled S = 0 correctly labeled S = 1 true label
- bservable label
Prob of correctly labeled Prob of mislabel
SLIDE 9
Bias in standard supervised approach
ideal loss actual loss(ERM) Inconsistent!
9
SLIDE 10
Our Solution
Importance Weight Approach
10
SLIDE 11
Importance Weight(FSIW) approach
11
ideal-loss Unbiased-loss (consistent?)
We propose consistent loss based on the Importance Weight(Propensity Score)
Importance Weight
SLIDE 12
Importance Weight(FSIW) approach
Our empirical loss The basic idea is to weight each sample by the conditional density ratio.
12
Importance Weight
SLIDE 13
How to estimate FSIW
We estimate these probability from data old enough to observe S and C.
13
SLIDE 14
14
week 1 week 2 week 3 discard training data Counterfactual Dead Line
SLIDE 15
15
week 1 week 2 week 3 discard
Train models for
training data Counterfactual Dead Line
SLIDE 16
16
week 1 week 2 week 3 discard
Train models for
training data Counterfactual Dead Line week 1 week 2 week 3 Importance weight
Train the CVR model
training data
SLIDE 17
features of our proposed method
It is just a importance weight ○ can be used for any CVR model ○ can fit the delay nonparametrically ○ does not increase the time complexity of CVR models
17
SLIDE 18
Experiment
18
SLIDE 19
Conversion Logs Dataset
- Open data provided by Criteo（Link）
- 30days of click and CV log
- Used in Chapelle(2014)
- bservation period is 30days
19
SLIDE 20
Experiment procedure
20
train(3 weeks) test train(3 weeks) test train(3 weeks) test
time
averaging these results
day = 22 day = 23 day = 24
train(3 weeks) test
day = 28
iterate for 7days
SLIDE 21
Result 1
Proposed Method Chapelle(2014) Pure-Logistic Regression
- Normalized-logloss(NLL) is the most important metrics
○ we use prediction probability for bidding ○ logloss(LL) is sensitive to the base CVR
21
SLIDE 22
Dynalyst Data
22
- DSP in Cyberagent.inc
- 2 experiments
○ the same procedure as the first experiment ■ focus on three campaigns ■ baseline model is FFM (Juan 2017) ○ Online A/B test
SLIDE 23
Three Campaigns
- Observational period is different by campaings
○ S: 1days ○ M: 3days ○ L: 7days
23
SLIDE 24
Result 2
Only Campaign L shows the improvement.
24
SLIDE 25
Follow Up Online Experiment@Campaign-L
25
- Improved cost consumption and CV.
- CPA does not change or slightly decreased.
SLIDE 26
Conclusion
- We proposed a consistent loss to predict CVR under Delayed
Feedback.
- Our method performs better in two offline and one online
experiment. Thank you for listening! 26
SLIDE 27
appendix
SLIDE 28
cumulative distribution of delay
28
SLIDE 29
effect of counterfactual deadline
29