SLIDE 1
DOLEAD @ Py.Paris –NOV 2018
WWW.DOLEAD.COM
Beat Google’s bidder using ML
Dolead RD team Arnaud Fouchet
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
2
Paid search basics
Beat Googles bidder using ML Dolead RD team Arnaud Fouchet DOLEAD - - PowerPoint PPT Presentation
Beat Googles bidder using ML Dolead RD team Arnaud Fouchet DOLEAD - - PowerPoint PPT Presentation
Beat Googles bidder using ML Dolead RD team Arnaud Fouchet DOLEAD @ Py.Paris NOV 2018 WWW.DOLEAD.COM Paid search basics 2 DOLEAD @ Py.Paris - NOV 2018 WWW.DOLEAD.COM Paid search basics 3 DOLEAD @ Py.Paris - NOV 2018
SLIDE 2
SLIDE 3
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
3
Paid search basics
SLIDE 4
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
4
Paid search basics
Click
SLIDE 5
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
5
Paid search basics
Click Visit Landing page
SLIDE 6
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
6
Paid search basics
Click Visit Landing page Subscription (victory) Not interested (defeat)
SLIDE 7
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
7
Paid search basics
Typed: “BTS a distance” Will he click ? Will he subscribe ? New user Impression Click Subscription
SLIDE 8
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
8
Paid search basics
Typed: “BTS a distance” Will he click ? Will he subscribe ? New user Impression Click Subscription
100$ question : how much should I pay the ad ?
SLIDE 9
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
9
Paid search basics
Typed: “BTS a distance” Will he click ? Will he subscribe ? New user Impression Click Subscription
100$ question : how much should I pay the ad ?
Google’s problem
SLIDE 10
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
10
Paid search basics
Typed: “BTS a distance” Will he click ? Will he subscribe ? New user Impression Click Subscription
10$ question : how much should I pay the click ?
Google’s problem
SLIDE 11
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
11
Paid search basics
How much should I pay the click ?
- 1. Find revenue by click
SLIDE 12
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
12
Paid search basics
How much should I pay the click ?
- 1. Find revenue by click
- 2. Pay less than that
SLIDE 13
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
13
Paid search basics
How much should I pay the click ?
- 1. Find revenue by click
(using Machine learning: search query → revenue by click)
- 2. Pay less than that
SLIDE 14
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
14
Paid search basics
Typed: “BTS a distance” Will he click ? Will he subscribe ? New user Impression Click Subscription
10$ question : how much should I pay the click ?
Google’s problem Google’s bidder
SLIDE 15
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
15
Paid search basics
Typed: “BTS a distance” Will he click ? Will he subscribe ? New user Impression Click Subscription
1$ for 2 questions : Do I accept giving Google all power ? Can we do better ?
Google’s problem Google’s bidder
SLIDE 16
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
16
Paid search basics
Typed: “BTS a distance” Will he click ? Will he convert ? New user CPA: Cost Per Acquisition (Cost of a conversion) Impression Click Conversion
1$ for 2 questions : Do I accept giving Google all power ? Can we do better ?
Google’s problem Google’s bidder
SLIDE 17
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
17
Basic ML steps
- Identify learnable outcome (revenue by click)
- Transform input (user’s search query) into features (X)
- Identify relevant loss function
- Try and optimize models
SLIDE 18
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
18
Our ML research
SLIDE 19
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
19
Our ML research
SLIDE 20
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
20
Our ML research
SLIDE 21
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
21
Consensus
RF GBDT Log Reg ...
SLIDE 22
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
22
Our ML research
SLIDE 23
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
23
Neural networks
SLIDE 24
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
24
Paid search basics
Typed: “BTS a distance” Will he click ? Will he convert ? New user Impression Click Conversion
10$ question : how much should I pay the click ?
Google’s problem
SLIDE 25
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
25
Our ML research
SLIDE 26
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
26
Paid search basics
Typed: “BTS a distance” Will he click ? Will he convert ? New user Impression Click Conversion
10$ question : how much should I pay the click ?
Google’s problem
SLIDE 27
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
27
Our ML research
SLIDE 28
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
28
Our ML research
SLIDE 29
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
29
Consensus
RF GBDT NN ... Better
SLIDE 30
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
30
Consensus
RF GBDT NN ... Better a x RF b x GBDT c x NN ...
Weighted Consensus
SLIDE 31
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
31
Consensus
RF GBDT NN ... Better a x RF b x GBDT c x NN ...
Weighted Consensus
RF GBDT NN ...
Learned model
f
SLIDE 32
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
32
Stacked models
x RF NN GBDT ... x’ Learning problem on (x’, y)
SLIDE 33
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
33
Stacked models
x RF NN GBDT ... x’ Learning problem on (x’, y)
➝ data leakage problem
Train
SLIDE 34
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
34
Stacked models
x RF NN GBDT DEVIL x’ Learning problem on (x’, y)
➝ data leakage problem
Train
SLIDE 35
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
35
Measure Stacked Models Perf
Campaign 1 Train Test Predictions RF NN GBDT ... Campaign 2 Train Test Predictions RF NN GBDT ... Campaign ...
SLIDE 36
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
36
Measure Stacked Models Perf
Campaign 1 Train Test Predictions RF NN GBDT ... Campaign 2 Train Test Predictions RF NN GBDT ... Campaign ... Campaign T+1 Train Test Predictions RF NN GBDT ... Campaign T+2 Train Test Predictions RF NN GBDT ... Campaign T+ ...
Test !
SLIDE 37
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
37
Our ML research
SLIDE 38
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
38
Bidder comparison: context
A/B test
Campaign Experiment 1: 50% bids Google Experiment 2: 50% bids Dolead
SLIDE 39
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
39
Bidder comparison: context
A/B test 50 campaigns 4 weeks Goal : maximize conversions while respecting target CPA
Campaign Experiment 1: 50% bids Google Experiment 2: 50% bids Dolead
SLIDE 40
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
40
Bidder comparison: % Below target CPA
Raw results :
- Google 33%
- Dolead 63%
Distribution
CPA(Google) / target CPA CPA(Dolead) / target CPA
SLIDE 41
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
41
Bidder comparison: Conversions & CPA
Conversions : In 67% of campaigns, Google had more convs Distribution
Convs(Google) / Convs(Dolead)
SLIDE 42
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
42
Bidder comparison: Conversions & CPA
Conversions : In 67% of campaigns, Google had more convs Distribution CPA : In 83% of campaigns, Dolead was cheaper Distribution
Convs(Google) / Convs(Dolead) CPA(Google) / CPA(Dolead)
SLIDE 43
DOLEAD @ Py.Paris - NOV 2018
WWW.DOLEAD.COM
43
Bidder comparison: Conversions & CPA
Conversions : In 67% of campaigns, Google had more convs Distribution Google’s margin: 9.8K CPA : In 83% of campaigns, Dolead was cheaper Distribution Dolead’s margin: 13.3K (~+35%)
Convs(Google) / Convs(Dolead) CPA(Dolead) / CPA(Google)
SLIDE 44
DOLEAD @ Py.Paris –NOV 2018
WWW.DOLEAD.COM
Thanks for your attention !
PyApero 13 décember 2018 www.pyapero.com