PERSONALIZING PERSONALIZING PERSONALIZING PERSONALIZING - - PowerPoint PPT Presentation

▶
personalizing personalizing personalizing personalizing
SMART_READER_LITE
LIVE PREVIEW

PERSONALIZING PERSONALIZING PERSONALIZING PERSONALIZING - - PowerPoint PPT Presentation

Apr 22, 2023 198 likes •664 views

PERSONALIZING PERSONALIZING PERSONALIZING PERSONALIZING PERSONALIZING PERSONALIZING PERSONALIZING PERSONALIZING PERSONALIZING PERSONALIZING PERSONALIZING PERSONALIZING SEARCH RESULTS IN REAL-TIME Grebennikov Roman / ndify.io

slide-1
SLIDE 1

PERSONALIZING PERSONALIZING PERSONALIZING PERSONALIZING PERSONALIZING PERSONALIZING PERSONALIZING PERSONALIZING PERSONALIZING PERSONALIZING PERSONALIZING PERSONALIZING

SEARCH RESULTS IN REAL-TIME

Grebennikov Roman / / / ndify.io @public_void_grv grv@dfdx.me 1
slide-2
SLIDE 2

ABOUT FINDIFY

white-label eCommerce SaaS search 1500 stores, 20M products 50M customers per month

2
slide-3
SLIDE 3

FINDIFY IN 2014

UI-focused Shopify search addon Backed by ElasticSearch Nothing special about product ranking

3
slide-4
SLIDE 4

RANKING IS IMPORTANT

nobody is scrolling down

4
slide-5
SLIDE 5

RANKING IS IMPORTANT

no second search

5
slide-6
SLIDE 6

RANKING IS IMPORTANT

no second visit

6
slide-7
SLIDE 7

TYPICAL CUSTOMER SESSION

  • 1. Arrive on a landing/product page (0s)
  • 2. Click on product collections (+10s)
  • 3. Make a search (+20s)
  • 4. Leave forever (+30s)
7
slide-8
SLIDE 8

BETTER RANKING

8
slide-9
SLIDE 9

BETTER RANKING?

9
slide-10
SLIDE 10 10
slide-11
SLIDE 11

AI ML (LINEAR REGRESSION)

Algorithm Conversion AOV Elasticsearch baseline baseline Regression +3.1% +2.5%

11
slide-12
SLIDE 12

REINVENTING THE WHEEL

Learn to Rank LambdaMART XGBoost/LightGBM/CatBoost

12
slide-13
SLIDE 13

ELASTICSEARCH INTEGRATION

13
slide-14
SLIDE 14

ELASTICSEARCH INTEGRATION

14
slide-15
SLIDE 15

TRAINING

Historical click/purchase data Model per merchant Optimize for NDCG, watch for conversion

15
slide-16
SLIDE 16

FEATURE GROUPS

search: # of terms, # of lters product: price, # of pageviews variant: color, size current session: price sensitivity, # of searches historical sessions: # of sessions product and search: # of pageviews within context + different time windows

16
slide-17
SLIDE 17 17
slide-18
SLIDE 18

MIXED RESULTS

18
slide-19
SLIDE 19

MIXED RESULTS

Algorithm Conversion AOV Elasticsearch baseline baseline Regression +3.1% +2.5% LMART v1 +6.1% (+8.1%) no data

19
slide-20
SLIDE 20

TRAINING ISSUES

Historical click/purchase data Model per merchant Optimize for NDCG

20
slide-21
SLIDE 21

POSITIVE FEEDBACK LOOP

21
slide-22
SLIDE 22

POSITIVE FEEDBACK LOOP

22
slide-23
SLIDE 23

POSITION BIAS POSITION BIAS POSITION BIAS POSITION BIAS POSITION BIAS POSITION BIAS POSITION BIAS POSITION BIAS POSITION BIAS POSITION BIAS POSITION BIAS POSITION BIAS

CUSTOMERS ARE CLICKING ONLY ON FIRST PRODUCTS

23
slide-24
SLIDE 24

RANDOM RANKING

24
slide-25
SLIDE 25

RANDOM RANKING

Algorithm Conversion AOV Elasticsearch baseline baseline Regression +3.1% +2.5% LMART v1 +6.1% (+8.1%) no data Random

  • 2.8%
  • 1.3%
25
slide-26
SLIDE 26

POSITION BIAS

  • L. Li, W. Chu, J. Langford, R. Schapire. 2010.
A contextual-bandit approach to personalized news article recommendation.

Exploration and exploitation segments Un-biasing the training data

26
slide-27
SLIDE 27

EXPLORATION SEGMENT

tiny segment, 0.1-1% of trafc rst page is shufed used for training

27
slide-28
SLIDE 28

TRAINING ISSUES

Historical Unbiased click/purchase data Model per merchant Optimize for NDCG

28
slide-29
SLIDE 29

MODEL PER MERCHANT

Low-trafc merchants Onboarding and data collection time Sacricing ranking for "Exploration segment"

29
slide-30
SLIDE 30

SUGGESTIONS HACKATHON

Replace heuristics with ML Simpler problem than search All features are language-specic small, medium, large merchant

30
slide-31
SLIDE 31

BETTER SUGGESTIONS?

31
slide-32
SLIDE 32

MODEL TRANSPLANT

from large-trafc merchant to small-trafc:

32
slide-33
SLIDE 33

GENERIC MODEL

More training samples More diverse dataset No need for per-merchant data collection All features need to be scaled

33
slide-34
SLIDE 34

TRAINING ISSUES

Historical Unbiased click/purchase data Model per merchant Generic model Optimize for NDCG

34
slide-35
SLIDE 35

NDCG

1.0 - good, 0.0 - bad, 0.4-0.7 - normal compares perfect ranking with real what is a perfect ranking?

35
slide-36
SLIDE 36

PERFECT RANKING

36
slide-37
SLIDE 37

STANLEY BONG ISSUE

Rank improved from #20 to #1 Never bought Costs 3500$

37
slide-38
SLIDE 38

STANLEY BONG ISSUE

  • ver-optimized for clicks
38
slide-39
SLIDE 39

PERFECT RANKING

39
slide-40
SLIDE 40

TRAINING ISSUES

Historical Unbiased click/purchase data Model per merchant Generic model Optimize for NDCG (with proper weights)

40
slide-41
SLIDE 41

RESULTS RESULTS RESULTS RESULTS RESULTS RESULTS RESULTS RESULTS RESULTS RESULTS RESULTS RESULTS

41
slide-42
SLIDE 42

NDCG WITH PERSONALIZATION

Algorithm NDCG (ofine) Random 0.544 Popularity 0.578 Elasticsearch 0.601 Regression 0.615 LMART v1 ~0.621 LMART unbiased 0.635

42
slide-43
SLIDE 43

NDCG AND BUSINESS METRICS

Algorithm NDCG CTR Conversion AOV Elasticsearch 0.601 baseline baseline baseline Random 0.544

  • 7.1%
  • 2.8%
  • 1.3%

Regression 0.615

  • 1.1%

+3.1% +2.5% LMART v1 ~0.621 no data +6.1% no data LMART unbiased 0.635 no data +8.1% (est) no data

43
slide-44
SLIDE 44

CONCLUSION

Better ranking = more $$$ A lot of pitfalls Multiply development estimates by

44
slide-45
SLIDE 45 45