E-COMMERCE SEARCH: APPLICATIONS AND REQUIREMENTS OTTO @ MICES 2019 - - PowerPoint PPT Presentation

e commerce search
SMART_READER_LITE
LIVE PREVIEW

E-COMMERCE SEARCH: APPLICATIONS AND REQUIREMENTS OTTO @ MICES 2019 - - PowerPoint PPT Presentation

OFFLINE EVALUATION IN E-COMMERCE SEARCH: APPLICATIONS AND REQUIREMENTS OTTO @ MICES 2019 // Berlin 24.06.2019 1 About OTTO and otto.de Founded in 1949 On average 1.6 million visits on otto.de per day Number of employees


slide-1
SLIDE 1

OTTO @ MICES 2019 // Berlin 1

OFFLINE EVALUATION IN E-COMMERCE SEARCH: APPLICATIONS AND REQUIREMENTS

24.06.2019

slide-2
SLIDE 2

About OTTO and otto.de

24.06.2019 2

OTTO‘s headquarter in Hamburg

▪ Founded in 1949 ▪ Number of employees 4,900 ▪ Revenue in 2018/19 3.2 billion Euro

OTTO @ MICES 2019// Berlin

▪ On average 1.6 million visits on otto.de per day ▪ Up to 10 ordersper second ▪ More than 3 million items on otto.de ▪ More than 400 OTTO market partners ▪

  • Approx. 6,800 brands on otto.de

▪ Expansion of the business model towards becoming a marketplace

slide-3
SLIDE 3

About Us

24.06.2019 OTTO @ MICES 2019// Berlin

Jens Kürsten Tech Lead Search @otto.de Andreas Wagenmann Software Developer Search @otto.de

slide-4
SLIDE 4

About Our Product Search @otto.de in 2018

24.06.2019 4

Ø search queries per day total search queries

  • max. search queries per day

unique search terms

~0.9 million ~320 million ~3 million ~40 million

OTTO @ MICES 2019// Berlin

slide-5
SLIDE 5

What Is the Business Impact of Our Queries?

24.06.2019 5 OTTO @ MICES 2019// Berlin

Top Searches: ~500 Queries (~0.01%); ~25% Search Traffic; ~20% Sales Frequent Searches: ~30,000 Queries (~0.6%); ~45% Search Traffic; ~30% Sales Rare Searches: ~5,000,000 Queries (~99%); ~30% Search Traffic; ~50% Sales

slide-6
SLIDE 6

Finding the balance between the user intent and the business perspective is a challenge.

Key Requirements for Search @otto.de

24.06.2019 6

!

OTTO @ MICES 2019// Berlin

BUSINESS

USER

Relevance @otto.de is determined by

  • user queries & product data (quality)
  • performance indicators of our products
  • category-specific business goals
  • user interaction data
slide-7
SLIDE 7

OTTO @ MICES 2019 // Berlin 7

OFFLINE EVALUATION SETUP

24.06.2019

slide-8
SLIDE 8

Collecting Data

24.06.2019 8 OTTO @ MICES 2019// Berlin

Hose

1x

… … queries (with hits) clicks after search (with positions) judgements: query/product/ score

slide-9
SLIDE 9

Use Cases & Configurations

24.06.2019 9 OTTO @ MICES 2019// Berlin

queries frequencies product clusters some mo‘ tail ranking evaluation selection diff

slide-10
SLIDE 10

Offline Evaluation Architecture

24.06.2019 10 OTTO @ MICES 2019// Berlin

queries hits configs metrics # queries # clicks per product (in time slices) query judgement & score pairs (optionallysampled)

slide-11
SLIDE 11

OTTO @ MICES 2019 // Berlin 11

USE CASE 1: PRODUCT DATA CHANGES

24.06.2019

slide-12
SLIDE 12

Structural Changes in Indexed Data

24.06.2019 12 OTTO @ MICES 2019// Berlin

HitCount differences Subsamples 20,000 queries Index A Index B

?

&explain=true /terms

slide-13
SLIDE 13

Quality Metrics: Overview & Examples

24.06.2019 13 OTTO @ MICES 2019// Berlin

Relative Metric Changes

“Schlafzimmmer komplett mit boxspringbett” (missing: “komplettschlafzimm”): → 1 → 0 hits “popsocket” (missing: “popsocket”): → 88 → 39 hits, → P@4 +33%, P@All +56 %, NDCG@10: +11 %, Avg.Precision: +17 % “oberteil damen” (missing: “satinblus”): → 27558 → 27323 hits → P@4: +/- 0%, P@All: +1.9 %, NDCG: +/- 0 %, Recall: +1.9 %, Avg.Precision: - 2.9 %

slide-14
SLIDE 14

Quality Metrics: Subsamples

24.06.2019 14 OTTO @ MICES 2019// Berlin

More/less hits refers to the new configuration

Relative Metric Changes (more hits; n=2339) Relative Metric Changes (less hits; n=654) Relative Metric Changes (same hits; n=5191)

slide-15
SLIDE 15

Result Visualization & Interpretation

24.06.2019 15 OTTO @ MICES 2019// Berlin

Metric deltas: negative values → new configuration is better It looks like a draw → confirmed by on-site A/B test

slide-16
SLIDE 16

OTTO @ MICES 2019 // Berlin 16

USE CASE 2: THE PURSUIT OF PRECISION

24.06.2019

slide-17
SLIDE 17

Perpetual Challenge: Precision vs. Recall

24.06.2019 17 OTTO @ MICES 2019// Berlin

slide-18
SLIDE 18

Topical Relevance vs. Business Value

24.06.2019 18 OTTO @ MICES 2019// Berlin

10 20 30 40 50 60 70

Impact Rank Position

Business Value vs. Relevance– Query „TV“

Business Value Relevance

slide-19
SLIDE 19

Topical Relevance vs. Business Value

24.06.2019 19 OTTO @ MICES 2019// Berlin

10 20 30 40 50 60 70

Impact Rank Position

Business Value vs. Relevance- Query "belt"

Business Value Relevance

slide-20
SLIDE 20

Interaction-based Precision Improvement

24.06.2019 OTTO @ MICES 2019// Berlin 20

clicks & orders product attribute values for relevance search term & product performance filtered search results

slide-21
SLIDE 21

Precision Improvement PoC: Offline Results

24.06.2019 21 OTTO @ MICES 2019// Berlin

Configuration for Offline Evaluation Uplift P@4 Uplift P@30 Uplift P@100 Uplift AP@30 Uplift AP@100 % of changed traffic % of changed queries

  • Avg. difference
  • f hits

product_ci_producttype-clicks-cov90 4,22% 5,76% 13,33% 5,76% 10,14% 86,61% 63,37% 31,19% product_ci_producttype-clicks-cov95 4,19% 5,58% 12,87% 5,64% 9,89% 86,65% 63,38% 30,56% product_ci_category-clicks-cov90 2,66% 4,21% 8,76% 3,79% 6,48% 88,35% 84,12% 14,35% product_ci_category-clicks-cov95 2,62% 4,12% 8,50% 3,72% 6,31% 88,37% 84,14% 13,82% product_ci_producttype-a2b-cov90 1,84% 2,82% 7,40% 2,81% 5,45% 74,54% 19,69% 32,56% product_ci_producttype-a2b-cov95 1,84% 2,78% 7,33% 2,78% 5,41% 74,86% 19,69% 32,11% product_ci_category-a2b-cov90 1,13% 1,84% 4,74% 1,69% 3,34% 79,86% 29,84% 14,96% product_ci_category-a2b-cov95 1,16% 1,84% 4,72% 1,69% 3,33% 79,86% 29,84% 14,70% product_ci_assortmentsearch-clicks-cov90 0,86% 1,67% 4,20% 1,53% 3,12% 88,97% 87,63% 8,02% product_ci_assortmentsearch-clicks-cov95 0,85% 1,63% 4,06% 1,50% 3,02% 88,98% 87,63% 7,76% product_ci_assortmentsearch-a2b-cov90 0,47% 0,90% 2,44% 0,82% 1,78% 80,71% 31,63% 7,54% product_ci_assortmentsearch-a2b-cov95 0,48% 0,89% 2,43% 0,81% 1,77% 80,71% 31,63% 7,43%

slide-22
SLIDE 22

OTTO @ MICES 2019 // Berlin 22

OUR LEARNINGS FROM CONTINUOUS OFFLINE EVALUATION

24.06.2019

slide-23
SLIDE 23

Limitations of Click-based Judgements

24.06.2019 23 OTTO @ MICES 2019// Berlin

Top Searches: ~500 Queries (~0.01%); ~25% Search Traffic; ~20% Sales Frequent Searches: ~30,000 Queries (~0.6%); ~45% Search Traffic; ~30% Sales Rare Searches: ~5,000,000 Queries (~99%); ~30% Search Traffic; ~50% Sales Rare Searches: ~5,000,000 Queries (~99%); ~30% Search Traffic; ~50% Sales

slide-24
SLIDE 24

Data Quality & Comparability Challenges

24.06.2019 24 OTTO @ MICES 2019// Berlin

? ? ?

slide-25
SLIDE 25

Query Log Segmentation & Judgement Features

24.06.2019 25 OTTO @ MICES 2019// Berlin

slide-26
SLIDE 26

OTTO @ MICES 2019 // Berlin 26

SUMMARY

24.06.2019

slide-27
SLIDE 27

27 24.06.2019

Connect with us.

jens.kuersten@otto.de @faultfinder80 andreas.wagenmann@otto.de @andiwagen

We are hiring.

OTTO @ MICES 2019 // Berlin