SLIDE 13 Are marketing chefs satisfied with their recommenders?
Effectiveness criteria are not yet well defined or applied
L a c k
m e a s u r e m e n t m e t r i c s s y s t e m
“We measure the click-through and order conversion rate (of the recommended item) … but it seems not enough to find out where are the problems.”
Lack of New criteria other than relevance
“We improve algorithm and architecture... we think that user experience and trust is also important … but we don’t know how to measure them”
Good relevance but low click-through
“It’s good that quite a lot of consumers buy the items we recommended, but many of them didn’t click the recommendation link as we expected…”
G
c l i c k
h r
g h b u t p
c
v e r s i
“ O u r c l i c k
h r
g h r a t e i s a m a z i n g , b u t t h e
d e r c
v e r s i
r a t e i s f a r f r
s a t i s f a c t i
. . . m a n y p e
l e q u i t e t h e p a g e j u s t a f e w s e c
d s a f t e r … ”
A performing recommender with pure chance
“ W e t h
g h t
r r e c
m e n d e r w a s g r e a t , b u t w h e n w e r a n a n A / B t e s t , t h e c
t r
g r
p ( r e c
m e n d e r b a s e d
p u r e c h a n c e ) h a d t h e s i m i l a r p e r f
m a n c e ! ”
Effectiveness of multi-recommender system
“We put different kinds of recommenders across the shopping process, but it seems that our customers are very focused on what they plan to buy…”