Neural Outfit Recommendation DAPA Workshop @ WSDM 2019 Maarten de - - PowerPoint PPT Presentation

Neural Outfit Recommendation

DAPA Workshop @ WSDM 2019

Maarten de Rijke February 15, 2019

University of Amsterdam derijke@uva.nl

Based on joint work with Jun Ma, Pengjie Ren, Yujie Lin, Zhaochun Ren, and Zhumin Chen

1

Background Outfit recommendation Fashion recommendation machine Some results Conclusion

2

Neural IR

Big uptake and injection of energy in the field

• Learning to match
• Learning to rank
• Content understanding – text, image, video, . . .
• Behavior understanding
• . . .

3

The need to take stock, repeatedly

Quickly building up a rich body of knowledge

• Li and Xu (2013) – Semantic matching in search
• Onal et al. (2018) – Neural information retrieval: At the end of the early years
• Mitra and Craswell (2019) – An introduction to neural information retrieval
• Li et al. (20XX) – . . .

4

Rough edges

Lin (2018) – The Neural Hype and Comparisons Against Weak Baselines

• Everyone is trying to win
• “demonstrating that a new method beats previous methods on a given task or

benchmark”

• Often, our baselines are weak

5

Rough edges

How to improve ourselves

• Compare apples to apples
• Work on insights – reasons for success, reasons for failure
• Use reference baselines

6

Rough edges

How to improve ourselves

• Compare apples to apples
• Work on insights – reasons for success, reasons for failure
• Use reference baselines
• Share everything
• Use reference implementations
• Engage with product owners for additional eyes and checks
• Win in different ways – task, constraints, metrics, . . .

6

7

Background Outfit recommendation Fashion recommendation machine Some results Conclusion

8

Outfit recommendation

A different task, with a twist Fashion recommendation – increased attention Outfit recommendation – given a top (i.e., upper garment), recommend a list of bottoms (e.g., trousers or skirts) from a large collection that best match the top, and vice versa

• Allow users to provide some descriptions as conditions that the recommended

items should accord with as much as possible

9

Unpacking the task

Two main challenges

• visual understanding – aims to extract effective visual features
• visual matching – aims to model a human notion of compatibility to compute a

match between fashion items

10

Unpacking the task

Two main challenges

• visual understanding – aims to extract effective visual features
• visual matching – aims to model a human notion of compatibility to compute a

match between fashion items Typically, visual understanding and matching conducted based on recommendation loss alone

• Supervision signal is just whether two given items are matched or not and no

supervision is available to directly connect the visual signals of the fashion items

• Can we come up with a sense of esthetics?

10

Background Outfit recommendation Fashion recommendation machine Some results Conclusion

11

Fashion recommendation machine

Lin et al. (2019) – Improving Outfit Recommendation with Co-supervision of Fashion Generation

1 Neural co-supervision learning framework, FARM, for outfit recommendation that

simultaneously yields recommendation and generation

2 Layer-to-layer matching mechanism as a bridge between generation and

recommendation – improves recommendation by leveraging generation features

12

FARM architecture

13

FARM architecture

For the fashion generator

• Use CNN as top encoder to extract visual features from top image It
• Learn semantic representation for bag-of-words vector d of bottom description
• Use variational transformer to learn mapping from bottom distribution to Gaussian

distribution based on visual features of It and semantic representation of d

• Sample a random vector from Gaussian distribution and input it to a DCNN (as

bottom generator) to generate bottom image Ig that matches It and d

• Explicitly forces top encoder to encode more aesthetic matching information into

visual features

14

FARM architecture

For the fashion recommender

• Also employs CNN as bottom encoder to extract visual features from candidate

bottom image Ib

• Evaluate matching score between Ib and (It, d) pair from three angles

1 Visual matching between Ib and It 2 Description matching between Ib and d 3 Layer-to-layer matching between Ib and Ig, which leverages generation information

to improve recommendation

15

FARM architecture

FARM jointly trains the fashion generator and fashion recommender Three types of loss

1 Generation loss (visual + textual) 2 Loss based on ELBO 3 Recommendation loss (like BPR) 16

Background Outfit recommendation Fashion recommendation machine Some results Conclusion

17

A sample of results

FashionVC and ExpFashion datasets sampled from Polyvore online community 4-tuples (top, top description, bottom, bottom description)

18

Bake-off

19

Co-supervision learning

20

Layer-to-layer

21

Some samples: Real vs generated

22

Some samples: Recommendations

23

Some samples: Real vs generated

24

Background Outfit recommendation Fashion recommendation machine Some results Conclusion

25

What have we done?

Outfit recommendation

• Visual understanding
• Visual matching

Proposed a co-supervision learning framework, FARM

• For visual understanding, FARM captures more aesthetic characteristics with

supervision of generation learning

• For visual matching, FARM incorporates layer-to-layer matching mechanism to

evaluate matching score of candidate and generated items at different neural layers

26

What should we do next?

Effectiveness of generated images to explain the recommendations? Improvement in quality of generated images leads to improvement in recommendations? How to recommend complete outfits?

27

Playing the winning game

How to improve ourselves

• Compare apples to apples
• Work on insights – reasons for success, reasons for failure
• Use reference baselines
• Share everything
• Use reference implementations
• Engage with product owners for additional eyes and checks
• Win in different ways – task, constraints, metrics, . . .

28

References i

• H. Li and J. Xu. Semantic matching in search. Foundations and Trends in Information Retrieval, 7(5):343–469,

2013.

• J. Lin. The neural hype and comparisons against weak baselines. SIGIR Forum, 52(2):40–51, 2018.
• Y. Lin, P. Ren, Z. Chen, Z. Ren, J. Ma, and M. de Rijke. Improving outfit recommendation with co-supervision
• f fashion generation. In The Web Conference 2019, May 2019.
• B. Mitra and N. Craswell. An introduction to neural information retrieval. Foundations and Trends in

Information Retrieval, 13(1), January 2019.

• K. D. Onal, Y. Zhang, I. S. Altingovde, M. M. Rahman, P. Karagoz, A. Braylan, B. Dang, H.-L. Chang,
• H. Kim, Q. McNamara, A. Angert, E. Banner, V. Khetan, T. McDonnell, A. T. Nguyen, D. Xu, B. C.

Wallace, M. de Rijke, and M. Lease. Neural information retrieval: At the end of the early years. Information Retrieval Journal, 21(2–3):111–182, June 2018.

Acknowledgments

All content represents the opinion of the author(s), which is not necessarily shared or endorsed by their employers and/or sponsors.