Bootstrapping Food Preferences Through an Adaptive Visual Interface - - PowerPoint PPT Presentation

Bootstrapping Food Preferences Through an Adaptive Visual Interface

Longqi Yang, Yin Cui, Fan Zhang, JP Pollak, Serge Belongie, Deborah Estrin

MOTIVATION

Food preferences learning is important!

Health and Life

*Number of Americans Living with Diet-and Inactivity-Related Diseases

Unflavored Healthy diet recommendations are of NO Benefit!

Obesity HBP Diabetes

113M 50M 15M

Social Media and Commerce

Personalized diet profile is the Key to user experience!

Our Vision

Clinicians

Personalize Treatment Plan

Online Groceries

Customers Targeting

Recipes

Food environment at home

Restaurants

Customized dishes

Social Network

Content personalization

Nutritionists

Healthy recommendations Personalized Diet Profile

OUR SOLUTION

An adaptive visual interface

Start Diet Profile

Exploration, 2 iters

10 food items

Exploration-exploitation: <15 iters

Pairwise Comparison

ü Efficient: completed within a minute. ü Visual interface: low cognitive load, personalized and legible. ü Preference Elicitation: NO history required, NO ratings. ü Deep understanding of food images. ü Novel Online Learning Framework.

System Design

Pr Pre-trai aine ned d Deep ep Siamese se Ne Networ

• rk

Ingr gred edient ents 1000 dim 200 dim Image raw pixels Metadata Food Items Harvesting Ingredients Nutrients …… Us User er Pref efer eren ence Food

• d alread

eady expl plor

• red

ed Backend Online Learning

Explo lora rati tion

• n (2

2 iter eratio tions) Explo lora rati tion

• n – Explo

loita itati tion

• n (>2)

# Iterations

……

Food Similarity Embedding

Yuc uck Yuc uck

Visual User Interface

• nline
• ffline

Online Learning Ø What images to present to the user? Ø How to update users’ preferences? Online Learning framework (LE + EE) Food Similarity Embedding Users have close preferences for similar items Ø Feature representation that can reflect similarities Food Items Harvesting Ø Food images and metadata.

System Design: offline

Pr Pre-trai aine ned d Deep ep Siamese se Ne Networ

• rk

Ingr gred edient ents 1000 dim 200 dim Image raw pixels Metadata Food Items Harvesting Ingredients Nutrients …… Us User er Pref efer eren ence Food

• d alread

eady expl plor

• red

ed Backend Online Learning

Explo lora rati tion

• n (2

2 iter eratio tions) Explo lora rati tion

• n – Explo

loita itati tion

• n (>2)

# Iterations

……

Food Similarity Embedding

Yuc uck Yuc uck

Visual User Interface

• nline
• ffline

Food Items Harvesting

Ø 12,000 food items from Yummly API. Ø Images + Metadata (ingredients, nutrients etc.) Ø Outliers filtering, 10,028 items were used.

System Design: offline

Pr Pre-trai aine ned d Deep ep Siamese se Ne Networ

• rk

Ingr gred edient ents 1000 dim 200 dim Image raw pixels Metadata Food Items Harvesting Ingredients Nutrients …… Us User er Pref efer eren ence Food

• d alread

eady expl plor

• red

ed Backend Online Learning

Explo lora rati tion

• n (2

2 iter eratio tions) Explo lora rati tion

• n – Explo

loita itati tion

• n (>2)

# Iterations

……

Food Similarity Embedding

Yuc uck Yuc uck

Visual User Interface

• nline
• ffline

Food Similarity Embedding

Representation: 1000 dim visual + 200 dim ingredients 1000 dim visual feature from Food-CNN Image 2 Image 1 x y

A (CNN) B (CNN)

Contrastive Loss f(x) f(y)

System Design: offline

Pr Pre-trai aine ned d Deep ep Siamese se Ne Networ

• rk

Ingr gred edient ents 1000 dim 200 dim Image raw pixels Metadata Food Items Harvesting Ingredients Nutrients …… Us User er Pref efer eren ence Food

• d alread

eady expl plor

• red

ed Backend Online Learning

Explo lora rati tion

• n (2

2 iter eratio tions) Explo lora rati tion

• n – Explo

loita itati tion

• n (>2)

# Iterations

……

Food Similarity Embedding

Yuc uck Yuc uck

Visual User Interface

• nline
• ffline

Food Similarity Embedding

Representation: 1000 dim visual + 200 dim ingredients 1000 dim visual feature from Food-CNN

− ≈ 0 − > 𝑛 , , 𝓜 = 𝟐 𝟑𝒎𝑬𝟑 + 𝟐 𝟑 𝟐 − 𝒎 𝐧𝐛𝐲 (𝟏, 𝒏 − 𝑬) 𝟑 𝒎 = 𝟐 𝒎 = 𝟏

Pairs/Labels were sampled from Food-101 dataset

System Design: offline

Pr Pre-trai aine ned d Deep ep Siamese se Ne Networ

• rk

Ingr gred edient ents 1000 dim 200 dim Image raw pixels Metadata Food Items Harvesting Ingredients Nutrients …… Us User er Pref efer eren ence Food

• d alread

eady expl plor

• red

ed Backend Online Learning

Explo lora rati tion

• n (2

2 iter eratio tions) Explo lora rati tion

• n – Explo

loita itati tion

• n (>2)

# Iterations

……

Food Similarity Embedding

Yuc uck Yuc uck

Visual User Interface

• nline
• ffline

Food Similarity Embedding

Representation: 1000 dim visual + 200 dim ingredients 200 dim ingredients feature Ø Lemmatization and preprocessing. Ø Filtering: Top 200 ingredients. Ø Feature vector: 0-1 vector denotes the existence of the ingredient. Visual and ingredients feature vectors are normalized separately with 𝒎𝟐 norm

System Design: online

Pr Pre-trai aine ned d Deep ep Siamese se Ne Networ

• rk

Ingr gred edient ents 1000 dim 200 dim Image raw pixels Metadata Food Items Harvesting Ingredients Nutrients …… Us User er Pref efer eren ence Food

• d alread

eady expl plor

• red

ed Backend Online Learning

Explo lora rati tion

• n (2

2 iter eratio tions) Explo lora rati tion

• n – Explo

loita itati tion

• n (>2)

# Iterations

……

Food Similarity Embedding

Yuc uck Yuc uck

Visual User Interface

• nline
• ffline

Online Learning

Food preferences representation:

𝒒 = 𝑞8, 𝑞9, … ,𝑞 𝒯 <𝑞=

=

= 1

Distribution of preferences over all food items in 𝒯

t t t t t

𝒒?:updated preference vector after iteration t

Two tasks at each iteration t: Ø User state update: update 𝒒? based on the items presented and user’s choices at iteration t-1. Ø Images selection: Select a set of images to show at iteration t.

System Design: online

Online Learning

Ø User state update: update 𝒒? based on the items presented and user’s choices at iteration t-1. Users’ selections Image Labeling Images selected Label “+1” Label “-1” Images not selected Images not presented Label “0”

System Design: online

Online Learning Label propagation with regularized optimization

min

𝒗

< 𝜕=E 𝑧= − 𝑣

E H 𝒯 EI9,EJ=

+ < 1 − 𝜕=E 𝑣

E − 𝑧E H 𝒯 EI9,EJ=

Smoothness Fitting

Label Propagation and Exponentiated Gradient Algorithm (LE)

𝑣

E = < 𝜕=E𝑧= 𝒯 =I9

𝑞=

?

𝑞=

?K9×𝑓 NOP

QRS

TP

QRS

𝜕=E = 𝑓

K9 HUV 𝒈XPK𝒈XY

Ø User state update: update 𝒒? based on the items presented and user’s choices at iteration t-1.

System Design: online

Online Learning

Ø Images selection: Select a set of images to show at iteration t. Exploration and Exploration-exploitation Algorithm (EE) Exploration (Ten images): 𝑢 ≤ 2 K-means++ Exploration-exploitation (Two images): 𝑢 > 2 One Item that user “prefer” (with high value of 𝑞) The other item that user hasn’t explored.

System Design: online

Online Learning

user state update images selection

EXPERIMENTS AND USER STUDY

Evaluation, findings and evidence

Experiments: embedding

0.0 0.2 0.4 0.6 0.8 1.0

5eFall

10-2 10-1 100

PreFLsLRn(LRgarLWhPLF 6Fale)

)RRG-C11 (PAP: 0.216) Alex1eW (PAP: 0.051) 6I)T+BR: (PAP: 0.019) 5anGRP Guess (PAP: 0.01)

Clustering performance of Food-CNN (Tested on Food-101 dataset). Ø K-neighbors of each test image, calculate the precision-recall for each K

Experiments: user study

Ø 227 anonymous users. Ø Two factors were controlled in the study. Online Perceptron (OP) Label Propagation and Exponentiated Gradient (LE) Exploration and Exploration- exploitation (EE) Random Selection (RS)

• 1st. Algorithm:
• 2nd. Number of iterations: 5/10/15

Experiments: user study

Ø Algorithm to test: LE+EE

One image from top 1% of preference value. (unexplored) The other image from bottom 1% of preference value. (unexplored)

Exploration Exploration-exploitation PlateClick (10 iters) Testing (10 iters)

Ø Trials: 1/3

Experiments: user study

One image from top 1% of preference value. (unexplored) The other image from bottom 1% of preference value. (unexplored)

Exploration Exploration-exploitation PlateClick (5 iters) Testing (10 iters)

Ø Algorithm to test: LE+EE Ø Trials: 2/3

Experiments: user study

One image from top 1% of preference value. (unexplored) The other image from bottom 1% of preference value. (unexplored)

Exploration Exploration-exploitation PlateClick (15 iters) Testing (10 iters)

Ø Algorithm to test: LE+EE Ø Trials: 3/3

Experiments: user study

* * ** ** *** *** * **

Prediction accuracy under different algorithms and number of iterations

Experiments: user study

0.0 0.2 0.4 0.6 0.8 1.0

PredLctLon AccurDcy

0.0 0.2 0.4 0.6 0.8 1.0

CuPulDtLve DLstrLEutLon

LE+EE:5 LE+EE:10 LE+EE:15

Cumulative distribution of prediction accuracy for LE+EE algorithm

Conclusions and Future work

Ø Engine for food preferences learning. Ø Applicable to general human-in-the-loop problems.

For more information:

http://www.cs.cornell.edu/~ylongqi

Try it out online:

http://bit.ly/plateclick http://smalldata.io/ @ylongqi ylongqi@cs.cornell.edu