Inferring Restaurant Styles by Mining Crowd Sourced Photos from - - PowerPoint PPT Presentation

▶
inferring restaurant styles by mining crowd sourced
SMART_READER_LITE
LIVE PREVIEW

Inferring Restaurant Styles by Mining Crowd Sourced Photos from - - PowerPoint PPT Presentation

May 14, 2023 355 likes •622 views

@IEEE BigData 2016 Inferring Restaurant Styles by Mining Crowd Sourced Photos from User-Review Websites Haofu Liao, Yuncheng Li, Tianran Hu and Jiebo Luo Department of Computer Science UNIVERSITY of ROCHESTER Part I - The Problem UNIVERSITY

slide-1
SLIDE 1

Inferring Restaurant Styles by Mining Crowd Sourced Photos from User-Review Websites

Haofu Liao, Yuncheng Li, Tianran Hu and Jiebo Luo Department of Computer Science

@IEEE BigData 2016 UNIVERSITY of ROCHESTER

slide-2
SLIDE 2

UNIVERSITY of ROCHESTER

Part I - The Problem

slide-3
SLIDE 3

Restaurant Style Classification

Business meetings Romantic Restaurant Photos UNIVERSITY of ROCHESTER

slide-4
SLIDE 4

Multi-Instance Multi-Label Learning (MIML)

UNIVERSITY of ROCHESTER

Instance Label

  • bject

Instance Label

  • bject

Label Label

...

Instance Label

  • bject

Instance Instance

...

Label

  • bject

Label Label

...

Instance Instance Instance

...

(c) Multi-label learning (a) Traditional Supervised learning (b) Multi-instance learning (d) Multi-instance multi-label learning

slide-5
SLIDE 5

Multi-Instance Multi-Label Learning (MIML)

UNIVERSITY of ROCHESTER

Business meetings

Restaurant

Special Occasion

Romantic

...

Photo 2 Photo 1 Photo N

...

  • Each restaurant (object) is described by a set
  • f photos (instances) and associated with

several class restaurant style tags (label).

  • Conventionally, MIML is based on the

assumption that there exists a “key” instance that contributes the object’s class labels.

  • For restaurant style classification, such

assumption is not guaranteed. For example, just one picture of a delicate dish does not mean the restaurant itself is romantic

slide-6
SLIDE 6

UNIVERSITY of ROCHESTER

Part II - Solution

slide-7
SLIDE 7

Proposed Architecture

UNIVERSITY of ROCHESTER

Label Initialization Multi-Label CNN Multi-Label CNN Pseudo Tagging Restaurant Profiling SVM SVM SVM

...

Business meetings Romantic

slide-8
SLIDE 8

Overall Architecture

UNIVERSITY of ROCHESTER

Label Initialization Multi-Label CNN Multi-Lacbel CNN Pseudo Tagging Restaurant Profiling SVM SVM SVM

...

Business meetings Romantic

slide-9
SLIDE 9

Multi-Label CNN

UNIVERSITY of ROCHESTER

CONV3 CONV2 CONV5 CONV1 CONV4 POOL3 POOL2 POOL5 FC6 FC7 FC8

Sigmoid Cross Entropy Loss

Data Label

AlexNet

  • Sigmoid Cross Entropy Loss
slide-10
SLIDE 10

Overall Architecture

UNIVERSITY of ROCHESTER

Label Initialization Multi-Label CNN Multi-Label CNN Pseudo Tagging Restaurant Profiling SVM SVM SVM

...

Business meetings Romantic

slide-11
SLIDE 11

Photo 6

  • 3.7

Photo 2

  • 5.4

Photo 1

7.1

Pseudo Tagging

UNIVERSITY of ROCHESTER

Photo 5

9.5

Photo 3

0.9

Photo 7

3.2

Photo 8

  • 0.2

Photo 4

  • 1.1

Images Scores of tag k Images from restaurants with tag k Images from restaurants without tag k

slide-12
SLIDE 12

Pseudo Tagging

UNIVERSITY of ROCHESTER

Photo 5 Photo 1

9.5

Photo 7 Photo 3

0.9 7.1 3.2

Photo 8 Photo 4

  • 0.2

Photo 6 Photo 2

  • 5.4
  • 1.1
  • 3.7

Ordered Steps: 1. Order images according their scores

slide-13
SLIDE 13

Pseudo Tagging

UNIVERSITY of ROCHESTER

Photo 5 Photo 1

9.5

Photo 7 Photo 3

0.9 7.1 3.2

Photo 8 Photo 4

  • 0.2

Photo 6 Photo 2

  • 5.4
  • 1.1
  • 3.7

Ordered Steps: 1. Order images according their scores 2. Add tag k to the image that has the highest score among the images that do not have tag k. Photo 1

slide-14
SLIDE 14

Pseudo Tagging

UNIVERSITY of ROCHESTER

Photo 5 Photo 1

9.5

Photo 7 Photo 3

0.9 7.1 3.2

Photo 8 Photo 4

  • 0.2

Photo 6 Photo 2

  • 5.4
  • 1.1
  • 3.7

Ordered Steps: 1. Order images according their scores 2. Add tag k to the image that has the highest score among the images that do not have tag k. 3. Drop tag k from the image that has the lowest score among the images that have tag k. Photo 1 Photo 2

slide-15
SLIDE 15

Pseudo Tagging

UNIVERSITY of ROCHESTER

Photo 5 Photo 1

9.5

Photo 7 Photo 3

0.9 7.1 3.2

Photo 8 Photo 4

  • 0.2

Photo 6 Photo 2

  • 5.4
  • 1.1
  • 3.7

Ordered Steps: 1. Order images according their scores 2. Add tag k to the image that has the highest score among the images that do not have tag k. 3. Drop tag k from the image that has the lowest score among the images that have tag k. Photo 1 Photo 2

slide-16
SLIDE 16

Pseudo Tagging

UNIVERSITY of ROCHESTER

Photo 5 Photo 1

9.5

Photo 7 Photo 3

0.9 7.1 3.2

Photo 8 Photo 4

  • 0.2

Photo 6 Photo 2

  • 5.4
  • 1.1
  • 3.7

Ordered Steps: 1. Order images according their scores 2. Add tag k to the image that has the highest score among the images that do not have tag k. 3. Drop tag k from the image that has the lowest score among the images that have tag k. 4. Repeat step 3 & step 4 until the scores of images reach a predefined threshold Photo 1 Photo 2

slide-17
SLIDE 17

Overall Architecture

UNIVERSITY of ROCHESTER

Label Initialization Multi-Label CNN Multi-Label CNN Pseudo Tagging Restaurant Profiling SVM SVM SVM

...

Business meetings Romantic

slide-18
SLIDE 18

Restaurant Profiling

UNIVERSITY of ROCHESTER

SVM Multi-Label CNN Features

Dining on a budget

slide-19
SLIDE 19

UNIVERSITY of ROCHESTER

Part III - Experimental Results

slide-20
SLIDE 20

UNIVERSITY of ROCHESTER

Top Scored Images

slide-21
SLIDE 21

UNIVERSITY of ROCHESTER

Distribution of Image Scores

1st Round Multi-Label CNN 2nd Round Multi-Label CNN

slide-22
SLIDE 22

UNIVERSITY of ROCHESTER

Overall Performance

slide-23
SLIDE 23

UNIVERSITY of ROCHESTER

F-Measure @m

slide-24
SLIDE 24

UNIVERSITY of ROCHESTER

Part IV - Questions?

slide-25
SLIDE 25

UNIVERSITY of ROCHESTER

Thank You!