NEIL: E XTRACTING V ISUAL K NOWLEDGE FROM W EB D ATA Xinlei Chen, - - PowerPoint PPT Presentation

▶
neil e xtracting v isual k nowledge from w eb d ata
SMART_READER_LITE
LIVE PREVIEW

NEIL: E XTRACTING V ISUAL K NOWLEDGE FROM W EB D ATA Xinlei Chen, - - PowerPoint PPT Presentation

Nov 09, 2022 45 likes •461 views

NEIL: E XTRACTING V ISUAL K NOWLEDGE FROM W EB D ATA Xinlei Chen, Abhinav Shrivastava, Abhinav Gupta Carnegie Mellon University Lots of labeled data & common-sense relationships have helped improve performance! . But how do we label

slide-1
SLIDE 1

NEIL: EXTRACTING VISUAL KNOWLEDGE

FROM WEB DATA

Xinlei Chen, Abhinav Shrivastava, Abhinav Gupta

Carnegie Mellon University

slide-2
SLIDE 2

[Everingham et al., IJCV’10], [Xiao et al., CVPR’10], [Deng et al., CVPR’09], [Gupta et al. ECCV’08], [Farhadi et al., CVPR’09], [Lampert et al., CVPR’09], [Parikh et al., ICCV’11], [Shrivastava et al., ECCV’12]

….

Lots of labeled data & common-sense relationships have helped improve performance!

But how do we label data and collect common sense at a large scale?

slide-3
SLIDE 3

Labeled Data: Common Sense Relationships:

slide-4
SLIDE 4

Labeled Data:

1M Boxes 5 years

2M Rules 30 years

and still continuing…

400M images daily!

?

Common Sense Relationships:

slide-5
SLIDE 5

NEIL

NEVER ENDING IMAGE LEARNER

Trying to understand images on the web and build world’s largest Visual Knowledge Base automatically…

Running 24 hours a day, 7 days a week

slide-6
SLIDE 6

NEIL’S KNOWLEDGE BASE

Concepts Relationships

slide-7
SLIDE 7

OBJECTS

Camry

slide-8
SLIDE 8

SCENES

Parking Lot Raceway

slide-9
SLIDE 9

ATTRIBUTES

Round Shape Crowded

slide-10
SLIDE 10

RELATIONSHIPS

Object - Object

Partonomy Taxonomy

  • r

Similarity

Wheel is a part of Car Corolla is a kind of/looks similar to of Car

slide-11
SLIDE 11

RELATIONSHIPS

Object - Scene

Car is found in Raceway

slide-12
SLIDE 12

RELATIONSHIPS

Object - Attribute

Wheel is/has Round shape

slide-13
SLIDE 13

RELATIONSHIPS

Scene – Attribute

Bamboo forest is/has Vertical lines

slide-14
SLIDE 14

NEIL’S KNOWLEDGE BASE

Concepts

  • Objects
  • Scenes
  • Attributes

Relationships

  • Object-Object

Partonomy, Taxonomy/Similarity

  • Object-Scene
  • Object-Attribute
  • Scene-Attribute

(Provided by Text Analysis)

slide-15
SLIDE 15

NEIL

NEVER ENDING IMAGE LEARNER

  • 4 months on 200 cores
  • >2500 Concepts
  • >1500 Objects
  • >1034 Scenes
  • >87 Attributes
  • >5 million images analyzed
  • Labeled
  • 600K images
  • 3000 relationships

www.neil-kb.com Train Your Own Concept!

slide-16
SLIDE 16
slide-17
SLIDE 17

NEIL

NEVER ENDING IMAGE LEARNER

  • 4 months on 200 cores
  • >2500 Concepts
  • >1500 Objects
  • >1034 Scenes
  • >87 Attributes
  • >5 million images analyzed
  • Labeled
  • 600K images
  • 3000 relationships

www.neil-kb.com

slide-18
SLIDE 18

WHY DOES NEIL WORK?

slide-19
SLIDE 19

MICRO-VISION

slide-20
SLIDE 20

MACRO-VISION

slide-21
SLIDE 21

Car is found on Street Sheep are White

STRUCTURE IN THE VISUAL WORLD

Constrained Semi-supervised Learning

slide-22
SLIDE 22

SEMANTICALLY-DRIVEN ACQUISITION

Yoda

[Fergus et al. ECCV’04], [Berg et al. CVPR’06], [Snavely et al. SIGGRAPH’06], [Schroff et al. ICCV’07], [Simon et

  • al. ICCV’07], [Hays et al., CVPR’08], [Li et al. ECCV’08], [Shrivastava et al. ToG’11], [Rubenstein et al. CVPR’13] ...
slide-23
SLIDE 23

HOW DOES NEIL WORK?

slide-24
SLIDE 24

(0) Seed Images

Desktop Computer Monitor Keyboard

  • 1. No Bounding-boxes
  • 2. Noise
  • 3. Multiple Meanings (Polysemy)
slide-25
SLIDE 25

(0) Seed Images

Desktop Computer Monitor Keyboard

(1) (2) (3)

Desktop Computer

(1) (2) (3)

Monitor

(1) (2) (3)

Keyboard

(1) (2) (3)

Television

(1) Subcategory Discovery

slide-26
SLIDE 26

EXEMPLAR DETECTORS

Car

slide-27
SLIDE 27

AFFINITY GRAPH

slide-28
SLIDE 28

POLYSEMY

Falcon

slide-29
SLIDE 29

(0) Seed Images

Desktop Computer Monitor Keyboard

(1) (2) (3)

Desktop Computer

(1) (2) (3)

Monitor

(1) (2) (3)

Keyboard

(1) (2) (3)

Television

Desktop Computer (1) Desktop Computer (2) Desktop Computer (3) … Monitor (1) …

(1) Subcategory Discovery (2) Train Models

slide-30
SLIDE 30

TRAIN MODELS

  • Latent SVM
  • Objects, Attributes
  • CHOG
  • Linear SVM
  • Scenes, Attributes
  • Color, Texton, HOG, SIFT, GIST
  • … Your model?
slide-31
SLIDE 31

(1) (2) (3)

Desktop Computer

(1) (2) (3)

Monitor

(1) (2) (3)

Keyboard

(1) (2) (3)

Television

Desktop Computer (1) Desktop Computer (2) Desktop Computer (3) … Monitor (1) …

(1) Subcategory Discovery (2) Train Models (3) Relationship Discovery (0) Seed Images

Desktop Computer Monitor Keyboard

slide-32
SLIDE 32

RELATIONSHIP DISCOVERY

  • Keyboard is a part of Desktop Computer
  • Monitor is a part of Desktop Computer
  • Television looks similar to Monitor

Learned relationships:

N Concepts N Concepts

Keyboard Desktop Computer

Keyboard Desktop Computer

Macro Vision

slide-33
SLIDE 33

(1) (2) (3)

Desktop Computer

(1) (2) (3)

Monitor

(1) (2) (3)

Keyboard

(1) (2) (3)

Television

  • Keyboard is a part of Desktop Computer
  • Monitor is a part of Desktop Computer
  • Television looks similar to Monitor

Learned relationships:

Desktop Computer (1) Desktop Computer (2) Desktop Computer (3) … Monitor (1) …

(1) Subcategory Discovery (2) Train Models (3) Relationship Discovery (0) Seed Images

Desktop Computer Monitor Keyboard

slide-34
SLIDE 34

(2) Retrain Models

(1) (2) (3)

Desktop Computer

(1) (2) (3)

Monitor

(1) (2) (3)

Keyboard

(1) (2) (3)

Television

Desktop Computer (1) Desktop Computer (2) Desktop Computer (3) … Monitor (1) …

(1) Subcategory Discovery (2) Train Models (3) Relationship Discovery

Desktop Computer Monitor Television

(4) Add New Instances (0) Seed Images

Desktop Computer Monitor Keyboard

  • Keyboard is a part of Desktop Computer
  • Monitor is a part of Desktop Computer
  • Television looks similar to Monitor

Learned relationships:

slide-35
SLIDE 35

RESULTS

Pink Nilgai Bean

slide-36
SLIDE 36

COMMON SENSE RELATIONSHIPS

Object - Object

Eye is a part of Baby Sparrow is a kind of/looks similar to bird

Object - Scene

Helicopter is found in Airfield Ferris wheel is found in Amusement park

slide-37
SLIDE 37

COMMON SENSE RELATIONSHIPS

slide-38
SLIDE 38

Scene Classification

mAP Seed Classifier (15 Google Images) 0.52 Bootstrapping (without relationships) 0.54 NEIL Scene Classifiers 0.57 NEIL (Classifiers + Relationships) 0.62

CAN NEIL HELP VISION TASKS?

slide-39
SLIDE 39

Object Detection

CAN NEIL HELP VISION TASKS?

mAP Latent SVM (450 Google Images) 0.28 Latent SVM (450, Aspect Ratio Clustering) 0.30 Latent SVM (450, HOG-based Clustering) 0.33 Seed Detector (NEIL Clustering) 0.44 Bootstrapping (without relationships) 0.45 NEIL Detector 0.49 NEIL Detector + Relationships 0.51

slide-40
SLIDE 40

NEIL

NEVER ENDING IMAGE LEARNER

Forever to Label data, Learn relationships

Running 24 hours a day, 7 days a week

slide-41
SLIDE 41

THANK YOU!

Opera house is found in Sydney All Models and Relationships will be found* on www.neil-kb.com

*20 Dec, 2013