Shifting from Naming to Describing: Semantic Attribute Models - - PowerPoint PPT Presentation

▶
shifting from naming to describing semantic attribute
SMART_READER_LITE
LIVE PREVIEW

Shifting from Naming to Describing: Semantic Attribute Models - - PowerPoint PPT Presentation

Sep 11, 2022 151 likes •569 views

Shifting from Naming to Describing: Semantic Attribute Models Rogerio Feris, June 2014 Recap Large-Scale Semantic Modeling Feature Coding and Pooling Low-Level Feature Extraction Training Data Slide credit: Rogerio Feris Learning Visual

slide-1
SLIDE 1

Shifting from Naming to Describing: Semantic Attribute Models

Rogerio Feris, June 2014

slide-2
SLIDE 2

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Recap

Training Data Low-Level Feature Extraction Feature Coding and Pooling Large-Scale Semantic Modeling

Slide credit: Rogerio Feris

slide-3
SLIDE 3

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

What if no training samples are available for the target class? Is this a practical setting?

Slide credit: Rogerio Feris

slide-4
SLIDE 4

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Motivation

ImageNet has 30 mushroom synsets, each with ≈1000 images.

Slide credit: Christoph Lampert

slide-5
SLIDE 5

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Motivation

In nature, there are ≈14,000 mushroom species.

Slide adapted from Christoph Lampert Image: http://www.evogeneao.com/

  • Zero-data: Many fine-grained visual categorization tasks may

have classes with few or no training examples at all.

slide-6
SLIDE 6

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Motivation

Slide credit: Rogerio Feris

Suspect Search in Surveillance Videos

[Feris et al, IBM]

  • Zero-data: often no example images from suspects are

available, only textual descriptions.

slide-7
SLIDE 7

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Motivation

Slide credit: Rogerio Feris

Prediction of concrete nouns from neural imaging data (mind reading) [Mark Palatucci et al, NIPS 2009]

Noun Prediction

  • Zero-Data: many nouns without corresponding neural image

examples (costly label acquisition)

slide-8
SLIDE 8

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Motivation

Slide credit: Rogerio Feris

Similar problems in other fields:

  • Zero-Data: Infeasible to acquire training samples for

each word (need sub-word modeling like phonemes)

Large Vocabulary Speech Recognition

  • Zero-Data: Newly released apps without any user

ratings (also known as “cold-start problem”) [Schin et al, SIGIR 2002]

Recommendation Systems

slide-9
SLIDE 9

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Semantic Attribute Models: Zero-Shot Learning for Visual Recognition

[Lampert et al, CVPR 2009] [Farhadi et al, CVPR 2009] [Palatucci et al, NIPS 2009]

slide-10
SLIDE 10

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Attribute-based Classification

Slide adapted from Christoph Lampert

Attributes:

  • Semantic/nameable

properties that are shared across classes

  • Intuitive mid-level feature

representation

slide-11
SLIDE 11

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Attribute-based Classification

Unseen categories Unseen categories

Semantic Attribute Classifiers

Standard multi-class classification Attribute-based classification [Lampert et al, CVPR 2009]

Slide credit: Rogerio Feris

Semantic Output Code Classifier (SOCC)

[Palatucci et al, NIPS 2009]

Similar to Error-Correcting Output codes (ECOC [Dietterich & Bakiri, 1995]), but semantic codes are used instead

slide-12
SLIDE 12

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Image-Attributes Prediction

Slide credit: Rogerio Feris

  • For each attribute , collect a set of positive and

negative samples and train a classifier (e.g., using SVM or Neural networks) Positive (Stripe) Negative (Non-Stripe) Binary Attribute Model Example: “Stripe” Attribute Attributes transcend class boundaries

Learning “stripe” attribute with images of zebras, clothing, …

slide-13
SLIDE 13

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Image-Attributes Prediction

[Parikh and Grauman, ICCV 2011]

Issue with Binary Attribute Models

Smiling Not smiling ??? Natural Not natural ???

slide-14
SLIDE 14

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Image-Attributes Prediction

Max-margin learning to rank formulation of Joachims 2002

i j i j

Relative Attributes

  • Replace binary model by a ranking function

[Parikh and Grauman, ICCV 2011]

slide-15
SLIDE 15

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Attribute-Class Associations

Manual Specification of Class-Attribute Associations

slide-16
SLIDE 16

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Attribute-Class Associations

Associations may be extracted automatically from other sources

[Rohrbach et al, CVPR 2010]

slide-17
SLIDE 17

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Attributes as “classes”

[Rohrbach et al, CVPR 2010] [Felix Yu et al, CVPR 2013] [Mensink et al, CVPR 2014]

Attribute-based Direct similarity

“giant pandas are similar to grizzly and polar bears”

slide-18
SLIDE 18

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Generalization: Label Embedding

[Akata et al , CVPR 2013] Check talk by Florent Perronnin on “Output embedding for large-scale visual recognition” (LSVR CVPR 2014 tutorial)

slide-19
SLIDE 19

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Generalization: Label Embedding

Frome et al . "DeViSE: A Deep Visual-Semantic Embedding Model", NIPS 2013

Label Embedding Framework

Automatic Discovery of word associations

Label Image Real-Value word vector representation Skip-gram model: Semantically related words are mapped to similar vector representations

Deep Learning

slide-20
SLIDE 20

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Generalization: Label Embedding

Language Model Source Code: https://code.google.com/p/word2vec/

Zero-Shot Learning / Semantically close mistakes

Label Embedding Framework

Automatic Discovery of word associations [Frome et al, NIPS 2013]

slide-21
SLIDE 21

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

In addition to zero-shot classification, semantic attribute models have shown to be useful for many other tasks

slide-22
SLIDE 22

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014 Slide credit: Rogerio Feris

Other Uses of Semantic Attributes

Check the CVPR 2013 tutorial on Attributes: https://filebox.ece.vt.edu/~parikh/attributes/

slide-23
SLIDE 23

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Attribute-based Search Application: Smart Surveillance

[Feris et al, IBM - WACV 2009, CVPR 2011, ICMR 2014]

slide-24
SLIDE 24

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014 Slide credit: Rogerio Feris

Attribute-based People Search

http://www.today.com/video/today/51630165/#51630165

slide-25
SLIDE 25

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014 Slide credit: Rogerio Feris

Attribute-based People Search

People Search in Surveillance Videos

Traditional Approaches: Face Recognition (“Naming”)

  • Face recognition is very challenging under lighting changes, pose variation, and low-

resolution imagery (typical conditions in surveillance scenarios).

Attribute-based People Search (“Describing”)

  • Rather than relying on face recognition only, we provide a complementary people

search framework based on fine-grained semantic attributes.

Query Example: “Show me all people with a beard and sunglasses, wearing a white hat and a patterned blue shirt, from all metro cameras in the downtown area, from 2pm to 4pm last Saturday".

slide-26
SLIDE 26

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014 Slide credit: Rogerio Feris

Attribute-based People Search

Suspect Description Form

slide-27
SLIDE 27

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014 Slide credit: Rogerio Feris

Attribute-based People Search

System Architecture

slide-28
SLIDE 28

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014 Slide credit: Rogerio Feris

[Siddiquie et al, CVPR 2011]

  • Facial Attributes: bald, hair, color of hair, hat, color of hat,

sunglasses, eyeglasses, absence of glasses, beard, mustache, absence of facial hair, skin tone (dark, medium,light), gender, …

  • Torso Attributes: clothing color, patterned, solid, …
  • Timestamp, Camera ID

Attribute-based People Search

slide-29
SLIDE 29

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014 Slide credit: Rogerio Feris

Attribute-based People Search

Attribute Ranking [Siddiquie, Feris and Davis, CVPR 2011]

 “Learning to rank”- confidence of individual attributes as features  Pairwise attribute modeling

slide-30
SLIDE 30

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014 Slide credit: Rogerio Feris

Structured Learning Formulation

Improved performance over other ranking methods (RankSVM, RankBoost, DORM, TagProp) in three standard datasets (LFW, FaceTracer, PASCAL)

See [Siddiquie, Feris and Davis, CVPR 2011]

slide-31
SLIDE 31

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014 Slide credit: Rogerio Feris

Attribute-based People Search

Top-1 Ranking Results

[Feris et al, ICMR 2014]

slide-32
SLIDE 32

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014 Slide credit: Rogerio Feris

Boston Bombing Event

 “Show me all images of people matching the suspect description from time X to time Y from all cameras in area Z.”

Ability to spot a person with e.g., a white hat in a crowded scene

Suspect #1 found in 4 images in top 8 results Suspect #2 found in 3 images in top page

 1071 detected faces from 50 high-res Boston images (all from Flickr)

slide-33
SLIDE 33

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014 Slide credit: Rogerio Feris

Extension to Vehicle Search

 “Show me all blue trucks larger than 7ft length traveling at high speed northbound last Saturday, from 2pm to 5pm.” [Feris et al, IEEE Trans on Multimedia, 2012]

slide-34
SLIDE 34

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Attribute-based Search Application: Product Search

[Kovashka et al, CVPR 2012, ICCV 2013] [Yu & Grauman, CVPR 2014]

slide-35
SLIDE 35

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Whittle Search

Slide credit: Kristen Grauman

slide-36
SLIDE 36

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Whittle Search

Check Whittle Search demo at: http://godel.ece.vt.edu/whittle/

slide-37
SLIDE 37

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Resources

slide-38
SLIDE 38

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

http://rogerioferis.com/VisualRecognitionAndSearch2014/Resources.html

Resources

slide-39
SLIDE 39

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Resources

Galaxy Morphological Attributes

Data available at: http://data.galaxyzoo.org/

Slide credit: Rogerio Feris  304,122 Galaxy Images  58,719,719 Annotations  83,943 volunteers  11 tasks / 38 answers (fine morphological attributes)

slide-40
SLIDE 40

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Resources

http://www.snapshotserengeti.org/

Slide credit: Rogerio Feris

5 Terabytes of annotated data

Data will be made publicly available soon!

slide-41
SLIDE 41

Learning Visual Semantics: Models, Massive Computation, and Innovative Applications CVPR 2014

Parts and Attributes Workshop

https://filebox.ece.vt.edu/~parikh/PnA2014/ http://rogerioferis.com/PartsAndAttributes/ http://pub.ist.ac.at/~chl/PnA2012/ (ECCV 2010) (ECCV 2012) (ECCV 2014) Check the Call for Extended Abstracts (Posters) Submission deadline: June 30th, 2014