COSTA: Co-occurrence statistics for zero-shot classification Thomas - - PowerPoint PPT Presentation

▶
costa co occurrence statistics for zero shot
SMART_READER_LITE
LIVE PREVIEW

COSTA: Co-occurrence statistics for zero-shot classification Thomas - - PowerPoint PPT Presentation

Aug 05, 2023 434 likes •749 views

COSTA: Co-occurrence statistics for zero-shot classification Thomas Mensink University of Amsterdam Parts & Attributes Workshop ECCV 2014 September 12th Parts & Attributes PnA 2014 COSTA 2 Parts & Attributes Semantic

slide-1
SLIDE 1

COSTA: Co-occurrence statistics for zero-shot classification

Thomas Mensink – University of Amsterdam

Parts & Attributes Workshop – ECCV 2014 September 12th

slide-2
SLIDE 2

Parts & Attributes

PnA 2014 COSTA 2

slide-3
SLIDE 3

Parts & Attributes

  • Semantic representation of images

– Properties of class / context of class – Each attribute relevant for a few classes

  • Interesting for

– Zero-shot prediction – Few-shot prediction – Recounting of visual content

PnA 2014 COSTA 3

slide-4
SLIDE 4

Parts & Attributes: Disadvantages

  • Unnatural distinction between

– Attributes to be detected – Classes of interest

  • Binary map from classes to attributes
  • Inherently multi-class zero-shot prediction

PnA 2014 COSTA 4

slide-5
SLIDE 5

CAN’T WE DO ZERO-SHOT PREDICTION IN MULTI-LABELED DATASETS?

PnA 2014 COSTA 5

slide-6
SLIDE 6

Multi-label zero-shot classification

  • I’m looking for a label, which I have not seen
  • before. However, this picture contains also:

– Indoor – Living room – Table – …

  • We can classify based on context

PnA 2014 COSTA 6

slide-7
SLIDE 7

COSTA: Intuition

PnA 2014 COSTA 7

slide-8
SLIDE 8

COSTA: Intuition (2)

v

(2)

where

1)

PnA 2014 COSTA 8

slide-9
SLIDE 9

COSTA: Design

PnA 2014 COSTA 9

slide-10
SLIDE 10

COSTA: Classifier

  • Goal: Estimate classifier for unseen label
  • Assumption: k trained classifiers
  • Zero-shot classifier:
  • Where

is based on co-occurrence stats

PnA 2014 COSTA 10

slide-11
SLIDE 11

Co-Occurrence Statistics

How to set a weight s, based on counts c

  • Normalized
  • Binarized
  • Burstiness corrected
  • Dice coefficient

PnA 2014 COSTA 11

slide-12
SLIDE 12

Co-Occurrence Statistics (2)

Co-occurrences can be obtained from:

  • Ground-truth data (proof-of-concept)
  • Web search engines
  • Flickr Tags
  • Microsoft COCO

PnA 2014 COSTA 12

slide-13
SLIDE 13

Example: Beach Holiday

PnA 2014 COSTA 13

Concept Normalized Co-Oc Weight Sea 0.1810 Water 0.0992 Summer 0.0548 LandscapeNature 0.0435 SunsetSunrise 0.0383 Sports 0.0367 Travel 0.0347 Ship 0.0346 Sunny 0.0319 Big Group 0.0282

slide-14
SLIDE 14

Example: Beach Holidays

PnA 2014 COSTA 14

slide-15
SLIDE 15

TWO EXTENSIONS

PnA 2014 COSTA 15

slide-16
SLIDE 16

Defining a concept by what it is not

  • Knowing what is not related to a visual concept is

informative for its visual scope

  • Related: used in image retrieval [Jegou&Chum ECCV 12]
  • Example: a car is never* together with a table
  • Solution: positive and negative co-occurrences:

* Ok. Never say never, but it is very unlikely

PnA 2014 COSTA 16

slide-17
SLIDE 17

Regression to improve COSTA

  • Our problem is estimating a classifier:
  • Objective: the estimated classifier should be

as close as possible to the learned classifier if we would have visual labels.

PnA 2014 COSTA 17

slide-18
SLIDE 18

Regression to improve COSTA (2)

  • Idea: learn a weight ak per classifier
  • Note: Weights are independent of novel class
  • Solve: Regression objective
  • Train: Using a leave-one-out setting over train classes

PnA 2014 COSTA 18

slide-19
SLIDE 19

EXPERIMENTS

PnA 2014 COSTA 19

slide-20
SLIDE 20

Experimental setup

  • Hierarchical SUN dataset [Choi et al. CVPR’10]

– 107 Labels – 4367 train 4317 test images – 5.34 labels per image

  • Fisher Vectors (3096 dim)
  • SVMs with 2 fold cross-validation
  • In paper also experiments on:

– ImageCLEF’10 and CUB-Attributes

PnA 2014 COSTA 20

slide-21
SLIDE 21

Multi-label Zero-Shot Classification

PnA 2014 COSTA 21

slide-22
SLIDE 22

AP per Concept

PnA 2014 COSTA 22

slide-23
SLIDE 23

Co-occurrences from the Web

PnA 2014 COSTA 23

slide-24
SLIDE 24
  • Ok. But?

PnA 2014 COSTA 24

slide-25
SLIDE 25

How about DeepNets?

  • Related works: DeViSe and CONSe

– Very similar to COSTA, few differences – Predict 1000 ImageNet Classes – Measure relatedness by Word2Vec

  • Preliminary result: co-occurrences capture

visual semantics better than Word2Vec

PnA 2014 COSTA 25

slide-26
SLIDE 26

Failure mode(s)?

  • Fine-grained classification:

– Co-occurrences are not sufficient to distinguish: Italian Sparrow Great Sparrow

PnA 2014 COSTA 26

slide-27
SLIDE 27

Failure mode(s)?

  • Fine-grained classification:

Attributes make sense on segmented objects

  • Z. Li, E. Gavves, T. Mensink, and C.G.M. Snoek , ECCV 2014

PnA 2014 COSTA 27

slide-28
SLIDE 28

Conclusion: COSTA

PnA 2014 COSTA 28

  • First method designed for multi-label zero-shot
  • Many visual concepts can be described as an open

set of concept-to-concept relations

  • Describe latent image semantics with co-occurrences
  • Exploit natural bias in natural images
slide-29
SLIDE 29

COSTA: Co-occurrence statistics for zero-shot classification

  • T. Mensink, E. Gavves, and C.G.M. Snoek , CVPR 2014