Robust Scene Categorization by Learning Image Statistics in Context - - PowerPoint PPT Presentation

Intelligent Sensory Information Systems University of Amsterdam

Jan van Gemert,Jan-Mark Geusebroek, Cees Snoek, Dennis Koelma, Cor Veenman, Frank Seinstra, Marcel Worring and Arnold Smeulders

Robust Scene Categorization by Learning Image Statistics in Context for BBC rushes

Slide 2

Robust Scene Categorization by Learning Image Statistics in Context

• Visual Concepts by Scene Recognition
• How to

recognize an airplane? Context!

Overview

Introduction Visual Features Contextures Evaluation Conclusions

Sky Sky Sky Sky Sky Sky Sky Sky Sky

• Use Proto-Concepts to Describe Context
• SVM: link Context to Concepts
• Learn Models on News data, evaluate on BBC rushes

Slide 3

Robust Scene Categorization by Learning Image Statistics in Context

• Color Invariance
• Invariant to Global Illumination Changes

Low Level Features

Introduction Visual Features Contextures Evaluation Conclusions

λλ

E

E

λ λ λ

Blue-Yellow Luminance Red-Green

λ

E

Slide 4

Robust Scene Categorization by Learning Image Statistics in Context

• Distribution of Edge responses: Integrated Weibull

Natural Image Statistics

• −

− Γ

γ γ

β µ γ γ γ γ r 1 1 exp ) 1 ( 2

/ 1

Introduction Visual Features Contextures Evaluation Conclusions

• There are more non-edges than edges

Slide 5

Robust Scene Categorization by Learning Image Statistics in Context

Proto-Concepts

Introduction Visual Features Contextures Evaluation Conclusions

Sky Sky Sky Road Flag_USA Building (321) Car (192), Charts (52) Crowd (270) Desert (82) Fire (67) Flag_USA (98) Maps (44) Mountain (41) Road (143) Sky (291) Smoke (64) Snow (24), Vegetation (242) Water (108) In brackets: nr. Annotations at least 20 frames

Slide 6

Robust Scene Categorization by Learning Image Statistics in Context

Region Detection

Introduction Visual Features Contextures Evaluation Conclusions

Slide 7

Robust Scene Categorization by Learning Image Statistics in Context

Contextures

• Contexture: Occurrence Histogram of Proto-Concepts

Global (Accumulate) Local (Arg Max)

Building Car Road Sky …

Introduction Visual Features Contextures Evaluation Conclusions

Building Car Road Sky … Building Car Road Sky … Building Car Road Sky … Building Car Road Sky … Building Car Road Sky … Building Car Road Sky … Building Car Road Sky … Building Car Road Sky … Building Car Road Sky … Building Car Road Sky … Building Car Road Sky …

Slide 8

Robust Scene Categorization by Learning Image Statistics in Context

Learning Concepts in Video

Introduction Visual Features Contextures Evaluation Conclusions

• Image to Shot: sample every second.
• Use SVM to link Contextures to 101 Concepts
• Performance on TrecVid Testset

0.1 0.2 0.3 0.4 0.5 0.6 B u i l d i n g C a r E x p l

• s

i

• n

F l a g _ U S A M a p s M

• u

n t a i n P r i s

• n

e r S p

• r

t s W a l k i n g W a t e r s c a p e Concepts Average Performance Visual Only Best-Submission

Slide 9

Robust Scene Categorization by Learning Image Statistics in Context

BBC Rushes

• Evaluate the SVM-models trained on

TRECVID data on the BBC rushes

• 25 ‘survive’:
• aircraft, bird, boat, building, car, charts, cloud, crowd,

face, female, food, government building, grass, meeting, mountain, outdoor, overlayed text, sky, smoke, tower, tree, urban, vegetation, vehicle, waterscape

Introduction Visual Features Contextures Evaluation Conclusions

Slide 10

Robust Scene Categorization by Learning Image Statistics in Context

BBC rushes Screenshots (I)

Introduction Visual Features Contextures Evaluation Conclusions

• Tower
• Building

Slide 11

Robust Scene Categorization by Learning Image Statistics in Context

BBC rushes Screenshots (II)

Introduction Visual Features Contextures Evaluation Conclusions

• Food
• Face

Slide 12

Robust Scene Categorization by Learning Image Statistics in Context

Conclusions & Discussion

• Does a picture say more than a thousand words?

– According to our Trec results: Not (yet)

• Robust methods provide a rich untapped information source:

– Re-use of annotations – Re-use of Training Models – Ideally: train a concept once, apply everywhere Introduction Visual Features Contextures Evaluation Conclusions