Scene Recognition Scene Recognition Adriana Kovashka Adriana - - PowerPoint PPT Presentation

▶
scene recognition scene recognition
SMART_READER_LITE
LIVE PREVIEW

Scene Recognition Scene Recognition Adriana Kovashka Adriana - - PowerPoint PPT Presentation

Jul 26, 2023 149 likes •619 views

Scene Recognition Scene Recognition Adriana Kovashka Adriana Kovashka UTCS, PhD student UTCS, PhD student Problem Problem Problem Problem Statement Statement Distinguish between different types of scenes Applications

slide-1
SLIDE 1

Scene Recognition Scene Recognition

Adriana Kovashka Adriana Kovashka UTCS, PhD student UTCS, PhD student

slide-2
SLIDE 2

Problem Problem Problem Problem

  • Statement

Statement

– Distinguish between different types of scenes

Applications

  • Applications

– Matching human perception – Understanding the environment

  • Indexing of images / video

R b ti

  • Robotics

– Graphics

  • In painting
  • In-painting
slide-3
SLIDE 3

Background Background Background Background

  • Definition of a scene

Definition of a scene

– “[A] scene is mainly characterized as a place in which we can move“ [Oliva 2001] in which we can move [Oliva 2001]

  • Assumptions

Human categorization – Human categorization

  • Approaches

– Parsing of the scene – as a whole, or in parts

slide-4
SLIDE 4

Coast [Oliva 2001] Mountain [Oliva 2001]

slide-5
SLIDE 5

Inside City [Oliva 2001] Street [Oliva 2001]

slide-6
SLIDE 6

Kitchen [Lazebnik 2006] Industrial [Lazebnik 2006]

slide-7
SLIDE 7

Scene? Scene? Scene? Scene?

slide-8
SLIDE 8

Scene? Scene? Scene? Scene?

slide-9
SLIDE 9

Urban or natural? Urban or natural? Urban or natural? Urban or natural?

slide-10
SLIDE 10

Urban or natural? Urban or natural? Urban or natural? Urban or natural?

slide-11
SLIDE 11

Spatial Envelope [Oliva 2001] Spatial Envelope [Oliva 2001] Spatial Envelope [Oliva 2001] Spatial Envelope [Oliva 2001]

  • Inspiration from human perception

Inspiration from human perception

– Naturalness, openness, roughness Expansion ruggedness – Expansion, ruggedness

  • Holistic, no recognition of objects
  • Three levels

– “cars and people” vs. “street” vs. “urban environment”

slide-12
SLIDE 12

Spatial Envelope (cont’d) Spatial Envelope (cont’d) Spatial Envelope (cont d) Spatial Envelope (cont d)

  • Scene modeling

Scene modeling

– Discrete Fourier Transform – Windowed Fourier Transform

[Oliva 2001] [Oliva 2001]

slide-13
SLIDE 13

Spatial Envelope (cont’d) Spatial Envelope (cont’d) Spatial Envelope (cont d) Spatial Envelope (cont d)

– Principal Components Analysis Principal Components Analysis

[Oliva 2001]

slide-14
SLIDE 14

Spatial Envelope (cont’d) Spatial Envelope (cont’d) Spatial Envelope (cont d) Spatial Envelope (cont d)

[Oliva 2001]

slide-15
SLIDE 15

Spatial Envelope (cont’d) Spatial Envelope (cont’d) Spatial Envelope (cont d) Spatial Envelope (cont d)

  • Properties of the spatial envelope

Properties of the spatial envelope

– Discriminant spectral template (DST)

  • Relates spectral components to properties of the
  • Relates spectral components to properties of the

spatial envelope

  • Parameter d learned through matching of feature

vectors and property values

– Windowed discriminant spectral template (WDST) (WDST)

slide-16
SLIDE 16

Spatial Envelope (cont’d) Spatial Envelope (cont’d) Spatial Envelope (cont d) Spatial Envelope (cont d)

[Oliva 2001]

slide-17
SLIDE 17

Spatial Envelope (cont’d) Spatial Envelope (cont’d) Spatial Envelope (cont d) Spatial Envelope (cont d)

[Oliva 2001]

slide-18
SLIDE 18

Spatial Envelope (cont’d) Spatial Envelope (cont’d) Spatial Envelope (cont d) Spatial Envelope (cont d)

  • Results

Results

– Scene properties

[Oliva 2001]

slide-19
SLIDE 19

Spatial Spatial Envelope Envelope (cont’d) (cont’d)

[Oliva 2001]

slide-20
SLIDE 20

Spatial Spatial Envelope Envelope (cont’d) (cont’d)

[Oliva 2001]

slide-21
SLIDE 21

Spatial Envelope (cont’d) Spatial Envelope (cont’d) Spatial Envelope (cont d) Spatial Envelope (cont d)

– Classification Classification

  • K-nn
  • 4 out of 7 neighbors

g picked by humans

[Oliva 2001]

slide-22
SLIDE 22

Spatial Envelope (cont’d) Spatial Envelope (cont’d) Spatial Envelope (cont d) Spatial Envelope (cont d)

  • Strengths

Strengths

– Higher-level descriptions Low dimensionality – Low dimensionality – Invariance to object composition Weak local information – Weak local information

  • Weaknesses

– Significant number of human labels

slide-23
SLIDE 23

[Oliva 2001]

slide-24
SLIDE 24

Spatial Pyramid [Lazebnik 2006] Spatial Pyramid [Lazebnik 2006] Spatial Pyramid [Lazebnik 2006] Spatial Pyramid [Lazebnik 2006]

  • Global locally orderless

Global, locally orderless

  • Bag-of-features

E t i f P id M t h K l i 2 d

  • Extension of Pyramid Match Kernel in 2-d
  • Regular clustering of features
slide-25
SLIDE 25

Spatial Pyramid (cont’d) Spatial Pyramid (cont’d) Spatial Pyramid (cont d) Spatial Pyramid (cont d)

[Grauman 2005] as quoted in [Lazebnik 2006] [Lazebnik 2006] [Lazebnik 2006] [Lazebnik 2006]

slide-26
SLIDE 26

Spatial Pyramid (cont’d) Spatial Pyramid (cont’d) Spatial Pyramid (cont d) Spatial Pyramid (cont d)

[Lazebnik 2006]

slide-27
SLIDE 27

Spatial Pyramid (cont’d) Spatial Pyramid (cont’d) Spatial Pyramid (cont d) Spatial Pyramid (cont d)

[Lazebnik 2006]

slide-28
SLIDE 28

Spatial Pyramid (cont’d) Spatial Pyramid (cont’d) Spatial Pyramid (cont d) Spatial Pyramid (cont d)

  • Results

Results

– SVM classification Scene recognition – Scene recognition

[Lazebnik 2006]

65.2% for [Fei-Fei 2005]

slide-29
SLIDE 29

Spatial Pyramid (cont’d) Spatial Pyramid (cont’d)

[Lazebnik 2006]

slide-30
SLIDE 30

Spatial Pyramid (cont’d) Spatial Pyramid (cont’d) Spatial Pyramid (cont d) Spatial Pyramid (cont d)

– Object recognition Object recognition

[Lazebnik 2006]

slide-31
SLIDE 31

Spatial Pyramid (cont’d) Spatial Pyramid (cont’d) Spatial Pyramid (cont d) Spatial Pyramid (cont d)

  • Strengths

Strengths

– Reasonable dimensionality “Locally orderless” – Locally orderless – Dense representation “Robust to failures at individual levels” – “Robust to failures at individual levels”

  • Weaknesses

– No invariability to composition of image – Not robust to clutter

slide-32
SLIDE 32

Scene Completion Scene Completion

http://graphics.cs.cmu.edu/ projects/scene-completion/ [Hays 2007]

…

Input image Scene Descriptor

Image Collection

…

200 matches 20 completions Context matching + blending

Hays and Efros, SIGGRAPH 2007

slide-33
SLIDE 33

[Oliva 2001]

slide-34
SLIDE 34

Topic Models [Fei Topic Models [Fei-Fei 2005] Fei 2005] Topic Models [Fei Topic Models [Fei Fei 2005] Fei 2005]

  • Bayesian hierarchical model

Bayesian hierarchical model

  • Intermediate representations

B f f t

  • Bag-of-features

– 4 ways to extract regions – 2 types of features

slide-35
SLIDE 35

Topic Models (cont’d) Topic Models (cont’d) Topic Models (cont d) Topic Models (cont d)

[Fei-Fei 2005]

slide-36
SLIDE 36

Hierarchical Bayesian text models Hierarchical Bayesian text models

[Fei-Fei 2005]

“beach”

Latent Dirichlet Allocation (LDA)

w

N

c z π

N D

Fei-Fei et al. ICCV 2005

slide-37
SLIDE 37

Topic Models (cont’d) Topic Models (cont’d) Topic Models (cont d) Topic Models (cont d)

  • η – distribution of

l l b l class labels

  • θ – parameter

(estimated by EM)

  • c – class label
  • π – distribution of

themes for image

  • z – theme
  • x – patch

x patch

  • β – parameter

(estimated by EM)

[Fei-Fei 2005]

slide-38
SLIDE 38

Topic Models (cont’d) Topic Models (cont’d) Topic Models (cont d) Topic Models (cont d)

  • Codebook

Codebook

– 174 codewords

[Fei-Fei 2005] [Fei-Fei 2005]

slide-39
SLIDE 39

Topic Models (cont’d) Topic Models (cont’d)

[Fei-Fei 2005]

slide-40
SLIDE 40

Topic Topic Topic Topic Models Models (cont’d) (cont’d) (cont d) (cont d)

  • Results

[Fei-Fei 2005]

slide-41
SLIDE 41

Topic Models (cont’d) Topic Models (cont’d) Topic Models (cont d) Topic Models (cont d)

  • Strengths

Strengths

– Unsupervised Invariant to composition – Invariant to composition

  • Weaknesses

– No geometry – Matches of themes to categories – No correspondence to semantic categories

slide-42
SLIDE 42

Comparison Comparison Comparison Comparison

  • Global vs local

Global vs. local

– Spatial Envelope, Spatial Pyramid Topic Models – Topic Models

  • Viewpoint / location biases vs. invariability

S – Spatial Pyramid – Topic Models, Spatial Envelope

slide-43
SLIDE 43

Comparison (cont’d) Comparison (cont’d) Comparison (cont d) Comparison (cont d)

  • Intermediate representations

Intermediate representations

– Spatial Envelope, Topic Models

Supervision vs no supervision

  • Supervision vs. no supervision

– Spatial Envelope S – Topic Models, Spatial Pyramid

  • Object recognition?
slide-44
SLIDE 44

Discussion Discussion Discussion Discussion

  • Object recognition vs scene recognition

Object recognition vs. scene recognition

– Global approaches

  • Spatial Pyramid scenes vs objects results
  • Spatial Pyramid, scenes vs. objects results

– Bag-of-features

  • Use of scene recognition
  • Use of scene recognition
  • Ambiguous scenes
  • Human recognition of scenes

– Importance

slide-45
SLIDE 45

References References References References

[Fei-Fei 2005] L. Fei-Fei and P. Perona. A Bayesian Hi hi l M d l f L i N t l S Hierarchical Model for Learning Natural Scene

  • Categories. CVPR 2005.

[Grauman 2005] K. Grauman and T. Darrell. The Pyramid M t h K l Di i i ti Cl ifi ti ith S t f Match Kernel: Discriminative Classification with Sets of Image Features. ICCV 2005. [Hays 2007] J. Hays and A.A. Efros. Scene completion using millions of photographs SIGGRAPH 2007 using millions of photographs. SIGGRAPH 2007. [Lazebnik 2006] S. Lazebnik, C. Schmid, and J. Ponce. Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories CVPR 2006 Recognizing Natural Scene Categories. CVPR 2006. [Oliva 2001] A. Oliva and A. Torralba. Modeling the Shape

  • f the Scene: a Holistic Representation of the Spatial

Envelope IJCV 2001

  • Envelope. IJCV 2001.