Coloring Visual Codebooks Coloring Visual Codebooks for Concept - - PowerPoint PPT Presentation

▶
coloring visual codebooks coloring visual codebooks for
SMART_READER_LITE
LIVE PREVIEW

Coloring Visual Codebooks Coloring Visual Codebooks for Concept - - PowerPoint PPT Presentation

May 24, 2023 271 likes •610 views

Coloring Visual Codebooks Coloring Visual Codebooks for Concept Detection in Video for Concept Detection in Video Koen van de Sande Koen van de Sande Cees Snoek Cees Snoek Jan van Gemert Gemert Jan van Jasper Uijlings Uijlings Jasper

slide-1
SLIDE 1

18 18-

  • 11

11-

  • 2008

2008 TRECVID workshop TRECVID workshop

Coloring Visual Codebooks Coloring Visual Codebooks for Concept Detection in Video for Concept Detection in Video

Koen van de Sande Koen van de Sande Cees Cees Snoek Snoek Jan van Jan van Gemert Gemert Jasper Jasper Uijlings Uijlings Jan Jan-

  • Mark

Mark Geusebroek Geusebroek Theo Theo Gevers Gevers Arnold Arnold Smeulders Smeulders University of Amsterdam University of Amsterdam MediaMill

slide-2
SLIDE 2

2 2

Introduction Introduction

Concept detection: Concept detection:

  • Machine learning based on image descriptors only

Machine learning based on image descriptors only In a real In a real-

  • world video:

world video:

  • Large variations in viewing and lighting conditions

Large variations in viewing and lighting conditions

  • image description complicated

image description complicated How do changes in viewpoint and illumination How do changes in viewpoint and illumination conditions affect concept detection? conditions affect concept detection?

slide-3
SLIDE 3

3 3

Viewpoint Changes Viewpoint Changes

  • Orientation and scale of object changes

Orientation and scale of object changes

  • Salient point methods robustly detect regions

Salient point methods robustly detect regions

  • INRIA

INRIA-

  • LEAR (VOC 2007 winner): preferred for

LEAR (VOC 2007 winner): preferred for concept detection accuracy are concept detection accuracy are

  • Harris

Harris-

  • Laplace salient points

Laplace salient points

  • Dense sampling

Dense sampling

Lowe IJCV 2004 Mikolajczyk IJCV 2005 Zhang IJCV 2007 Marszalek VOC 2007

Harris-Laplace Dense sampling

slide-4
SLIDE 4

4 4

Concept Detection Stages Concept Detection Stages

Spatio‐ temporal sampling Visual feature extraction Codebook transform Kernel‐ based learning

slide-5
SLIDE 5

5 5

Spatio Spatio-

  • Temporal Sampling

Temporal Sampling

multi‐ frame Dense sampling Spatial pyramid Harris‐ Laplace Lazebnik CVPR 2006 Marszalek VOC 2007

  • Spatial pyramid

Spatial pyramid

  • 1x1

1x1 whole image whole image

  • 2x2

2x2 image quarters image quarters

  • 1x3

1x3 horizontal bars horizontal bars

  • Temporal analysis of up to 5 frames per shot

Temporal analysis of up to 5 frames per shot

slide-6
SLIDE 6

6 6

Illumination Changes Illumination Changes

Concept detection suffers from unstable region description Concept detection suffers from unstable region description SIFT descriptor: SIFT descriptor:

  • Most well

Most well-

  • known

known

  • State

State-

  • of
  • f-
  • the

the-

  • art performance

art performance

  • Intensity

Intensity-

  • based descriptor:

based descriptor: no color no color

Proposed color descriptors: Proposed color descriptors:

  • HueSIFT

HueSIFT, HSV , HSV-

  • SIFT,

SIFT, OpponentSIFT OpponentSIFT, C , C-

  • SIFT,

SIFT, rg rgSIFT SIFT

  • Increase discriminative power

Increase discriminative power

  • Increase illumination invariance

Increase illumination invariance

Research questions Research questions

  • What are the properties of these color descriptors?

What are the properties of these color descriptors?

  • How do they perform?

How do they perform?

  • See the evaluation in our CVPR 2008 paper

See the evaluation in our CVPR 2008 paper

van de Weijer PAMI 2006 Bosch CIVR 2007 Burghouts CVIU 2008 van de Sande CVPR 2008

slide-7
SLIDE 7

7 7

Example: light color change Example: light color change

Transformed color SIFT descriptor is invariant Transformed color SIFT descriptor is invariant

slide-8
SLIDE 8

8 8

Lambertian Lambertian reflectance model reflectance model Corresponds to diagonal Corresponds to diagonal-

  • offset model of illumination change
  • ffset model of illumination change

Unified framework for modeling: Unified framework for modeling:

  • Shadows

Shadows

  • Shading

Shading

  • Light color changes

Light color changes

  • Highlights

Highlights

  • Scattering

Scattering

Invariance properties: Diagonal model Invariance properties: Diagonal model

Von Kries 1970 Finlayson ICIP 2005 Canonical illuminant Unknown illuminant Illuminant parameters

slide-9
SLIDE 9

9 9

Color Descriptor Taxonomy Color Descriptor Taxonomy

Light intensity change Light intensity shift Light intensity change and shift Light color change Light color change and shift SIFT

+ + + + +

OpponentSIFT

+/- + +/- +/- +/-

C-SIFT

+ + + +/- +/-

rgSIFT

+ + + +/- +/-

Transformed color SIFT

+ + + + +

van de Sande CVPR 2008

Invariance properties of the descriptors used Descriptor

slide-10
SLIDE 10

10 10

Invariant Visual Descriptors Invariant Visual Descriptors

Color SIFT: Color SIFT:

  • Intensity

Intensity-

  • based SIFT

based SIFT

  • OpponentSIFT

OpponentSIFT

  • C

C-

  • SIFT

SIFT

  • rg

rgSIFT SIFT

  • Transformed color SIFT

Transformed color SIFT

Add color, but also keep intensity information Add color, but also keep intensity information TV2007test results: TV2007test results:

  • Trained on TRECVID2007 development set

Trained on TRECVID2007 development set

  • Evaluated on TRECVID2007 test set

Evaluated on TRECVID2007 test set

  • TRECVID2007 development + test = 2008 development

TRECVID2007 development + test = 2008 development

Visual Descriptors MAP on TV2007test Intensity SIFT Intensity SIFT 0,144 0,144 5x Color SIFT 5x Color SIFT 0,155 0,155 relative

+8%

slide-11
SLIDE 11

11 11

Concept Detection Stages Concept Detection Stages

Spatio‐ temporal sampling Visual feature extraction Codebook model Kernel‐ based learning

slide-12
SLIDE 12

12 12

Visual Codebook Model Visual Codebook Model

  • Codebook consists of

Codebook consists of codewords codewords

  • Constructed with k

Constructed with k-

  • means clustering on descriptors

means clustering on descriptors

  • We use 4,000

We use 4,000 codewords codewords per codebook per codebook

Cluster Assign Dense+OpponentSIFT Feature vector (length 4000)

slide-13
SLIDE 13

13 13

Codebook Assignment Codebook Assignment

Soft assignment using Gaussian kernel Soft assignment using Gaussian kernel

Assignment MAP on TV2007test Hard Hard 0,155 0,155 Soft Soft 0,166 0,166 relative

+7%

Hard assignment Soft assignment

van Gemert ECCV 2008

  • Codeword
slide-14
SLIDE 14

14 14

Codebook Library Codebook Library

Single codebook depends on Single codebook depends on

  • Sampling method

Sampling method

  • Descriptor

Descriptor

  • Codebook construction method

Codebook construction method

  • Codebook assignment

Codebook assignment

Codebook library is Codebook library is… …

  • a configuration of several codebooks

a configuration of several codebooks

Codebook Codebook Sampling method Sampling method Descriptor Descriptor Construction Construction Assignment Assignment #1 #1 Dense Dense OpponentSIFT OpponentSIFT K K-

  • means

means Soft Soft #2 #2 Harris Harris-

  • Laplace

Laplace SIFT SIFT Radius Radius-

  • based

based Soft Soft #3 #3 Dense Dense rg rgSIFT SIFT K K-

  • means

means Hard Hard … … Dense Dense C C-

  • SIFT

SIFT K K-

  • means

means Hard Hard

slide-15
SLIDE 15

15 15

Codebook Library Codebook Library

(cont (cont’ ’d) d) For a frame: For a frame:

  • Each codebook in the library has feature vector of length 4,000

Each codebook in the library has feature vector of length 4,000

  • Final feature vector is concatenation (4 books ~ length 16,000)

Final feature vector is concatenation (4 books ~ length 16,000)

  • Spatial pyramid adds more dimensions:

Spatial pyramid adds more dimensions:

  • Feature vector length easily >100,000

Feature vector length easily >100,000… …

  • 1x3

1x3 12,000 12,000

  • 1x1

1x1 4,000 4,000

  • 2x2

2x2 16,000 16,000

slide-16
SLIDE 16

16 16

SVM kernel trick: SVM kernel trick: precompute precompute kernel kernel

SVM learning does not need feature vectors SVM learning does not need feature vectors SVM learning needs distance between vectors only: SVM learning needs distance between vectors only: Very large decrease in computation time Very large decrease in computation time

  • Precompute

Precompute the SVM kernel matrix the SVM kernel matrix

  • Long vectors possible: only need 2 in memory at once

Long vectors possible: only need 2 in memory at once

  • Parameter optimization re

Parameter optimization re-

  • uses

uses precomputed precomputed matrix matrix

  • γ dist( , )

K( , ) = e

slide-17
SLIDE 17

17 17

Impact of annotations Impact of annotations

Ours = common annotation effort + ICT Ours = common annotation effort + ICT-

  • CAS + verifying positives

CAS + verifying positives

Codebook library Ours* (type B) Common ann. effort* (type A) 3x Color SIFT 3x Color SIFT 0,152 0,152 0,152 0,152 5x Color SIFT 5x Color SIFT 0,155 0,155 0,155 0,155 Codebook library Ours* Common ann. effort* 3x Color SIFT 3x Color SIFT 0,1516 0,1516 0,1521 0,1521 5x Color SIFT 5x Color SIFT 0,1548 0,1548 0,1549 0,1549

Add a digit Add a digit… … On average, didn On average, didn’ ’t help t help

*MiAP

  • n TV2008test
slide-18
SLIDE 18

18 18

Concept Detection Stages Concept Detection Stages

Spatio‐ temporal sampling Visual feature extraction Codebook model Kernel‐ based learning

slide-19
SLIDE 19

19 19

Robust Temporal Approach Robust Temporal Approach

  • No cloud computing yet: need to be efficient

No cloud computing yet: need to be efficient ☺ ☺

  • Process 5 frames per shot in test set

Process 5 frames per shot in test set

  • Linear increase in computation: x5

Linear increase in computation: x5

  • In 2005 paper 7.5% to 38% improvement noted for multi

In 2005 paper 7.5% to 38% improvement noted for multi-

  • frame

frame (worst (worst-

  • case vs. best

case vs. best-

  • case using oracle)

case using oracle)

  • Robust color SIFT

Robust color SIFT with with temporal = ~20% improvement temporal = ~20% improvement

Codebook library Frames/shot MiAP on TV2008test 3x Color SIFT 3x Color SIFT 1 1 0,152 0,152 3x Color SIFT 3x Color SIFT 5 5 0,184 0,184 relative

+20%

Snoek ICME 2005

slide-20
SLIDE 20

20 20

The Good The Good

  • Close

Close-

  • up of hands

up of hands

  • Boats and ships

Boats and ships

  • Cityscape

Cityscape

slide-21
SLIDE 21

21 21

The Bad The Bad

  • Emergency Vehicle (only 46 examples, many at night)

Emergency Vehicle (only 46 examples, many at night)

  • Bus (only 64 examples)

Bus (only 64 examples)

slide-22
SLIDE 22

22 22

… … and the trivial and the trivial

  • Dog (in trailer)

Dog (in trailer)

  • Flower (in trailer)

Flower (in trailer)

  • Mountain (in trailer)

Mountain (in trailer)

slide-23
SLIDE 23

23 23

Conclusions Conclusions

  • Illumination conditions affect concept detection

Illumination conditions affect concept detection

  • SIFT+colorSIFT

SIFT+colorSIFT improves ~8% improves ~8%

  • Soft codebook assignment improves ~7%

Soft codebook assignment improves ~7%

  • Robust

Robust colorSIFT colorSIFT with simple multi with simple multi-

  • frame improves ~20%:

frame improves ~20%:

  • Room for more advanced methods in TRECVID 2009

Room for more advanced methods in TRECVID 2009

  • Precomputed

Precomputed kernel matrix reduces SVM computation time kernel matrix reduces SVM computation time

  • Near

Near-

  • duplicates from trailers hamper progress:

duplicates from trailers hamper progress:

  • We suggest to exclude them, or count only once

We suggest to exclude them, or count only once

slide-24
SLIDE 24

24 24

Visit http://www.science.uva.nl/~ksande/ for color descriptor software

slide-25
SLIDE 25

25 25

References References

  • K. E. A. van de Sande, T. Gevers and C. G. M. Snoek, “Evaluation of

Color Descriptors for Object and Scene Recognition”, CVPR 2008

  • M. Marszalek, C. Schmid, H. Harzallah and J. van de Weijer, “Learning

Object Representations for Visual Object Class Recognition”, Visual Recognition Workshop in conjunction with ICCV 2007

  • J.C. van Gemert, J.M. Geusebroek, C.J. Veenman, A.W.M. Smeulders,

“Kernel Codebooks for Scene Categorization”, ECCV 2008

  • K. Mikolajczyk and C. Schmid, “A Performance Evaluation of Local

Descriptors”, PAMI 2005

  • D. G. Lowe, “Distinctive Image Features from Scale-Invariant Keypoints”,

IJCV 2004

  • J. Zhang, M. Marszalek, S. Lazebnik and C. Schmid, “Local Features and

Kernels for Classification of Texture and Object Categories: A Comprehensive Study”, IJCV 2007

  • C. G. M. Snoek et al, “The MediaMill TRECVID 2008 Semantic Video

Search Engine”, TRECVID Workshop 2008

slide-26
SLIDE 26

26 26

Results per Concept Results per Concept

slide-27
SLIDE 27

27 27

Codebook Library Definitions Codebook Library Definitions

  • Intensity SIFT

Intensity SIFT

  • 5x Color SIFT / Hard

5x Color SIFT / Hard

Codebook Codebook Sampling method Sampling method Descriptor Descriptor Construction Construction Assignment Assignment #1 #1 Dense Dense SIFT SIFT K K-

  • means

means Hard Hard #2 #2 Harris Harris-

  • Laplace

Laplace SIFT SIFT K K-

  • means

means Hard Hard Codebook Codebook Sampling method Sampling method Descriptor Descriptor Construction Construction Assignment Assignment #1 #1 Dense Dense OpponentSIFT OpponentSIFT K K-

  • means

means Hard Hard #2 #2 Harris Harris-

  • Laplace

Laplace OpponentSIFT OpponentSIFT K K-

  • means

means Hard Hard #3 #3 Dense Dense Transformed Transformed color SIFT color SIFT K K-

  • means

means Hard Hard #4 #4 Harris Harris-

  • Laplace

Laplace Transformed Transformed color SIFT color SIFT K K-

  • means

means Hard Hard #5 #5 Dense Dense SIFT SIFT K K-

  • means

means Hard Hard #6 #6 Harris Harris-

  • Laplace

Laplace SIFT SIFT K K-

  • means

means Hard Hard #7 #7 Dense Dense C C-

  • SIFT

SIFT K K-

  • means

means Hard Hard #8 #8 Harris Harris-

  • Laplace

Laplace C C-

  • SIFT

SIFT K K-

  • means

means Hard Hard #9 #9 Dense Dense rg rgSIFT SIFT K K-

  • means

means Hard Hard #10 #10 Harris Harris-

  • Laplace

Laplace rg rgSIFT SIFT K K-

  • means

means Hard Hard

slide-28
SLIDE 28

28 28

Codebook Library Definitions (2) Codebook Library Definitions (2)

  • 5x Color SIFT / Soft

5x Color SIFT / Soft

Codebook Codebook Sampling method Sampling method Descriptor Descriptor Construction Construction Assignment Assignment #1 #1 Dense Dense OpponentSIFT OpponentSIFT K K-

  • means

means Soft Soft #2 #2 Harris Harris-

  • Laplace

Laplace OpponentSIFT OpponentSIFT K K-

  • means

means Soft Soft #3 #3 Dense Dense Transformed Transformed color SIFT color SIFT K K-

  • means

means Soft Soft #4 #4 Harris Harris-

  • Laplace

Laplace Transformed Transformed color SIFT color SIFT K K-

  • means

means Soft Soft #5 #5 Dense Dense SIFT SIFT K K-

  • means

means Soft Soft #6 #6 Harris Harris-

  • Laplace

Laplace SIFT SIFT K K-

  • means

means Soft Soft #7 #7 Dense Dense C C-

  • SIFT

SIFT K K-

  • means

means Soft Soft #8 #8 Harris Harris-

  • Laplace

Laplace C C-

  • SIFT

SIFT K K-

  • means

means Soft Soft #9 #9 Dense Dense rg rgSIFT SIFT K K-

  • means

means Soft Soft #10 #10 Harris Harris-

  • Laplace

Laplace rg rgSIFT SIFT K K-

  • means

means Soft Soft

slide-29
SLIDE 29

29 29

Codebook Library Definitions (3) Codebook Library Definitions (3)

  • 3x Color SIFT

3x Color SIFT

Codebook Codebook Sampling method Sampling method Descriptor Descriptor Construction Construction Assignment Assignment #1 #1 Dense Dense OpponentSIFT OpponentSIFT K K-

  • means

means Soft Soft #2 #2 Harris Harris-

  • Laplace

Laplace OpponentSIFT OpponentSIFT K K-

  • means

means Soft Soft #3 #3 Dense Dense Transformed Transformed color SIFT color SIFT K K-

  • means

means Soft Soft #4 #4 Harris Harris-

  • Laplace

Laplace Transformed Transformed color SIFT color SIFT K K-

  • means

means Soft Soft #5 #5 Dense Dense SIFT SIFT K K-

  • means

means Soft Soft #6 #6 Harris Harris-

  • Laplace

Laplace SIFT SIFT K K-

  • means

means Soft Soft

slide-30
SLIDE 30

30 30

Positive Examples Needed Positive Examples Needed

Concept Relative #positive examples TwoPeople |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Street ||||||||||||||||||||||||||||||||||||||||||||||||||||| Hand |||||||||||||||||||||||||||||||||||||||||||| Flower ||||||||||||||||| Singing |||||||||||||||| BoatShip ||||||||||||| Driver ||||||||||||| Nighttime ||||||||||||| Mountain |||||||| Harbor ||||||| Classroom ||||||| Telephone ||||||| DemonstrationOrProtest ||||| Cityscape |||| Bridge |||| Kitchen |||| Dog ||| EmergencyVehicle | AirplaneFlying | Bus |

██ = highest overall infAP for MediaMill

slide-31
SLIDE 31

31 31

Annotation effects: 5x Color SIFT Annotation effects: 5x Color SIFT

Type B (ours) Type A (common ann. effort) Classroom 0,044 0,035 Bridge 0,026 0,049 EmergencyVehicle 0,010 0,016 Dog 0,124 0,128 Kitchen 0,135 0,109 AirplaneFlying 0,227 0,181 TwoPeople 0,134 0,128 Bus 0,022 0,014 Driver 0,234 0,276 Cityscape 0,191 0,195 Harbor 0,089 0,094 Telephone 0,128 0,149 Street 0,299 0,295 DemonstrationOrProtest 0,116 0,100 Hand 0,315 0,286 Mountain 0,168 0,249 Nighttime 0,274 0,232 BoatShip 0,277 0,273 Flower 0,127 0,155 Singing 0,157 0,134 MiAP 0,1548 0,1549

slide-32
SLIDE 32

32 32

Annotation effects: 3x Color SIFT Annotation effects: 3x Color SIFT

Type B (ours) Type A (common ann. effort) Classroom 0,044 0,044 Bridge 0,053 0,076 EmergencyVehicle 0,010 0,010 Dog 0,122 0,123 Kitchen 0,132 0,115 AirplaneFlying 0,212 0,154 TwoPeople 0,128 0,128 Bus 0,016 0,009 Driver 0,209 0,258 Cityscape 0,210 0,216 Harbor 0,064 0,059 Telephone 0,109 0,124 Street 0,276 0,269 DemonstrationOrProtest 0,138 0,145 Hand 0,279 0,271 Mountain 0,162 0,224 Nighttime 0,282 0,244 BoatShip 0,308 0,309 Flower 0,104 0,089 Singing 0,177 0,178 MAP 0,1516 0,1521