Beyond bags of Features Spatial Pyramid Matching for Recognizing - - PowerPoint PPT Presentation

beyond bags of features
SMART_READER_LITE
LIVE PREVIEW

Beyond bags of Features Spatial Pyramid Matching for Recognizing - - PowerPoint PPT Presentation

Beyond bags of Features Spatial Pyramid Matching for Recognizing Natural Scene Categories Camille Schreck, Romain Vavassori Ensimag December 14, 2012 Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 1 / 32 State of art


slide-1
SLIDE 1

Beyond bags of Features

Spatial Pyramid Matching for Recognizing Natural Scene Categories Camille Schreck, Romain Vavassori

Ensimag

December 14, 2012

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 1 / 32

slide-2
SLIDE 2

1

State of art

2

Method Pyramids Spatial Matching Scheme Features extraction Summary

3

Results Scene Category Recognition Caltech-101 Graz Dataset

4

Conclusion

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 2 / 32

slide-3
SLIDE 3

Introduction

Overall objective : semantic categorisation. Use spatial information. Global representation.

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 3 / 32

slide-4
SLIDE 4

1

State of art

2

Method

3

Results

4

Conclusion

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 4 / 32

slide-5
SLIDE 5

State of art

Bags of feature

Images described as an orderless collection of features. Good performances. Do not use the information about the spatial layout of the features.

Multiresolution histograms

Subsampling an image and compute a global histogram at each level.

Generalized histograms to locally orderless images

For each Gaussian aperture at a given location and scale, the locally

  • rderless image returns the histogram of image features aggregated over

that aperture.

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 5 / 32

slide-6
SLIDE 6

1

State of art

2

Method Pyramids Spatial Matching Scheme Features extraction Summary

3

Results

4

Conclusion

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 6 / 32

slide-7
SLIDE 7

1

State of art

2

Method Pyramids Spatial Matching Scheme Features extraction Summary

3

Results Scene Category Recognition Caltech-101 Graz Dataset

4

Conclusion

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 7 / 32

slide-8
SLIDE 8

Pyramid Match Kernel

Goal

Find the approximate correspondance between 2 set of vectors, X and Y, in a d-dimentional feature space.

Idea

correpondances are computed at different levels of resolution. at each level, a finer grid is set on the space. 2 vectors are said to match if they are on the same cell. take the weighted sum of all the matches.

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 8 / 32

slide-9
SLIDE 9

Subdivisions of the feature space

We compute matches at different level of resolution 0,..,L.

At the level of resolution l

the grid is divided in 2l along each dimension. the grid has D = 2dl cells where d is the number of dimensions.

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 9 / 32

slide-10
SLIDE 10

Histograms intersection

Histograms of X and Y

Hl

X and Hl Y are the histograms of X and Y at level l

where Hl

X(i) is the number of vectors of X in the ith cell of the grid.

Histogram Intersection function

Give the number of matches at level l : I(Hl

X, Hl Y ) = D

  • i=1

min(Hl

X(i), Hl Y (i))

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 10 / 32

slide-11
SLIDE 11

Histograms intersection

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 11 / 32

slide-12
SLIDE 12

Histograms intersection

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 12 / 32

slide-13
SLIDE 13

Histograms intersection

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 13 / 32

slide-14
SLIDE 14

Computation of the kernel

All matches found at level l + 1 are found also at level l. → Number of new matches at level l is given by Il − Il+1. We sum all the matches weighted by

1 2L−l . The more the grid is coarse, the less the

matches are weighted.

pyramid match kernel

Mercer kernel : κl(X, Y ) = IL +

L−1

  • l=0

1 2L−l (Il − Il−1)

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 14 / 32

slide-15
SLIDE 15

1

State of art

2

Method Pyramids Spatial Matching Scheme Features extraction Summary

3

Results Scene Category Recognition Caltech-101 Graz Dataset

4

Conclusion

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 15 / 32

slide-16
SLIDE 16

“Orthogonal” approach

Matching of two collection of features in a high-dimensional appearance space

quantize all feature vectors into M discrete types, giving M channels. perform pyramid matching in the 2-dimensional image space for each channel m = 1..M.

Assumption

Only features of the same type m can be matched to one another.

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 16 / 32

slide-17
SLIDE 17

Final kernel is the sum of separate channel kernels

K L(X, Y ) =

M

  • m=1

κL(Xm, Ym) where Xm and Ym are the coordinates of features of type m found in the respective image. KL can be computed as the intersection of the histograms obtain by concatenating the histograms of each channel.

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 17 / 32

slide-18
SLIDE 18

Example

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 18 / 32

slide-19
SLIDE 19

1

State of art

2

Method Pyramids Spatial Matching Scheme Features extraction Summary

3

Results Scene Category Recognition Caltech-101 Graz Dataset

4

Conclusion

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 19 / 32

slide-20
SLIDE 20

Two kinds of features for the experiments

Weak features

Edge points at two scales and eight orientations. → M = 16 channels.

Strong features

SIFT descriptors of 16x16 pixels. k-mean clustering to get a visual vocabulary. In the experiments vocalubary size is M = 200 or M = 400.

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 20 / 32

slide-21
SLIDE 21

1

State of art

2

Method Pyramids Spatial Matching Scheme Features extraction Summary

3

Results Scene Category Recognition Caltech-101 Graz Dataset

4

Conclusion

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 21 / 32

slide-22
SLIDE 22

Summary of the method

Clustering features from a training set. Computation of the “description” of a query image. Comparison with the description of each image in test set.

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 22 / 32

slide-23
SLIDE 23

1

State of art

2

Method

3

Results Scene Category Recognition Caltech-101 Graz Dataset

4

Conclusion

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 23 / 32

slide-24
SLIDE 24

1

State of art

2

Method Pyramids Spatial Matching Scheme Features extraction Summary

3

Results Scene Category Recognition Caltech-101 Graz Dataset

4

Conclusion

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 24 / 32

slide-25
SLIDE 25

Scene Category Recognition

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 25 / 32

slide-26
SLIDE 26

Example

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 26 / 32

slide-27
SLIDE 27

1

State of art

2

Method Pyramids Spatial Matching Scheme Features extraction Summary

3

Results Scene Category Recognition Caltech-101 Graz Dataset

4

Conclusion

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 27 / 32

slide-28
SLIDE 28

Caltech-101

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 28 / 32

slide-29
SLIDE 29

1

State of art

2

Method Pyramids Spatial Matching Scheme Features extraction Summary

3

Results Scene Category Recognition Caltech-101 Graz Dataset

4

Conclusion

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 29 / 32

slide-30
SLIDE 30

Graz Dataset

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 30 / 32

slide-31
SLIDE 31

1

State of art

2

Method

3

Results

4

Conclusion

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 31 / 32

slide-32
SLIDE 32

Conclusion

“holistic” approach for categorisation. Simple method. Gives better results than bag-of-features.

Schreck, Vavassori (Ensimag) Beyond bags of Features December 14, 2012 32 / 32