EE 6882 Visual Search Engine Prof. Shih Fu Chang, Jan. 30, 2012 - - PDF document

▶
ee 6882 visual search engine
SMART_READER_LITE
LIVE PREVIEW

EE 6882 Visual Search Engine Prof. Shih Fu Chang, Jan. 30, 2012 - - PDF document

Apr 12, 2023 293 likes •450 views

1/30/2012 EE 6882 Visual Search Engine Prof. Shih Fu Chang, Jan. 30, 2012 Lecture #2 Visual Features: Global features and matching Evaluation metrics (Many slides from A. Efors, W. Freeman, C. Kambhamettu, L. Xie, and likely others)

slide-1
SLIDE 1

1/30/2012 1

EE 6882 Visual Search Engine

  • Prof. Shih‐Fu Chang, Jan. 30, 2012

Lecture #2

 Visual Features: Global features and matching  Evaluation metrics

(Many slides from A. Efors, W. Freeman, C. Kambhamettu, L. Xie, and likely others) (Slides preparation assisted by Rong‐Rong Ji)

2

Course Format

Lectures + two hands‐on homeworks (due 2/13, 2/27)

Mid‐term project

Review and implement topics of interest, 2 students each team

Proposal due 3/5, narrated slides due 3/26

Selected projects presented and discussed in class (3/26‐4/9)

Final project

Extension of mid‐term projects encouraged, 2 students each team

Proposal due 4/2, narrated slides due 4/30

Selected projects presented and discussed in class (4/30‐5/7)

Grading:

Class participation (20%), homework (20%), mid‐term (20%), final (40%)

Late policy: a total “budget” of 4 days for late submissions. No other delays accepted.

slide-2
SLIDE 2

1/30/2012 2

Image Features

 Why features are needed?

 Finding similar images in database  Classifying images to categories  Tracking objects in video  Creating panorama  Stereo matching ‐> 3D

 Desired properties

 Compact (~100 – 1000 dimensions)  Easy to compute (30 fps for video)  Robust (invariant to photometric,

geometric, content variations)

3 photoguides.net

20 40 60 80 100 0.2 0.4 0.6 0.8 1

merl.com

Desired Properties of Visual Features

Invariance:

 Rotation, scaling, cropping, shift, etc.  illumination, pose, clutter, occlusion,

viewpoint

slide-3
SLIDE 3

1/30/2012 3

Invariant Local Features

Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging parameters

Features Descriptors

(Slide of A. Efros) 6

(review) Imaging Formation

R G R G R G B G B G R G R G R G B G B G R G R G R

Lens CCD Sensor

Demosaicking Filter

Camera Response Function

Additive Noise DSP (White Balance, Contrast Enhancement … etc)

irradiance Image intensity

slide-4
SLIDE 4

1/30/2012 4

Color Spaces and Color Order Systems

 Color Spaces

 RGB – cube in Euclidean space  Standard representation used in color displays  Drawbacks

 RGB basis not related to human color judgments  Intensity should be one of the dimensions of color  Important perceptual components of color are

hue, saturation, and brightness

 Perceptual color spaces: HIS, HSV

R G B r g b R G B R G B R G B         

8

Understanding HSI from RGB

Turn the RGB cube so that Black‐ White axis is vertical

Each plane containing the B‐W axis and a color point contains all the colors of the same hue

Hue represented as angle between the plane and a reference plane (e.g. Red)

Saturation: distance to axis, less saturated by mixing more grey colors

Intensity measured by intersection with the B‐W axis.

Cross section shape: triangle – hexagon ‐ triangle

Images from Gonzalez and Woods

slide-5
SLIDE 5

1/30/2012 5

9

Colors on the HSI color cone

Cross section approximated by triangle or circle

HSI values computed by various geometrical models, e.g.,

More suitable for measuring perceptual distance

Can be quantized unevenly, e.g., Columbia VisualSEEK System: 16M colors (in RGB) quantized to 166 HSV colors (18 Hue, 3 Sat, 3 Val, 4 Gray)

                                  B G R V V I 6 / 1 6 / 1 6 / 2 6 / 1 6 / 1 3 / 1 3 / 1 3 / 1

2 1

) ( tan

1 2 1

V V H

2 / 1 2 2 2 1

) ( V V Chroma  

10

Manipulations in the HSI space

HSI values of primary/secondary colors

HSI allows independent manipulations of colors

Hue of Green & Blue set to 0.

Saturation of Cyan reduced by half.

Intensity of White reduced by half.

slide-6
SLIDE 6

1/30/2012 6

Color Histogram

Feature extraction from color images

 Choose GOOD color space  Quantize color space to reduce number of colors

Invariance?

 Scale, shift, rotation, crop, view angle, illumination, clutter,

  • cclusion

Advantages

 Easy to compute and compare

Cons

 Lack spatial information, dimension may be high

1 [ , ] , [ , ] , [ , ] [ , , ]

R G B RGB m n

if I m n r I m n g I m n b h r g b

  • therwise

      



Color Moments

 Is there a more compact representation than

color histogram?

 Compute moment statistics in each color channel.

?

slide-7
SLIDE 7

1/30/2012 7

Localizing

http://www.ai.mit.edu/courses/6.801/Fall2002/

Color Layout Search

Query results

Columbia VisualSEEk (Smith & Chang, ’96) IBM QBIC (Flickner et al ’95)

slide-8
SLIDE 8

1/30/2012 8

Color correlogram

http://www.ai.mit.edu/courses/6.801/Fall2002/ http://www.ai.mit.edu/courses/6.801/Fall2002/

slide-9
SLIDE 9

1/30/2012 9

http://www.ai.mit.edu/courses/6.801/Fall2002/

E E A A B B E E A A B B E A A B B B A A B D C B A A B C B B A A B B C B

Color Coherence Vector (CCV) (Pass et al, 1997)

           

1 1 1 1 1 1

, ,..., , , ,..., ,

I n n I n n n n G i i i i H i i i i i i G H

G G by triangular inequality                

 

                     

 

฀ ฀

Not just count of colors, also check adjacency

3 3 1 1 2 2 3 3 1 1 2 2 3 1 1 2 2 2 1 1 2 3 1 2 1 1 2 1 2 2 1 1 2 2 1 2

5 1 3 15 12 3 3 1 2 1 size color E D C B A regions 1 3 5 15 12 3 2 1   CCV color

Coherent! Size threshold: 3 Region segmentation

= =

slide-10
SLIDE 10

1/30/2012 10

Lp distance

Quadratic distance

Histogram Intersection

Mohalanobis distance

Distance Metrics between Feature Vectors

where Cx is the covariance matrix

Normalize distance in the major/minor axes

20 40 60 80 100 0.2 0.4 0.6 0.8 1

p i p p

i x i x D

/ 1 2 1

) ) ( ) ( (   

) ( ) ( ) ) ( ) ( ) , ( ) ( ) ( (

2 1 2 1 2 1 2 1

x x C x x j x j x j i C i x i x D

T j i q

           

C(i,j): color distance

Mohalanobis Metric

   

2 1 1 2 1 2 1

(1,1) (1,2) ... (1, ) ... ... ... ... ( ,1) ( ,2) ... ( , ) ( , ) ( ) ( ) ( ) ( ) / 1, :

T mah x x N k k k

D x x C x x c c c d covariance matrix C c d c d c d d c i j x i m i x j m j N N number of samples

 

                         

  • o o
  • xi

xj

  • o o
  • xi

xj

  • xi

xj

  • o
  • xi

xj

  • xi

xj

  • i

j

c s s  

1 2

i j

c s s  

c 

1 2

i j

c s s 

i j

c s s 

si, sj: std. deviation

d: dimension of features

slide-11
SLIDE 11

1/30/2012 11

Mohalanobis Metric

1 2 1 2 1 2 1 1 1 2 1 2 1 2

| ...| ( , ,..., ) | ...| | ...| ( ( , ,..., )) | ...|

T x d d d T x d d d

C e e e diag e e e C e e e diag e e e      

 

                             

e1 e2 Project data to the eigen vectors, divide with the sd of each eigen dimension, and compute Euclidian distance

where Cx is the covariance matrix

Normalize distance in the eigen vector axes

Mohalanobis Metric (cont.)

Advantages of Mahalanobis metric

 Account for scaling of coordinate axes  Invariant under linear transformation  Correct for correlation  Produce curved as well as linear decision boundaries

 

Potential issue

 Need enough training data to estimate Cov. Matrix

2 2

,

T y x y x

If y Ax C AC A D D    

. km cm . . . .. . . . . . . . . .. .. .

.. . . . ... . . . . . .. . . . .

  • Maha. Dist.
  • Maha. Dist.

c1 m1 cc mc xi Minimum Selector Selected class

slide-12
SLIDE 12

1/30/2012 12

Earth Mover’s Distance (EMD)

Rubner, Tomasi, Guibas ’98

Mallow’s distance in statistics in 1950’s

Transportation Problem [Dantzig’51] I J ci

j

I: set of suppliers J: set of consumers cij : cost of shipping a unit of supply from i to j

Problem: find the optimal flows fij

0, , , ,

i j ij i I i I ij ij j i I ij i j J j i j J

minimize c f s.t. f i I j J (No reverse shipping) f y j J (satisfy each consumer need /cacacity) f x i I (bounded by each supplier's limit) y x (

    

       

   

i I

feasibility)

EMD of Color Histogram

             

1 1 1 1

, ,..., , , ,..., , ( ) ( ) ,

j i M N ij ij i j M N ij i j

h h 1 h 2 h M g= g 1 g 2 h N assume g j h i C f EMD h g f

   

          

   

Earth Hole

1 1 1

/

M N N ij ij j i j j ij ij ij

= C f g Fill up each hole C : distance between color i in color space h and color j in color space g f : move f units of mass from color i in h to color j in g

  

 

Normalization by the denominator term

 Avoid bias toward low mass distributions (i.e., small images)  what’s the difference if both h and g are normalized first?

slide-13
SLIDE 13

1/30/2012 13

Evaluation

Detection

False Alarms

Misses

Correct Dismissals

2 / ) ( ) /( ) /( ) /(

1

R P R P F D B B F B A A P C A A R          1

  • N

" Irrelevant " Relevant" " 1    n Vn B V D A V C V B V A

N n n N n n K n n K n n

       

   

       

) ) 1 ( ( ) ( ) 1 (

1 1 1 1

N Images K Returned Results

Recall

Precision

Fallout

Combined D

B

C

A

“Returned” “Relevant Results” Ground truth search DB

Evaluation Measures

Precision at depth K Precision Recall Curve Receiver Operating Characteristic (ROC Curve)

) vs ( R P

P

R

K V P

k n n k

/ ) (

1

 

 

B A vs

A (hit) B (false)

slide-14
SLIDE 14

1/30/2012 14

Evaluation Metric: Average Precision

Ranked list of data in response to a query

3/7 3/6 3/5 3/4 2/3 1/2 1/1 Precision 1 1 1 truth Ground D D D D D

s

... ...

21 63 8 15

function indicator I total R D I P R K AP

j K j j

: data, relevant

  • f

# : )] correct is ( [ ) , min( 1

1

  

AP approximates areas under PR curve

1 2 3 4 5 6 7 Precision j

3  

i

P AP

1.0

Example:

Evaluation Metric: Average Precision

 Observations (AP)

 AP depends on the rankings of relevant data and the size of the

relevant data set. E.g., R= 10

Case I:

+ + + + + + + + +

  • - - -
  • +

Pre: 1

1 1 1 1 1 1 1 1 0 0 0 0 1

AP= 1 Case II:

  • +

Pre: 1/ 2 AP= 1/2

  • + - + - + - + - + - + - + - + - +

1/ 2 1/ 2 1/ 2 1/ 2 1/ 2 1/ 2 1/ 2 1/ 2 1/ 2

Case II: Pre:

  • - -
  • +

+ + + + + + + +

1/ 11 2/ 12 10/ 20 … …

AP~ 0.3

slide-15
SLIDE 15

1/30/2012 15

Homework # 1

Given a small image database and a few queries

Implement codes to extract color histogram

Implement codes to measure L2 image similarity

Use image object labels to measure precision/recall curves

Bonus:

Add new color features or similarity metrics to improve performance

Design GUI for result browsing

Reading List

  • Rui, Y., T.S. Huang, and S.‐F. Chang, Image retrieval: current techniques, promising

directions and open issues. Journal of Visual Communication and Image Representation, 1999. 10(4): p. 39‐62.

  • Smith, J.R. and S.‐F. Chang. VisualSEEk: a Fully Automated Content‐Based Image

Query System. in ACM International Conference on Multimedia. 1996. Boston, MA.

  • David G. Lowe, Distinctive Image Features from Scale‐Invariant Keypoints,

International Journal of Computer Vision, 60(2), 2004, pp91‐110.

  • Randen, T. and J. Husoy, Filtering for texture classification: A comparative study.

Pattern Analysis and Machine Intelligence, IEEE Transactions on, 2002. 21(4): p. 291‐310.

  • Mikolajczyk, K. and C. Schmid, A performance evaluation of local descriptors. IEEE

Transactions on Pattern Analysis and Machine Intelligence, 2005: p. 1615‐1630.

  • Brown, M., R. Szeliski, and S. Winder. Multi‐image matching using multi‐scale
  • riented patches. in IEEE CVPR. 2005.