TRECVID 2008 CBCD TRECVID 2008. CBCD MCG-ICT-CAS MCG-ICT-CAS - - PowerPoint PPT Presentation

TRECVID 2008 CBCD TRECVID 2008. CBCD MCG-ICT-CAS

Sheng Tang Yongdong Zhang Ke Gao Xiao Wu

MCG-ICT-CAS

Sheng Tang, Yongdong Zhang, Ke Gao, Xiao Wu, Xiaoyuan Cao, Huamin Ren,Yufen Wu, Jian Yang

Institute of Computing Technology, Chinese Academy of Sciences

N Ch ll New Challenges

Various Transformations Insertions of patterns, A l t f Picture in picture, Cam-coding, ……. A large amount of Dataset Videos:

200hours, 438 videos

Varied Contents

TVs, Moives, Sports,… 438 videos.

Query Videos:

2010 clips, 4000 minutes 4000 minutes.

2

O C ib i Our Contributions

PlacehoMulti-module processing lder text

Result fusion method

Novel feature for each module

ceholder text

Time sequency consistence q y For copy location

3

O li Outline

Multi-module system Multi module system Novel feature for each module Time sequence consistence Result fusion Result and discussion

4

i S Multi-module System

Module 3 Module 1 PIP

Picture in picture type 1

Global

Global quality decrease such (The original video is inserted in front of a background video) decrease such as blur, adding noise, ……

System

four modules

Module 4 Flip Module 2 Local p

Video horizontal mirroring

Local

Partial content alteration such as occlusion, shift, and crop, …… (including the Picture in Picture (including the Picture in Picture type 2, the original video is the background)

5

S O i System Overview

Query Feature

Query

R

Copy Module Fusion Query Extraction Global Local Local

Query features Frame matching Global Commit Global Commit

Local Depend Local Depend

R E S

py Detection Module Fusion Video Dataset PIP PIP Flip Flip

Feature set Index Ti C d

p

PIP Depend Flip Commit

U L T

Time matching Confidence l l ti

Flowchart of ICT MCG CBCD System

Dataset Flip Flip

Time Code

Flip Commit

T

calculating

_ _ y 6

O li Outline

Multi-module system Multi module system Novel feature for each module Time sequence consistence Result fusion Result and discussion

7

f G Feature for Global Module

Our Contributions for Global Module

System

DC coefficients based Block Gradient

y

Histogram Feature

【Advantage】fast, low-dimension, robust to global transformations global transformations

8

Block Gradient Histogram Feature

1 2 3

Image Block 1-Gradient of each pixel

1 2 3

Image Block 1-Gradient of each pixel

1 2 3 4 5 6

Image X

1 2 3 4 5 6

Image X

7 8 9

X axis

7 8 9

X axis Image Y axis Block Gradient Histogram (3, 1, 1, 1.5, 0.5, 3, 1, 0.8) Image Y axis Block Gradient Histogram (3, 1, 1, 1.5, 0.5, 3, 1, 0.8)

………… 1-8dims 2-8dims ………… 1-8dims 2-8dims ………… 1-8dims 2-8dims ………… 1-8dims 2-8dims

Illustration of Block Gradient Histogram for Global Module

9

Block Gradient Histogram Feature

The Global feature is robust to many transformations:

Change of image quality:

C f i

Change of image content:

10

f Feature for Local Module

Our Contributions for Local Module

System

KLT based Local Patch Feature with

y

Spatial Information

【Advantage】robust to partial occlusion, crop, and shift and shift

11

Local Feature with Spatial Local Feature with Spatial Information

To increase discriminability of local features, we present a method to add spatial information:

Local Patch - Gradient Histogram

Block Gradient Histogra

Local Patch - Gradient Histogram

Block Gradient Histogra Block Gradient Histogra Block Gradient Histogra

Local t h

Block Gradient Histogra (3, 1, 1, 1.5, 0.5, 3, 1, 0.8

Local t h

Block Gradient Histogra (3, 1, 1, 1.5, 0.5, 3, 1, 0.8 Block Gradient Histogra (3, 1, 1, 1.5, 0.5, 3, 1, 0.8 Block Gradient Histogra (3, 1, 1, 1.5, 0.5, 3, 1, 0.8

patch Spatial Neighborhood - block gray rank

3 4

(3 4 1 2) patch Spatial Neighborhood - block gray rank

3 4 3 4

(3 4 1 2)

a

Ill t ti f l l f t ith ti l i f ti

1 2

(3, 4, 1, 2)

1 2 1 2

(3, 4, 1, 2)

Illustration of local feature with spatial information 12

Local Feature with Spatial Local Feature with Spatial Information

• The introduction of spatial information could effectively increase

discriminability of local features, thus improve the matching precision: precision:

Comparison of matching effect Comparison of retrieval precision b f d f i i l i f i Comparison of matching effect before and after using spatial information before and after using spatial information

13

f Feature for PIP Module

Our Contributions for PIP Module

System

Edge detection based PIP boundary determination

y

Block Gradient Histogram features extraction for PIP i (th l b l d l ) PIP region(the same as global module ) 【Advantage】robust to change of scale and 【Advantage】robust to change of scale and

position, simple and fast than scale-invariant local feature based method 14

PIP B d D t ti PIP Boundary Detection

Illustration of PIP boundary location and some instances

15

f i Feature for Flip Module

Our Contributions for Flip Module

System

Vertical Mirror feature

y

Global features and local features extraction for flip module(the same as extraction for flip module(the same as

previous steps)

【Advantage】robust to vertical mirror, simple and fast

16

i i Rotation-Invariant Feature

Find Mirror Dimension Obtain Mirror Feature

17 (3,4,1,5,3,2,3,3)

O li Outline

Multi-module system Multi module system Novel feature for each module Time sequence consistence Result fusion Result and discussion

18

Time Sequence Consistency method

(2)

ng frame lt of matchin Resul

• Graph for copy location

, ,

max ( )

j l i j X l i

M weight node =

∑

tching video

Graph for copy location

(1)

Video similarity based on matching frame-pairs using time sequence consistency method

, , j l i =

i j x

frame frame of Video , if M > location none, else β → ⎧ = ⎨ ⎩

Result of mat

(2)

R

19

O li Outline

Multi-module system Multi module system Novel feature for each module Time sequence consistence Result fusion Result and discussion

20

Result Fusion Method

We tried number of fusion methods including non-hierarchical method and hierarchical method.

• Non-hierarchical method means we use 4 modules to calculate each

query separately at the same time, and only the one with high score will be submitted be submitted.

Query video

Global

Score: 0.95

Local PIP

Score: 0.87 Score: 0 32

【 character 】simple but slow，high recall but low precision, Flip

0.32 Score: 0.10

【 character 】simple but slow，high recall but low precision, hard to determine the score threshold

21

Result Fusion Method

We tried number of fusion methods including non- hierarchical method and hierarchical method.

• Hierarchical method submits the result of each module in some sequence.

For each query, if any previous module has found its corresponding video, we submit the result, and then turn to process the next query. p q y

Query video

Glob

PI

Loc Fli

Not copy Is copy

bal

IP

cal p ip

N N N N Submit? Submit? Submit? Submit?

【 character 】fast and high precision, but depend heavily on

Y Y Y Y Submit? Submit? Submit? Submit?

process sequence

22

O li Outline

Multi-module system Multi module system Novel feature for each module Time sequence consistence Result fusion Result and discussion

23

l d i i Result and Discussion

• Tested in the CIVR07_CBCD dataset, the hierarchical method

performs best for most queries, and processing time is reduced greatly. h l f i i d

• The results of our system in TRECVID2008_CBCD in accordance

with the phenomena. 24

l d i i Result and Discussion

• Tested in the CIVR07_CBCD dataset, the hierarchical method

performs best for most queries, and processing time is reduced greatly. h l f i i d

• The results of our system in TRECVID2008_CBCD in accordance

with the phenomena.

160

)

80 100 120 140

ing time(s)

20 40 60 80

an processi

1 2 3 4 5 6 7 8 9 10

Mea Transformation number

25

l d i i Result and Discussion

based Block Gradient Histogram The use of DC coefficients based Block Gradient Histogram based Block Gradient Histogram based Block Gradient Histogram Feature could avoid the influence

• f gamma change、recoding,

and can be calculated very fast. C P

In

• S

r C g D Q

D Q C C C C C E

Cam Codin PIP type1

nsertions

• f pattern

Strong recoding Change gamma Decrease Quality 3

Decrease Quality 5 Change Content 3 Change Content 5 Combine Everything 26

ng

g

l d i i Result and Discussion

The use of multi-module process could deal with some process could deal with some specific transformations effectively, such as PIP type1. C P

In

• S

r C g D Q

D Q C C C C C E

Cam Codin PIP type1

nsertions

• f pattern

Strong recoding Change gamma Decrease Quality 3

Decrease Quality 5 Change Content 3 Change Content 5 Combine Everything 27

ng

g

l d i i Result and Discussion

The use of multi-module process and fusion method process and fusion method could deal with some combined transformations. C P

In

• S

r C g D Q

D Q C C C C C E

Cam Codin PIP type1

nsertions

• f pattern

Strong recoding Change gamma Decrease Quality 3

Decrease Quality 5 Change Content 3 Change Content 5 Combine Everything 28

ng

g

l d i i Result and Discussion

It is important and practical to find a frame-level description scheme and establish a hierarchical process method, d h f t l i t ld C P

In

• S

r C g D Q

D Q C C C C C E

and the use of temporal consistency could improve the copy location effectively. Cam Codin PIP type1

nsertions

• f pattern

Strong recoding Change gamma Decrease Quality 3

Decrease Quality 5 Change Content 3 Change Content 5 Combine Everything 29

ng

g

l d i i Result and Discussion

Future work: Introduce more features such as color, trajectory… etc; color, trajectory… etc; Object-level copy detection; Scalable mining of large video databases for practical application databases for practical application.

30

Q A Q&A

Any further questions, please contact: ts@ict.ac.cn zhyd@ict.ac.cn y @

31