Co cept Co cept Concept Detection Based on Concept Detection Based - - PowerPoint PPT Presentation

TRECVID2008: MCG-ICT-CAS

Concept Detection Based on Concept Detection Based on LDA LDA-

• SVM

SVM

Sheng Tang(唐胜) Jin Tao Li (李锦涛) Ming Li(李明)

Co cept etect o ased o Co cept etect o ased o S

Sheng Tang(唐胜), Jin-Tao Li (李锦涛), Ming Li(李明) Cheng Xie (谢呈), Yi-Zhi Liu (刘毅志), Kun Tao (陶焜), Shao-Xi Xu (徐邵稀) Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China, 100080 E il t @i t T l 8610 62600617 Email: ts@ict.ac.cn Tel:8610-62600617

For TRECVI D 2008 concept detection task we focus on: For TRECVI D 2008 concept detection task we focus on:

• For TRECVI D 2008 concept detection task, we focus on:

For TRECVI D 2008 concept detection task, we focus on:

To improve the training efficiency and explore the

To improve the training efficiency and explore the p g y p p g y p knowledge between concepts or hidden sub knowledge between concepts or hidden sub-domains, we domains, we propose a novel method based on Latent Dirichlet Allocation: propose a novel method based on Latent Dirichlet Allocation: p p p p LDA LDA-based Multiple based Multiple-SVM (LDA SVM (LDA-SVM); SVM);

Early fusion of texture edge and color features TECM:

Early fusion of texture edge and color features TECM:

Early fusion of texture, edge and color features TECM:

Early fusion of texture, edge and color features TECM: TF* I DF weights based on SI FT features + Edge Histogram+ TF* I DF weights based on SI FT features + Edge Histogram+ Color Moments; Color Moments; Color Moments; Color Moments;

I ntroduction of Pseudo Relevance Feedback (PRF) into our

I ntroduction of Pseudo Relevance Feedback (PRF) into our ( ) ( ) concept detection system for the purpose of making re concept detection system for the purpose of making re- trained models more adaptive to the test data trained models more adaptive to the test data. p

1 LDA-SVM

1.2 Topic-simplex Representation Vector (TRV) 1.1 Flowchart of LDA-SVM

LDA T1 T2

……

TN

TRV

SVMΝ SVM1

……

SVM2

Fig 2 TRV of frames in a Topic

1.3 Our Novelties

Fusion

• Sample’s separability-keeping strategy during training

Unlike multi-bag SVM, we only use positive samples in Fig 1 Flowchart of LDA-SVM current topic for the sake of retaining sample’s separability, instead of all positive samples among the whole training set, and ignore the topics with too few positive samples.

• TRV-weight-based fusion strategy during testing

Whil t ti k f f i t d t TRV While testing a keyframe for a given concept, we adopt TRV as the weight vector, instead of equal weighting strategy, to bi th SVM t t f t i d l combine the SVM outputs of topic-models.

2 System overview y

2.1 Early Fusion 2.3 Pseudo Relevance Feedback (PRF)

E l f i f d d l f U lik i ti PRF t h i i t t d Early fusion of texture, edge and color features TECM (890 dims), abbreviation of the combined: TF*IDF i ht b d SIFT f t (345 Unlike existing PRF techniques in text and video retrieval, we propose two preliminary strategies to explore the visual features of positive

TF*IDF weights based on SIFT features (345

dims) Edge Histogram (320 dims) strategies to explore the visual features of positive training samples to improve the quality of pseudo positive samples:

Edge Histogram (320 dims) Color Moments (225 dims).

positive samples:

Similarity-based method

Select pseudo positive samples by calculating

2.2 Novel LDA-SVM Detection Method

Select pseudo positive samples by calculating the feature similarities between top-retrieval examples with positive training samples after

LDA clustering

After quantization of the TF*IDF weights we every retrieval process.

Detector-based method

After quantization of the TF*IDF weights, we use Latent Dirichlet Allocation to cluster all the keyframes into 20 topics according to the Select pseudo positive samples through the

• verall evaluation of positions among the

k d li t f l d t t keyframes into 20 topics according to the maximum element of the TRVof each keyframe.

SVM Training

ranked lists from several detectors.

2 4 Object-based features

g

Sample’s separability-keeping strategy

For all the 20 concepts, we get 344 models

2.4 Object-based features

Object-based features: we train models with For all the 20 concepts, we get 344 models after removing 56 topics with no more than 1 positive sample. j

• bject-based TF*IDF features within labeled

rectangles for positive training samples. But our lt i t d d t il bilit f h

SVM Test TRV-weight-based fusion strategy

result is not good due to unavailability of such

• bject-based features of test samples.

g gy

3 Annotation & Experiments p

3 1 A t ti f t i i d t 3 3 R lt A l i 3.1 Annotation of training data

• Effective: 35.4% improvement (run3 via run1)

3.3 Result Analysis

• Efficient:
• Topic size is greatly smaller

S l i h t i f hi h bilit

• Samples in each topic are of higher separability
• SVM training is very efficient, only about 20

i t f ll th 344 d l l t minutes for all the 344 models on our cluster server (dualcore 1.8ghz *15)

• Employing all samples in each topic for cross

Fi 3 Th i t f f t ti

• Employing all samples in each topic for cross-

validation becomes very practicable (about 12 hours for all 344 model on our cluster server )

In order to encourage researchers to propose methods

Fig 3 The interface for our annotation hours for all 344 model on our cluster server ).

3.4 Conclusion

extracting features based on object rather than the whole frame, we divided the 20 concepts into two groups: (1) Object related concepts

⑴ Eearly fusion TECM, clustering via LDA, sample’s separability-keeping strategy, and TRV-weight-based

3 2 I fAP f

(1) Object-related concepts (2) Scene-related concepts

sepa ability keeping st ategy, a d V weight based fusion strategy together contribute to the high efficiency and effectiveness of our proposed method.

3.2 InfAP of our runs

HLF run InfAP Description

p p ⑵ Determination method of hidden topic number should be carefully studied for further improvement.

A_ICT_1 0.048 Visual Baseline A_ICT_2 0.038 LocalizationClassifier

⑶ PRF method is not stable since the introduction of pseudo positive samples may ruin the separability of

A_ICT_3 0.065 TECM_LDA_SVM A_ICT_4 0.037 TECM_LDA_SVM_PRF

topic samples. ⑷ More frames pershot should be used for test data. ⑸

A_ICT_5 0.076 TECM_LDA_SVM+Baseline A_ICT_6 0.078 Fusion All

⑸ Should combine LIG annotation to remove false annotations.