Columbia University TRECVID 2005 Search Task Shih-Fu Chang, Winston - - PowerPoint PPT Presentation
TRECVID 2005 Workshop Columbia University TRECVID 2005 Search Task Shih-Fu Chang, Winston Hsu, Lyndon Kennedy, Akira Yanagawa, Eric Zavesky, Dong-Qing Zhang Digital Video and Multimedia Lab Columbia University Nov. 14 2005
Shih-Fu Chang, Winston Hsu, Lyndon Kennedy, Akira Yanagawa, Eric Zavesky, Dong-Qing Zhang
http://www.ee.columbia.edu/dvmm
TRECVID 2005 Workshop
Multi-modal Search Tools
Content Exploitation
User Level Search Objects
http://www.ee.columbia.edu/cuvidsearch
automatic story segmentation video speech text near-duplicate detection concept detection feature extraction (text, video, prosody) concept search text search Image matching story browsing Near-duplicate search Interactive search automatic/manual search cue-X re-ranking mining query topic classes user search pattern mining
Information Bottleneck principle
low-level features
Information Bottleneck principle & Kernel Density Estimation (KDE) semantic label
semantic clustering
cluster cond. prob. (relevance to semantic label)
= topic “Arafat”
Y= story boundary Y=“demonstration” Y= search relevance
achieve accurate story segmentation
– Focus on visual and audio (prosody) features only – Without a priori manual selection of features – High accuracy across multi-lingual data sources
– Dataset
– Accuracy on the validation set
– Results donated to whole TRECIVD 2005 community
http://www.ee.columbia.edu/dvmm/downloads/cuex_story.htm
in other new s pope john paul the second w ill get his first look at the shroud of turin today that's the piece
years it has already draw n up million visitors the pope's visit to northw est italy has also included beatification services for three people the vatican says john paul is now the longest serving pope this century he has surpassed pope pious the tw elfth w ho served for nineteen years seven months and seven days
Story
Shot Shot Shot Shot Shot
Query
Find shots of Pope John Paul second
with coherent semantics
detected by Cue-X using audio- visual features
than 100% in TRECVID 2005 automatic search
performance of interactive video search
Relative contributions from different search tools
…
Color Moment Color Moment
Global or block-based features:
background clutter
variations
Part Part relation
Part-based model:
traditional enhanced
Individual images Salient points, high entropy regions Attributed Relational Graph (ARG)
Graph Representation
size; color; texture
Collection of training images Random Attributed Relational Graph (R-ARG)
Statistical Graph Representation
Statistics of attributes and relations
machine learning
spatial relation
consistently improves by more than 10% for all concepts
spatio-dominant concepts such as “US flag”.
with the discriminant classifiers using fixed features.
fixed feature Baseline Adding Parts-based
SVM fixed feature Baseline Adding Parts-based
Spatio-dominant concepts: “US Flag”
Scene Change Camera Change Digitization Digitization
Parts-based Stochastic Attribute Relational Graph Learning
Stochastic graph models the physics of scene transformation Measure IND likelihood ratio
Learning Pool Learning
multi-channel broadcast
variations
Similarity matching < IND detection <
Duplicate detection is the single most effective tool in our Interactive Search
Subshots
Query Text “Find shots of a road with one
Part-of-Speech Tags - keywords “road car” Map to concepts WordNet Resnik semantic similarity Concept Metadata Names and Definitions Concept Space 39 dimensions (1.0) road (0.1) fire (0.2) sports (1.0) car …. (0.6) boat (0.0) person Confidence for each concept Concept Models Simple SVM, Grid Color Moments, Gabor Texture Concept Reliability Expected AP for each concept. Concept Space 39 dimensions (0.9) road (0.1) fire (0.3) sports (0.9) car …. (0.2) boat (0.1) person (0.9) road (0.1) fire (0.3) sports (0.9) car …. (0.2) boat (0.1) person (0.9) road (0.1) fire (0.3) sports (0.9) car …. (0.2) boat (0.1) person (0.9) road (0.1) fire (0.3) sports (0.9) car …. (0.2) boat (0.1) person (0.9) road (0.1) fire (0.3) sports (0.9) car …. (0.2) boat (0.1) person Euclidean Distance
queries to high- level feature detection
defined keywords from concept definitions
semantic distance between query and concept
and reliability for subshot documents
.195 Fused .115 Concept .002 CBIR .169 Story Text AP Method
“Find shots of boats.”
.095 Fused .090 Concept .009 CBIR .053 Story Text AP Method
“Find shots of a road with one or more cars.”
Manual keyword selection allows more relationships to be found.
Query Text “Find shots of an office setting, i.e., one
computers and one or more people” Concepts Office Query Text “Find shots of a graphic map of Iraq, location of Bagdhad marked - not a weather map” Concepts Map Query Text “Find shots of one or more people entering or leaving a building” Concepts Person, Building, Urban Query Text Find shots of people with banners or signs Concepts March or protest
(2 )
rank clusters by
Query: “AL clinic bombing”
(1) (4 )
random variable: Y
(3 ) Text Search
(5 )
rank within-cluster features by density prob. use only estimated from rough search results (e.g., text search scores), user feedbacks, etc. low-level feature: X
cue-X clustering
TRECVID 2005)
– 17% in MAP, 46% in soccer (171), 36% in helicopter (158), 32% in Blair (153), 28% in Abbas (154), etc. – No external search examples provided but discovered automatically topic: soccer (171) reranked results text search (“goal soccer match” ) topic: Blair (153) reranked results text search (“tony blair” ) 32%↑ 46%↑
Find Person A Find Person B Find Person C Find Event D Find Event E Find Object F Find Object G Query Semantics Search Performance Video Text Audio Key:
Automatic Joint semantics- performanc e grouping Manually defined query classes
query topic pool
person-X
– named persons
– named objects, – sports, and – generic scene classes
expansion
– Google class for cats, birds and airport terminals.
Named persons Named
sports
Google expansion
Generic scenes
Post-Mortem Analysis
common error rate
relevance feedback rounds Example Log Detail
Detailed search and topic criterion Aggregate tool actions by search time Monitor labeling to understand interface usage Ground truth included in label actions
Text+Story+Anchor Removal +CueX Re-rank +CBIR+Concept Search 0.114 Text+Story+Anchor Removal +CueX Re-rank +CBIR 0.111 Text+Story+Anchor Removal +CueX Re-rank 0.107 Text+Story+Anchor Removal 0.095 Text+Story 0.087 Text 0.039 Components MAP
text baseline + story boundary + Cue-X re-rank (visual features) + concept search
MAP Run
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 R i c e A l l a w i K a r a m i J i n t a
l a i r A b b a s B a g h d a d m a p t e n n i s s h a k i n g h a n d s h e l i c
t e r B u s h f i r e b a n n e r s e n t e r b u i l d i n g m e e t i n g b
t b a s k e t b a l l p a l m t r e e s a i r p l a n e r
d c a r s m i l i t a r y v e h i c l e s b u i l d i n g s
c e r
f i c e Topic AP
Multimodal Automatic Search Max Official Median Official
Varied search strategies
duplicate and traditional search
lots of duplicate browsing Strategy dynamic for each topic
candidates for duplicates
discovery by story browsing
actions usually best in traditional search methods
Top-ranking interactive searches User 1 User 2
Formula for Success: 1. Find positives through any search method 2. Iteratively browse through the near-duplicates or story browsing Close to Best 149 (Rice), 151 (Karami), 153 (Blair), 154 (Abbas), 157 (shaking hands), 161 (banners), 166 (palm trees), 168 (roads/cars), 169 (military vehicles), and 171 (soccer) Best Overall Performance 160 (fire), 164 (boat), and 162 (entering building)