Joint Visual-Text Modeling for Multimedia Retrieval JHU CLSP - - PowerPoint PPT Presentation
Joint Visual-Text Modeling for Multimedia Retrieval JHU CLSP Workshop 2004 Final Presentation, August 17 2004 1 Team Undergraduate Students Desislava Petkova (Mt. Holyoke), Matthew Krause (Georgetown) Graduate Students
1
2
(Georgetown)
(JHU)
3
“ … Palestinian leader Yes Sir You’re Fat today said …”
Find clips showing Yasser Arafat
VIDEO CLIPS
“ … Palestinian leader Yasser Arafat today said …”
Multimedia Retrieval System
Yasser Arafat
Process Query Image Process Query Text Spoken Document Retrieval Image
Content-based
Retrieval
Most research has addressed: I. Text queries, text (or degraded text) documents
Combine Scores
“ … Palestinian leader Yasser Arafat today said …”
4
Process Query Text Joint word- visterm retrieval Process Query Image
Yasser Arafat
VIDEO CLIPS
“ … [Yes sir, you’re fat today said]…
Query of Words and Visterms Document
words Query
words Document of Words and Visterms Retrieve documents using p(Document|Query) Retrieve documents using p(dw,dv | qw,qv)
5
Show that joint visual-text modeling
Demonstrate and Evaluate performance of
6
Automatically annotate video with
E.g. Video contains a face, in a studio-
Retrieve video
Given a query, select suitable meta-data
Combine with text-retrieval in a unified
7
Corpus
Broadcast news videos used for Hub4
120 Hours of video
Tasks
Shot-boundary detection News Story segmentation (multimodal) Concept detection (Annotation) Search task
8
COREL photograph database
5000 high-quality photographs with
Task
Annotation
9
Statement of Information need + Examples Manual Selection of System Parameters Ranked list of video shots Manual Interactive NIST Evaluation
10
Statement of Information need + Examples Automatic Selection of System Parameters Ranked list of video shots Isolate Algorithmic issues from interface and user issues NIST Evaluation
11
Visterms Words Words Visterms
Baseline model Relating document visterms to query words (MT,Relevance Model,HMMs) Relating document words to query images (Text Classification experiments) Visual-only retrieval models
Rank documents with p(qw,qv|dw,dv)
12
Concept annotation performance
Compare against manual ground truth
Retrieval task performance
Compare against NIST relevance
Both measured using Mean Average
13
T t relevant i
∈ ∈
} {
14
TRECVID03 Corpus 120 Hours Ground Truth on Dev data Train 38K shots Dev Test 10K shots TRECVID03 IR Collection 32K Shots Train 4500 images Test 500 images COREL Corpus 5000 images
15
Original L*a*b Edge Strength Co-occurrence
16
Greyscale image Interest points
17
Regular partition
L*a*b Moments (COLOR) Smoothed Edge Orientation Histogram
Grey-level Co-occurrence matrix
Interest Point neighborhood
COLOR, EDGE, TEXTURE
18
Words Visterms Visterms Words
Translation (MT) models (Paola), Relevance Models (Shao Lei,Desislava), Graphical Models (Pavel, Brock) Text classification models (Matt) Integration & Summary (Dietrich)
19
20
Words Visterms Visterms Words
Translation (MT) models
) | ( ) | ( ) | (
V w c V w
d c p c q p d q p
=
21
p(f|e) = ∑ p(f,a|e) a p(c|v) = ∑ p(c,a|v) a
Direct translation model
grass grass grass grass grass grass grass grass grass tiger tiger tiger tiger tiger tiger grass
22
concepts sun sky waves sea Solution : Vector quantization visterms In Machine Translation discrete tokens In our task However, the features extracted from regions are continuous
{fn1, fn2, …fnm} -> vk
sun sky sea waves tiger water grass water harbor sky clouds sea
v10 v22 v35 v43 c5 c1 c38 c71 v20 v21 v50 v10 c15 c21 c83 v78 v78 v1 v1 c21 c19 c1 c56 c38 v10 v22 v35 v43 v10 v20 v21 v50 v78 v78 v1 v1
23
p (sun | )
p(c|v) : Probabilities obtained from direct translation
∈
=
V
d v V V
v c P d d c P ) | ( 1 ) ( |
v10 v22 v35 v43
24
field foals horses mare tree horses foals mare field flowers leaf petals stems flowers leaf petals grass tulip people pool swimmers water swimmers pool people water sky mountain sky snow water sky mountain water clouds snow jet plane sky sky plane jet tree clouds people sand sky water sky water beach people hills Top: manual annotations, bottom : predicted words (top 5 words with the highest probability) Red : correct matches
25
Features : color, texture, edge Extracted from blocks, or around interest points
Observations
better performance than features extracted around interest points
Edge features give the best performance
visterms vocabulary didn’t help
the performance slightly
mAP values for different features
26
Strategies compared (a) IBM Model 1 p(c|v) (b) HMM on top of (a) (c) IBM Model 4 on top of (b)
Number of iterations in Giza training affects the performance
but cannot produce rare words
Corel TREC 0.125 0.124
27
but not for TREC
=
C j V j j i V i
1 1
Corel TREC Model 1 0.125 0.145 Model 1 + Word-CO 0.124 0.124
28
query comprising a concept (source language)
( )
4 4 3 4 4 2 1
V
d C V v V V
G c p d v p v c p d c p
∀ ∈
− + ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ =
same
) | ( ) 1 ( | ) | ( ) | ( λ λ
29
a query comprising a concept (source language)
∈
=
V
d v V V
v c p d v p d c p ) | ( | ) | (
∈
=
V
d v Train V
v c p v DF d c score ) | ( ) ( ) | (
30
Model 1 0.124 CLIR using Model 1 0.126
Significant at p=0.04
Average Precision values for the top 10 words For some concepts we achieved up to 0.6
31
32
33
34
Words Visterms Visterms Words
Use Relevance Models to estimate the probabilities of concepts given test keyframes
) | ( ) | ( ) | (
v w c v w
d c p c q p d q p
=
Goal:
35
Images are defined by spatial context.
Isolated pixels have no meaning. Context simplifies recognition/retrieval. E.g.Tiger is associated with grass, tree,
36
Originally introduced for text retrieval and
2002
A nice way of introducing context in images
images and words
37
training images
Tiger
Tree Grass
) | ( ) | ( ) ( ) , (
| | 1
J v P J c P J P d c P
i T J d i v
v
∈ =
=
and Relevance Using Cross-Media Relevance Models, In Proc. SIGIR’03.
38
Model
: Gaussian Kernel : Bandwidth
=
| | 1
J i Ji
β K
39
A generative model Concept words wj generated by an i.i.d. sample from a
multinomial
Visterms vi generated by a multi-variate (Gaussian)
density
40
Normalized CRM
CRM.
multinomial for words).
document in text retrieval).
Annotation, in CVPR’04
and Retrieval, in ICASSP04
41
On Corel data Set: Normalized-CRM works best Models
Mean average Precision
42
Sky train railroad locomotive water Cat tiger bengal tree forest Snow fox arctic tails water Mountain plane jet water sky Tree plane zebra herd water Birds leaf nest water sky
43
Model: Normalized continuous relevance
Features: color and texture
feature only get very slight improvement
Evaluate annotation on the development
44
45
46
Current models based on single frames
But video is dynamic
Has motion information.
Use dynamic (motion) information
Better image representations
Model events/actions
47
actions/events.
help.
than to segment static images.
48
Current approach
Retrieve videos using static frames.
Feature representations
Problem: visterms not correlated well with
49
“you are to be particularly careful”
50
Pictures from Patrick Bouthemy’s Website, INRIA
a: Segmentation using only still image information b: Segmentation using only motion information
51
Shot boundary detection
Segment moving objects
motion.
Visual features for object and background. Motion features for object
Motion features for background.
52
One approach - modify relevance model to
Probabilistically annotate shots in the test
Other models e.g. HMM also possible
) | ( ) | ( ) | ( ) ( )) , ( , (
| | 1
S m P S v P S c P S P d d c P
i T S d i i m v
∈ =
=
T: training set, S: shots in the training set
) | ( ) | ( ) ( ) , (
| | 1
J v P J c P J P d c P
i T J d i v
v
∈ =
=
53
If discrete visterms use smoothed
If continuous use kernel density
Take advantage of repeated instances
54
Modify models to include dynamic
Train on TrecVID03 development
Test on TrecVID03 test dataset
Retrieve using TrecVID 2003 queries. Evaluate retrieval performance using mean
55
56
Score probabilities
But there seems to
Try to use
57
P w s r s 1 r s
p w pdf w s w p w pdf w s w
= = =
58
For each word:
examples
Revise word probabilities
59
Did not improve annotation performance on
60
61
62
distribution of each word w in the vocabulary with the visterms of I
J T
J T
i 1 m
63
Use maximum-likelihood estimates
Smooth with the entire training set T
64
Estimating the relevance model of a
P(v|J): visterm distributions are sparse P(w|J): human annotations are incomplete
Use clustering to get better estimates
65
{cat, grass, tiger, water} {cat, grass, tiger} {water} {cat, grass, tiger, tree} {grass, tiger, water} {cat} Words in red are missing in the annotation
66
Cluster the training images using K-
Use both visterms and annotations
Compute the joint distribution of
Use clusters instead of individual images
C T
i 1 m
67
mAP
0.14
0.128
68
Smooth maximum likelihood estimates
P w J 1 a1 a2 c w , J J a1 c w ,C J C J a2 c w ,T T P v J 1 b1 b2 c v , J J b1 c v ,C J C J b2 c v ,T T
69
Optimize smoothing parameters
Divide training set
Find the best set of clusters
Query-dependent clusters Investigate soft clustering
70
Retrieval performance
Average precision and recall for one-word
71
72
Words Visterms Visterms Words
Translation (MT) models (Paola), Relevance Models (Shao Lei,Desislava), Graphical Models (Pavel, Brock) Text classification models (Matt) Integration & Summary (Dietrich)
73
tiger ground water grass
blocks and annotation words is a hidden variable, models are trained using the EM algorithm (HTK toolkit) Test HMM has |W| states, 2 scenarios: (a) p(w’|w) uniform (b) p(w’|w) from co-occurrence LM Posterior probability from forward-backward pass used for p(w|Image) Training HMMs:
training image – states given by manual annotations.
by annotation words
74
and image regions
is missed
later)
75
car transportation vehicle
non-studio setting nature-non-vegetation snow man-made object
image and they are not very suitable for describing visual properties
beach palm people tree
76
image regions
77
water,...)
individual emission probability distributions
distribution
more EM iterations
78
Results:
significantly
79
training
background and unlabelled foreground, with different strategies for parameter tying
80
Image features LM mAP No Yes No Yes 0.120 Discrete 0.150 0.140 Continuous (1 Gaussian per state) 0.157
81
Model LM mAP No Yes No Yes 0.094 1 Gaussian per state X 0.145 12 Gaussians per state X
Continuous features only, no language model
82
83
Very encouraging preliminary results
Effort started this summer, validated on Corel, and yielded
competitive annotation results on TREC
Initial findings
Proper normalization of the features is crucial for system
performance: bug found and fixed on Friday!
Simple HMMs seem to work best
More complex training topology didn’t really help More complex parameter tying was only marginally helpful
Glaring gaps
Need a good way to incorporate a language model
84
Brock Pytlik Johns Hopkins University bep@cs.jhu.edu
85
Graphical Models for Image Annotation
Hidden Markov Models
Two-Dimensional HMM’s
Joint Image Segmentation and Labeling
Tree Structure Models of Image
86
tiger ground water grass
water ground grass tiger
3
3
water ground grass tiger
2
2
1
water ground grass tiger
1
p(o |c) p(o |c) p(c |c
')
p(c |c
')
87
tiger ground water grass
water ground grass tiger
3
3
water ground grass tiger
2
2
1
water
1
p(o |c) p(o |c) p(c |c
')
p(c |c
')
88
tiger ground water grass
water ground grass tiger
3
3
water
2
2
1
water
1
p(o |c) p(o |c) p(c |c
')
p(c |c
')
89
tiger ground water grass tiger
3
3
water
2
2
1
water
1
p(o |c) p(o |c) ) | (
'
c c p p(c |c
')
90
An HMM for a 24-block Image
91
An HMM for a 24-block Image
92
An HMM for a 24-block Image
93
An HMM for a 24-block Image Transition probabilities represent spatial extent of objects
94
Transition probabilities represent spatial extent of objects A Two-Dimensional Model for a 24-block Image
95
Transition probabilities represent spatial extent of objects A Two-Dimensional Model for a 24-block Image
Model Training Time Per Image Training Time Per Iteration 1-D HMM .5 sec 37.5 min 2-D HMM 110 sec 8250 min = 137.5 hr
96
1 Constraint on
Ct
Ct
Tiger, Sky, Grass Mt
p(Mt =1) = 1 1 if ct ∈ tiger,grass,sky
{ }
⎧ ⎨ ⎩
97
Mt
(1) = Mt−1 (1) ∨(Ct = tiger)
Mt
(2) = Mt−1 (2) ∨(Ct = grass)
Only permit paths that visit every annotation word.
Ct
Mt
(3) = Mt−1 (3) ∨(Ct = sky)
Mt
(1)
Mt
(2)
Mt
(3)
98
Forward Decoding
99
Forward Decoding
1
c S N i i i
∋ =
100
Forward Decoding
1
S N i i i
=
1
c S N i i i
∋ =
101
Forward Decoding Viterbi Decoding
Path
1
S N i i i
=
1
c S N i i i
∋ =
102
Working with
mAP not yet
103
Improved Training for Two-Dimensional
separately
Other variations could be investigated
Next Idea
Joint Image Segmentation and Labeling
) | ( ) | ( ) , | (
, 1 , 1 1 , , 1 , j i j i j i j i j i
c c p c c p c c c p
− − − −
∝
104
tiger, grass, sky
105
tiger, grass, sky
106
tiger, grass, sky
107
tiger, grass, sky sky tiger grass
sky tiger grass
108
A Generative Model for Joint
Tree construction by agglomerative
tree
leaves
109
General Model
∈ ∈
=
)) ( tree ( cuts ) ( leaves
) | ) (
( ) , | ( ) ( ) , (
v
d u u l l l v
c l p l u c p u p d c p
110
General Model
∈ ∈
=
)) ( tree ( cuts ) ( leaves
) | ) (
( ) , | ( ) ( ) , (
v
d u u l l l v
c l p l u c p u p d c p
Probability of Cut
111
General Model
∈ ∈
=
)) ( tree ( cuts ) ( leaves
) | ) (
( ) , | ( ) ( ) , (
v
d u u l l l v
c l p l u c p u p d c p
Probability of Label Given Cut and Leaf
112
General Model
∈ ∈
=
)) ( tree ( cuts ) ( leaves
) | ) (
( ) , | ( ) ( ) , (
v
d u u l l l v
c l p l u c p u p d c p
Probability of Observation Given Label
113
General Model Independent Generation of Observations
∈ ∈
=
)) ( tree ( cuts ) ( leaves
) | ) (
( ) , | ( ) ( ) , (
v
d u u l l l v
c l p l u c p u p d c p
∈ ∈ ∈
)) tree( ( cuts ) ( leaves ) child
v
d u u l (l
l v
114
Suitable independence assumptions may
All cuts are equally likely? Given a cut, leaf labels have a Markov
Given a label, its image footprint is
Work out EM algorithm for this model
115
Uniform
Hypothesize number of segments produced
Greedy Choice
remaining that produces a valid segmentation
| | c
116
Uniform
Number of Children
Co-occurrence Language Model
exponential)
∈ ∈
⎥ ⎦ ⎤ ⎢ ⎣ ⎡ =
C a u m
a c p m a p l u c p ) | ( ) | ( ) , | (
) ( leaves 1 2
117
Label Generates its Observations
Parent Score
Label Generates Composite Observation at
118
Evaluate on Corel Image set using mAP TREC annotation task
119
120
121
Words Visterms Visterms Words
Translation (MT) models (Paola), Relevance Models (Shao Lei,Desislava), Graphical Models (Pavel, Brock) Text classification models (Matt) Integration & Summary (Dietrich)
122
123
<Word stime="177.09" dur="0.22" conf="0.727"> IT'S </Word> <Word stime="177.31" dur="0.25" conf="0.963"> MUCH </Word> <Word stime="177.56" dur="0.11" conf="0.976"> THE </Word> <Word stime="177.67" dur="0.29" conf="0.977"> SAME </Word> <Word stime="177.96" dur="0.14" conf="0.980"> IN </Word> <Word stime="178.10" dur="0.13" conf="0.603"> THE </Word> <Word stime="178.38" dur="0.57" conf="0.953"> SUMMERTIME </Word> <Word stime="178.95" dur="0.50" conf="0.976"> GLACIER </Word> <Word stime="179.45" dur="0.60" conf="0.974"> AVALANCHE </Word>
124
Assume there is a hidden
C C w v w w v w v
125
126
Insert Sentence Boundaries Case Restoration Noun Extraction Named Entity Detection WordNet Processing Feature Set
127
128
STEVE FOSSETT AND HIS BALLOON SOLO SPIRIT ARSENIDE.
OVER THE BLACK SEA DRIFTING SLOWLY TOWARDS THE COAST OF THE CAUCUSES. HIS TEAM PLANS IF NECESSARY TO BRING HIM DOWN AFTER DAYLIGHT TOMORROW. YOU THE CHECHEN CAPITAL OF GROZNY
129
Steve Fossett and his balloon Solo Spirit arsenide. Over the Black Sea drifting slowly towards the coast of the caucuses. His team plans if necessary to bring him down after daylight tomorrow. you the Chechan capital of Grozny….
130
Steve Fossett and his balloon Solo Spirit arsenide. Over the Black Sea drifting slowly towards the coast of the caucuses. His team plans if necessary to bring him down after daylight tomorrow. you the Chechan capital of Grozny.
131
Steve Fossett and his balloon Solo Spirit arsenide. Over the Black Sea drifting slowly towards the coast of the caucuses. His team plans if necessary to bring him down after daylight tomorrow. you the Chechan capital of Grozny.
132
Steve Fossett and his balloon Solo Spirit arsenide. Over the Black Sea drifting slowly towards the coast of the caucuses. His team plans if necessary to bring him down after daylight tomorrow. you the Chechan capital of Grozny.
caucus, team, daylight, Chechan, capital, Grozny
133
Basic feature set (nouns + NEs) has
6000 elements x {previous, this, next}
Using only a subset of the possible
Two strategies for feature selection:
Remove very rare words (18,000 7902) Eliminate low-value features
134
Measures the change in entropy given
∈
) (
F Values w
135
Basketball
1.
(empty)
2.
Location-city
3.
(empty) (previous)
4.
“game” (previous)
5.
“game”
6.
Person-male
7.
“point” (previous)
8.
“game” (next)
9.
“basketball (previous)
10.
“win”
11.
(empty) (next)
12.
“basketball”
13.
“point”
14.
“title” (previous)
15.
“win” (previous) Sky
1.
Person-male (previous)
2.
“car” (previous)
3.
Person
4.
Person-male
5.
“jury”
6.
Person (next)
7.
(empty) (next)
8.
“point”
9.
“report”
11.
“change” (previous)
136
0.56 0.565 0.57 0.575 0.58 250 750 1250 1750 2250 2750 3250 3750 4250 4750 5250 5750 6250 6750 7250 Number of Features AP
137
Naïve Bayes Decision Trees Support Vector Machines Voted Perceptrons Language Model
AdaBoosted Naïve Bayes & Decision Stumps
Maximum Entropy
138
Build a binary classifier
w w w
139
Conceptually similar to Naïve Bayes but
Multinomial Smoothed distributions Different feature selection
140
Binary constraints Single 75-concept model Ranked list of concepts for each shot.
141
0.1 0.2 0.3 0.4 0.5 0.6 AP
text non_studio face indoors
people person
Chance Lang Model Naïve Bayes MaxEnt
142
0.1 0.2 0.3 0.4 0.5 0.6 AP
weather basketball face sky indoors beach vehicle car
Chance Lang Model Naïve Bayes MaxEnt
143
0.02 0.04 0.06 0.08 0.1 0.12 0.14 AP
Chance Language Model SVM Naïve Bayes Max Ent
144
“Find shots of a person diving into some
person, water_body, non-studio_setting,
nature_non-vegetation, person_action, indoors
“Find shots of the front of the White House
building, outdoors, sky, water_body, cityscape,
house, nature_vegetation
“Find shots of Congressman Mark Souder.”
person, face, indoors, briefing_room_setting,
text_overlay
145
“Find shots of a person diving into some
person, water_body, non-studio_setting,
nature_non-vegetation, person_action, indoors
“Find shots of the front of the White House
building, outdoors, sky, water_body, cityscape,
house, nature_vegetation
“Find shots of Congressman Mark Souder.”
person, face, indoors, briefing_room_setting,
text_overlay
146
Concept Importance AP Chance
0.68 0.434 0.270 person 0.48 0.267 0.227 vehicle 0.36 0.106 0.043 man-made-obj. 0.28 0.190 0.156 sky 0.40 0.119 0.061 face 0.28 0.582 0.414 building 0.24 0.078 0.042 road 0.24 0.055 0.037 transportation 0.24 0.151 0.065 indoors 0.24 0.459 0.317
147
Predict visual concepts for ASR Tried Naïve Bayes, SVMs, MaxEnt,
Expect improvements in retrieval
148
149
TREC queries ask for:
specific persons specific places specific events specific locations
150
“Find shots of Congressman Mark Souder”
151
“Find shots of a graphic of Dow Jones
152
Find shots of the Tomb of the Unknown
153
154
Goal: Improve video OCR accuracy by
155
…. Sources tell C.N.N. there’s evidence
156
157
158
159
Text overlays provide high precision
160
Use existing tools and data from
161
Preprocessing
Normalize the text region’s height
Feature extraction
Color Edge Strength and Orientation
162
c1 c2 c3 c4 c5 c6
163
Augment the recognizers with an
Background language model Cache-based language model
“Interesting” words from the cache
3 2 1
λ λ λ
i interest i cache i bg i
164
Evaluate on TRECVID data Character Error Rate
Compare vs. manual transcriptions
Mean Average Precision
NIST-provided relevance judgments
165
Information from text overlays appears to
General character recognition is a Hard
Adding in external knowledge sources via the
166
1.
2.
3.
4.
167
Presented by Dietrich Klakow
168
Words Visterms Visterms Words
Translation (MT) models (Paola), Relevance Models (Shao Lei,Desislava), Graphical Models (Pavel, Brock) Text classification models (Matt) Integration & Summary (Dietrich)
169
v w v w
170
w w d
v w d
w v d
v v d
171
) |
v w d
p(q
) |
w w d
p(q
) |
w v d
p(q
) |
v v d
p(q
172
v w v v w w v w v w
v w w w w w v w w
Text Query Image Documents
173
v v v w v v v w w w w w v w v w
174
We want to estimate Assume pairwise marginals given: Setting: Maximum Entropy problem
4 constraints 1 iteration of GIS:
v w v w
, v w d q v w v w
w v
4 3 2 1
) | ( ) | ( ) | ( ) | ( ) |
λ λ λ λ v v w v v w w w v w v w
d q p d q p d q p d q p ,d d ,q p(q ∝
175
Retrieval mAP: 0.131
) |
w w d
p(q
Visterms dv Words dw Document Words qw Visterms qv Query Report best automatic run from literature (0.16)
176
) |
w w d
p(q
) |
v w d
p(q
Visterms dv Words dw Document Words qw Visterms qv Query
mAP: 0.131
177
Retrieval mAP: 0.139
) |
w w d
p(q
) |
v w d
p(q
Visterms dv Words dw Document Words qw Visterms qv Query
MT 0.126 Relevance Models 0.158 HMM 0.145 Concept Annotation
mAP on TRECVID
MT: Best overall performance so far MT
mAP: 0.131
178
Retrieval mAP: 0.149
MT
) |
w w d
p(q
) |
v w d
p(q
Visterms dv Words dw Document
) |
w v d
p(q
Words qw Visterms qv Query
Concepts from ASR: mAP=0.125 MT 0.126 Relevance Models 0.158 HMM 0.145 Concept Annotation
mAP on TRECVID
Best results reported in literature: retrieval mAP=0.162
mAP: 0.131
179
Improvements in high precision region
0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1 Precission Recall Best Basline
180
Annotations are
Inconsistent, sometimes abstract, …
Used plain vanilla features
Color, texture, edge on key-frames No time for exploration of alternatives
Uniform block segmentation of images Upper bound for concepts from ASR
181
in video
OCR-features
Desislava Shaolei and Brock Matt
182
183
184 From: http://www.nature.ca/notebooks/english/tiger.htm