Iden%fica%onofNarra%vePeaksin Clips:TextFeaturesPerformBest - PowerPoint PPT Presentation

Iden%fica%on
of
Narra%ve
Peaks
in
 Clips:
Text
Features
Perform
Best
 
Joep
J.M.
Kierkels,
Mohammad
Soleymani,
 Thierry
Pun
 Computer
Vision
and
Mul2media
Laboratory
(h8p:// cvml.unige.ch)
 Computer
Science
Department
 University
of
Geneva
 Switzerland
 15‐10‐2009
 CLEF
2009,
Corfu
 1


Outline
 • Mo2va2on
 • Contest
and
task
 • Methods
 • Results
 • Conclusion
and
future
work
 15‐10‐2009
 CLEF
2009,
Corfu
 2


Mo2va2on 
 Iden2fica2on
of
narra2ve
peaks
or
drama2c
 tension
moments 
 For
 
 Video
summariza2on
and
highligh2ng 
 15‐10‐2009
 CLEF
2009,
Corfu
 3


Task 
 No narrative peak Narrative peak 15‐10‐2009
 CLEF
2009,
Corfu
 4


Task 
 No narrative peak Narrative peak 15‐10‐2009
 CLEF
2009,
Corfu
 5


Video
features 
 • Frame
absolute
difference
(grayscale)
 • Thresholding
of
the
difference
score
 • Smoothed
over
10
seconds
long
window
 15‐10‐2009
 CLEF
2009,
Corfu
 6


Audio
features 
 • Pitch
increase
 • Neglec2ng
the
non‐ increasing
samples
 • Audio
Energy
 (loudness)
 • Smoothed
over
10
 second
long
window
 15‐10‐2009
 CLEF
2009,
Corfu
 7


Text
features 
 • The
introduc2on
of
a
new
topic
 • Removing
the
rare
and
frequent
words
 • Genera2on
of
the
vocabulary
vector
 v 
 15‐10‐2009
 CLEF
2009,
Corfu
 8


Narra2ve
peak
distribu2on 
 15‐10‐2009
 CLEF
2009,
Corfu
 9


Fusion 
 • Drama2c
tension
detec2on
based
on
 • The
first
three
peaks
with
the
distance
more
 than
5
seconds
were
chosen
 15‐10‐2009
 CLEF
2009,
Corfu
 10


Fusion
and
train
set
results 
 Scheme Features Weights Scheme Features Weights 1 Video Yes 5 Video,
Text Yes 2 Audio Yes 6 Audio,
Text Yes 3 Text Yes 7 Video,
Audio,
 Yes Text 4 Video,
 Yes 8 Text No Audio BG_36941 BG_37007 BG_37016 BG_37036 BG_37111 Total Scheme number 1 0 0 1 1 1 3 2 2 1 1 1 1 6 3 2 1 1 2 1 7 4 0 1 2 1 1 5 5 1 2 2 1 0 6 6 2 1 1 2 1 7 7 1 1 2 1 0 5 8 0 1 1 1 0 3 15‐10‐2009
 CLEF
2009,
Corfu
 11


Results
on
evalua2on
set 
 run number (scheme nr) Score (Peak- Score (Point- based) based) 1 3 33 39 2 7 30 41 3 6 33 42 4 8 32 43 5 Random 32 43 15‐10‐2009
 CLEF
2009,
Corfu
 12


Conclusion
and
future
work 
 • Challenging
an
difficult
task!
 • Text
features
and
seman2cs
 • Having
all
the
peaks
will
improve
the
methods
 • Using
facial
expressions
 • Pitch
based
detec2on
with
all
the
drama2c
 tension
moments
 • Larger
training
set,
supervised
learning
 15‐10‐2009
 CLEF
2009,
Corfu
 13


Machine
learning
protest 
 15‐10‐2009
 CLEF
2009,
Corfu
 14


Thank
you
for
your
a8en2on!
 Ques2ons,
comments,
sugges2ons
 15‐10‐2009
 CLEF
2009,
Corfu
 15