Iden%fica%onofNarra%vePeaksin Clips:TextFeaturesPerformBest - - PowerPoint PPT Presentation

▶
iden fica on of narra ve peaks in clips text features
SMART_READER_LITE
LIVE PREVIEW

Iden%fica%onofNarra%vePeaksin Clips:TextFeaturesPerformBest - - PowerPoint PPT Presentation

Oct 07, 2022 23 likes •175 views

Iden%fica%onofNarra%vePeaksin Clips:TextFeaturesPerformBest JoepJ.M.Kierkels,MohammadSoleymani, ThierryPun ComputerVisionandMul2mediaLaboratory(h8p:// cvml.unige.ch)

slide-1
SLIDE 1

Iden%fica%on
of
Narra%ve
Peaks
in
 Clips:
Text
Features
Perform
Best



Joep
J.M.
Kierkels,
Mohammad
Soleymani,
 Thierry
Pun


Computer
Vision
and
Mul2media
Laboratory
(h8p:// cvml.unige.ch)


Computer
Science
Department
 University
of
Geneva
 Switzerland


15‐10‐2009
 1
 CLEF
2009,
Corfu


slide-2
SLIDE 2

Outline


  • Mo2va2on

  • Contest
and
task

  • Methods

  • Results

  • Conclusion
and
future
work


15‐10‐2009
 2
 CLEF
2009,
Corfu


slide-3
SLIDE 3

Mo2va2on 


Iden2fica2on
of
narra2ve
peaks
or
drama2c
 tension
moments 
 For
 
 Video
summariza2on
and
highligh2ng 


15‐10‐2009
 CLEF
2009,
Corfu
 3


slide-4
SLIDE 4

Task 


15‐10‐2009
 CLEF
2009,
Corfu
 4


No narrative peak Narrative peak

slide-5
SLIDE 5

Task 


15‐10‐2009
 CLEF
2009,
Corfu
 5


No narrative peak Narrative peak

slide-6
SLIDE 6

Video
features 


  • Frame
absolute
difference
(grayscale)

  • Thresholding
of
the
difference
score

  • Smoothed
over
10
seconds
long
window


15‐10‐2009
 CLEF
2009,
Corfu
 6


slide-7
SLIDE 7

Audio
features 


  • Pitch
increase

  • Neglec2ng
the
non‐

increasing
samples


  • Audio
Energy


(loudness)


  • Smoothed
over
10


second
long
window


15‐10‐2009
 CLEF
2009,
Corfu
 7


slide-8
SLIDE 8

Text
features 


  • The
introduc2on
of
a
new
topic

  • Removing
the
rare
and
frequent
words

  • Genera2on
of
the
vocabulary
vector
v


15‐10‐2009
 CLEF
2009,
Corfu
 8


slide-9
SLIDE 9

Narra2ve
peak
distribu2on 


15‐10‐2009
 CLEF
2009,
Corfu
 9


slide-10
SLIDE 10

Fusion 


  • Drama2c
tension
detec2on
based
on

  • The
first
three
peaks
with
the
distance
more


than
5
seconds
were
chosen


15‐10‐2009
 CLEF
2009,
Corfu
 10


slide-11
SLIDE 11

Fusion
and
train
set
results 


Scheme Features Weights Scheme Features Weights 1 Video Yes 5 Video,
Text Yes 2 Audio Yes 6 Audio,
Text Yes 3 Text Yes 7 Video,
Audio,
 Text Yes 4 Video,
 Audio Yes 8 Text No

15‐10‐2009
 CLEF
2009,
Corfu
 11
 Scheme number BG_36941 BG_37007 BG_37016 BG_37036 BG_37111

Total

1 1 1 1

3

2 2 1 1 1 1

6

3 2 1 1 2 1

7

4 1 2 1 1

5

5 1 2 2 1

6

6 2 1 1 2 1

7

7 1 1 2 1

5

8 1 1 1

3

slide-12
SLIDE 12

Results
on
evalua2on
set 


run number (scheme nr) Score (Peak- based) Score (Point- based)

1 3 33 39 2 7 30 41 3 6 33 42 4 8 32 43 5 Random 32 43

15‐10‐2009
 CLEF
2009,
Corfu
 12


slide-13
SLIDE 13

Conclusion
and
future
work 


  • Challenging
an
difficult
task!

  • Text
features
and
seman2cs

  • Having
all
the
peaks
will
improve
the
methods

  • Using
facial
expressions

  • Pitch
based
detec2on
with
all
the
drama2c


tension
moments


  • Larger
training
set,
supervised
learning


15‐10‐2009
 CLEF
2009,
Corfu
 13


slide-14
SLIDE 14

Machine
learning
protest 


15‐10‐2009
 CLEF
2009,
Corfu
 14


slide-15
SLIDE 15

Thank
you
for
your
a8en2on!
 Ques2ons,
comments,
sugges2ons


15‐10‐2009
 CLEF
2009,
Corfu
 15