AXES @ TRECVid MED 2013 Matthijs Douze 1 , Zaid Harchaoui 1 , Dan - - PowerPoint PPT Presentation

▶
axes trecvid med 2013
SMART_READER_LITE
LIVE PREVIEW

AXES @ TRECVid MED 2013 Matthijs Douze 1 , Zaid Harchaoui 1 , Dan - - PowerPoint PPT Presentation

Dec 10, 2023 263 likes •721 views

AXES @ TRECVid MED 2013 Matthijs Douze 1 , Zaid Harchaoui 1 , Dan Oneat a 1 , Danila Potapov 1 , J ome Revaud 1 , Cordelia Schmid 1 , er Jochen Schwenninger 2 , Jakob Verbeek 1 , Heng Wang 1 1 INRIALEAR, Grenoble, France 2 Fraunhofer

slide-1
SLIDE 1

AXES @ TRECVid MED 2013

Matthijs Douze1, Zaid Harchaoui1, Dan Oneat ¸˘ a1, Danila Potapov1, J´ erˆ

  • me Revaud1, Cordelia Schmid1,

Jochen Schwenninger2, Jakob Verbeek1, Heng Wang1

1INRIA–LEAR, Grenoble, France 2Fraunhofer Sankt Augustin, Germany 1 / 16

slide-2
SLIDE 2

Outline

Low-level features Encoding High-level features Classification SIFT Color MFCC Improved trajectories Spatial Fisher vector Spatial Fisher vector Fisher vector Fisher vector OCR ASR Bag-of- words Bag-of- words Classifier Classifier Classifier Classifier Classifier Classifier +

2 / 16

slide-3
SLIDE 3

Outline

Low-level features Encoding High-level features Classification SIFT Color MFCC Improved trajectories Spatial Fisher vector Spatial Fisher vector Fisher vector Fisher vector OCR ASR Bag-of- words Bag-of- words Classifier Classifier Classifier Classifier Classifier Classifier +

2 / 16

slide-4
SLIDE 4

Outline

Low-level features Encoding High-level features Classification SIFT Color MFCC Improved trajectories Spatial Fisher vector Spatial Fisher vector Fisher vector Fisher vector OCR ASR Bag-of- words Bag-of- words Classifier Classifier Classifier Classifier Classifier Classifier +

2 / 16

slide-5
SLIDE 5

Outline

Low-level features Encoding High-level features Classification SIFT Color MFCC Improved trajectories Spatial Fisher vector Spatial Fisher vector Fisher vector Fisher vector OCR ASR Bag-of- words Bag-of- words Classifier Classifier Classifier Classifier Classifier Classifier +

2 / 16

slide-6
SLIDE 6

Outline

Low-level features Encoding High-level features Classification SIFT Color MFCC Improved trajectories Spatial Fisher vector Spatial Fisher vector Fisher vector Fisher vector OCR ASR Bag-of- words Bag-of- words Classifier Classifier Classifier Classifier Classifier Classifier +

2 / 16

slide-7
SLIDE 7

Outline

Low-level features Encoding High-level features Classification SIFT Color MFCC Improved trajectories Spatial Fisher vector Spatial Fisher vector Fisher vector Fisher vector OCR ASR Bag-of- words Bag-of- words Classifier Classifier Classifier Classifier Classifier Classifier +

2 / 16

slide-8
SLIDE 8

Table of Contents

1

Low-level features: static, motion, audio

2

Feature encoding: Fisher vector

3

High-level features

4

Experiments and results

3 / 16

slide-9
SLIDE 9

Static and audio features

Scale-invariant feature transform (SIFT, Lowe 2004) Mel-frequency cepstral coefficients (MFCC, Rabiner and Schafer 2007)

4 / 16

slide-10
SLIDE 10

Static and audio features

Scale-invariant feature transform (SIFT, Lowe 2004) Mel-frequency cepstral coefficients (MFCC, Rabiner and Schafer 2007) Color descriptors (Clinchant et al., 2007).

µ, σ µ, σ µ, σ µ, σ

Mean and variance. . . 2 . . . of RGB values. . . 3 . . . in 4 × 4 cells 16 Descriptor dimensionality 96

4 / 16

slide-11
SLIDE 11

Improved motion features (Wang and Schmid, ICCV, 2013)

Builds upon dense trajectory features (?, CVPR, ?)

Trajectory description HOG MBH HOF Tracking in each spatial scale separately Dense sampling in each spatial scale

5 / 16

slide-12
SLIDE 12

Improved motion features (Wang and Schmid, ICCV, 2013)

Builds upon dense trajectory features (?, CVPR, ?) Dense trajectories can be affected by camera motion.

Trajectory description HOG MBH HOF Tracking in each spatial scale separately Dense sampling in each spatial scale

5 / 16

slide-13
SLIDE 13

Improved motion features (Wang and Schmid, ICCV, 2013)

Idea: stabilize camera motion before computing optical flow.

6 / 16

slide-14
SLIDE 14

Improved motion features (Wang and Schmid, ICCV, 2013)

Idea: stabilize camera motion before computing optical flow. Method:

1

extract feature points (SURF descriptors and dense optical flow)

2

match feature points and estimate homography with RANSAC

3

warp the optical flow.

6 / 16

slide-15
SLIDE 15

Improved motion features (Wang and Schmid, ICCV, 2013)

Idea: stabilize camera motion before computing optical flow.

Two succesive frames

7 / 16

slide-16
SLIDE 16

Improved motion features (Wang and Schmid, ICCV, 2013)

Idea: stabilize camera motion before computing optical flow.

Two succesive frames Optical flow

7 / 16

slide-17
SLIDE 17

Improved motion features (Wang and Schmid, ICCV, 2013)

Idea: stabilize camera motion before computing optical flow.

improves flow estimation

Two succesive frames Optical flow Warped optical flow

7 / 16

slide-18
SLIDE 18

Improved motion features (Wang and Schmid, ICCV, 2013)

Idea: stabilize camera motion before computing optical flow.

improves flow estimation removes background tracks.

Two succesive frames Optical flow Warped optical flow Removed trajectories

7 / 16

slide-19
SLIDE 19

Removed trajectories under various camera motions

8 / 16

slide-20
SLIDE 20

Table of Contents

1

Low-level features: static, motion, audio

2

Feature encoding: Fisher vector

3

High-level features

4

Experiments and results

9 / 16

slide-21
SLIDE 21

Fisher vector for appearance

Generalization of the bag-of-words. Strong performance across multiple tasks:

action recognition, action detection, event recognition

(Oneat ¸˘ a et al., ICCV, 2013)

10 / 16

slide-22
SLIDE 22

Fisher vector for appearance

Generalization of the bag-of-words. Strong performance across multiple tasks:

action recognition, action detection, event recognition

(Oneat ¸˘ a et al., ICCV, 2013)

image classification (Chatfield et al., BMVC, 2011) image retrieval (J´

egou et al., PAMI, 2012)

fine-grained image classification (Gavves et al., ICCV, 2013) face verification (Simonyan et al., BMVC, 2013) word spotting (Almaz´

an et al., ICCV, 2013).

10 / 16

slide-23
SLIDE 23

Fisher vector for location

Spatial Fisher vector (SFV)

(Krapac et al., ICCV, 2011)

encodes first and second moments

  • f visual word locations

adds 6 entries for each visual word: µ and σ for (x, y, t) coordinates.

Schematic illustration of the spatial Fisher vector for three types of visual words ( , , ) in an image.

11 / 16

slide-24
SLIDE 24

Fisher vector for location

Spatial Fisher vector (SFV)

(Krapac et al., ICCV, 2011)

encodes first and second moments

  • f visual word locations

adds 6 entries for each visual word: µ and σ for (x, y, t) coordinates.

Schematic illustration of the spatial Fisher vector for three types of visual words ( , , ) in an image.

11 / 16

slide-25
SLIDE 25

Fisher vector for location

Spatial Fisher vector (SFV)

(Krapac et al., ICCV, 2011)

encodes first and second moments

  • f visual word locations

adds 6 entries for each visual word: µ and σ for (x, y, t) coordinates.

Compared to spatial pyramids:

(Oneat ¸˘ a et al., ICCV, 2013)

similar performance gain

Schematic illustration of the spatial Fisher vector for three types of visual words ( , , ) in an image.

11 / 16

slide-26
SLIDE 26

Fisher vector for location

Spatial Fisher vector (SFV)

(Krapac et al., ICCV, 2011)

encodes first and second moments

  • f visual word locations

adds 6 entries for each visual word: µ and σ for (x, y, t) coordinates.

Compared to spatial pyramids:

(Oneat ¸˘ a et al., ICCV, 2013)

similar performance gain SFV are more compact

Schematic illustration of the spatial Fisher vector for three types of visual words ( , , ) in an image.

11 / 16

slide-27
SLIDE 27

Fisher vector for location

Spatial Fisher vector (SFV)

(Krapac et al., ICCV, 2011)

encodes first and second moments

  • f visual word locations

adds 6 entries for each visual word: µ and σ for (x, y, t) coordinates.

Compared to spatial pyramids:

(Oneat ¸˘ a et al., ICCV, 2013)

similar performance gain SFV are more compact complementary.

Schematic illustration of the spatial Fisher vector for three types of visual words ( , , ) in an image.

11 / 16

slide-28
SLIDE 28

Table of Contents

1

Low-level features: static, motion, audio

2

Feature encoding: Fisher vector

3

High-level features

4

Experiments and results

12 / 16

slide-29
SLIDE 29

High-level features: OCR and ASR

Optical character recognition (OCR) Automatic speech recognition (ASR) (from Fraunhofer IAIS)

trained on 100 hours of English broadcasts language model trained on news articles and patents

For both systems:

bag-of-words encoding with 110, 000 words. tf-idf weighting ℓ2 normalization.

13 / 16

slide-30
SLIDE 30

Low-level features Encoding High-level features Classification SIFT Color MFCC Improved trajectories Spatial Fisher vector Spatial Fisher vector Fisher vector Fisher vector OCR ASR Bag-of- words Bag-of- words Classifier Classifier Classifier Classifier Classifier Classifier +

14 / 16

slide-31
SLIDE 31

Table of Contents

1

Low-level features: static, motion, audio

2

Feature encoding: Fisher vector

3

High-level features

4

Experiments and results

15 / 16

slide-32
SLIDE 32

Initial experiments on TRECVid ’11 subset

16 / 16

slide-33
SLIDE 33

Initial experiments on TRECVid ’11 subset

Spatial Fisher vectors improve for color and SIFT.

16 / 16

slide-34
SLIDE 34

Initial experiments on TRECVid ’11 subset

Spatial Fisher vectors improve for color and SIFT. Comparison of the motion features (HOG, HOF, MBH):

16 / 16

slide-35
SLIDE 35

Initial experiments on TRECVid ’11 subset

Spatial Fisher vectors improve for color and SIFT. Comparison of the motion features (HOG, HOF, MBH):

MBH > HOG > HOF

16 / 16

slide-36
SLIDE 36

Initial experiments on TRECVid ’11 subset

Spatial Fisher vectors improve for color and SIFT. Comparison of the motion features (HOG, HOF, MBH):

MBH > HOG > HOF HOG+MBH > HOF+MBH

16 / 16

slide-37
SLIDE 37

Initial experiments on TRECVid ’11 subset

Spatial Fisher vectors improve for color and SIFT. Comparison of the motion features (HOG, HOF, MBH):

MBH > HOG > HOF HOG+MBH > HOF+MBH The combination of all the three channels is the best.

16 / 16

slide-38
SLIDE 38

Initial experiments on TRECVid ’11 subset

Spatial Fisher vectors improve for color and SIFT. Comparison of the motion features (HOG, HOF, MBH):

MBH > HOG > HOF HOG+MBH > HOF+MBH The combination of all the three channels is the best.

SIFT descriptors are complementary to the motion features.

16 / 16

slide-39
SLIDE 39

Initial experiments on TRECVid ’11 subset

Spatial Fisher vectors improve for color and SIFT. Comparison of the motion features (HOG, HOF, MBH):

MBH > HOG > HOF HOG+MBH > HOF+MBH The combination of all the three channels is the best.

SIFT descriptors are complementary to the motion features. Total processing time was 27 times slower than real-time on a single core.

Overview of our system: descriptors’ dimensions and processing time.

×Real Modality Descriptor Encoding D time Motion HOG+HOF+MBH FV+H3 51k 10 Image SIFT FV+SFV 34k 2 Image Color FV+SFV 73k 10 Audio MFCC FV 20k 0.05 Image OCR BoW (sparse) 110k 1.5 Audio ASR BoW (sparse) 110k 3

16 / 16

slide-40
SLIDE 40

Results on TRECVid ’11 data

Comparison to our earlier systems.

DCR mAP Best TV’11 0.437 AXES 2011 0.642 AXES 2012 0.411 44.5 AXES 2013 0.379 52.6

17 / 16

slide-41
SLIDE 41

Results on TRECVid ’11 data

Comparison to our earlier systems. Performance for individual channels

DCR mAP Best TV’11 0.437 AXES 2011 0.642 AXES 2012 0.411 44.5 AXES 2013 0.379 52.6 Motion + SIFT 46.4 Color 27.7 Audio 18.2 ASR 8.2 OCR 10.8

17 / 16

slide-42
SLIDE 42

Results on TRECVid ’13 data

MED pre-specified MED ad-hoc Team mAP Team mAP AXES (1/15) 34.6 AXES (1/14) 36.6 BBNVISER (2/15) 33.0 CMU (2/14) 36.3 median 24.7 median 23.3 MED results, for the PROGAll, 100Ex challenge.

18 / 16

slide-43
SLIDE 43

Results on TRECVid ’13 data

MED pre-specified MED ad-hoc Team mAP Team mAP AXES (1/15) 34.6 AXES (1/14) 36.6 BBNVISER (2/15) 33.0 CMU (2/14) 36.3 median 24.7 median 23.3 MED results, for the PROGAll, 100Ex challenge. Team Full system ASR Audio OCR Visual AXES 36.6 1.0 12.4 1.1 29.4 BBNVISER 32.2 8.0 15.1 5.3 23.4 CMU 36.3 5.7 16.1 3.7 28.4 Genie 20.2 4.3 10.1 — 16.9 IBM-Columbia 2.8 — 0.2 — 2.8 MediaMill 25.3 — 5.6 — 23.8 NII 24.9 — 8.8 — 19.9 ORAND 3.8 — — — 3.8 PicSOM 0.6 — 0.1 — 0.6 SRIAURORA 24.2 3.9 9.6 4.3 20.4 Sesame 25.7 3.9 5.6 0.2 23.2 VisQMUL 0.2 — 0.2 — 0.2 Per-channel results on the MED ad-hoc 100Ex, challenge.

18 / 16

slide-44
SLIDE 44

Conclusions

Key components of our system:

Improved motion features Spatial Fisher vector.

Code available on our web-site

http://lear.inrialpes.fr/software

Check out our posters:

Action recognition with improved trajectories. Action and event recognition with Fisher vectors on a compact feature set.

19 / 16