TRECVID-2005 Low-level (camera motion) feature task Wessel Kraaij - - PowerPoint PPT Presentation

▶
trecvid 2005 low level camera motion feature task
SMART_READER_LITE
LIVE PREVIEW

TRECVID-2005 Low-level (camera motion) feature task Wessel Kraaij - - PowerPoint PPT Presentation

Sep 19, 2022 29 likes •329 views

TRECVID-2005 Low-level (camera motion) feature task Wessel Kraaij TNO & Tzveta Ianeva NIST Task definition TRECVID 2005 pilot task Ability to detect camera movement features: q Pan (left or right ) or track q Tilt (up or down) or

slide-1
SLIDE 1

TRECVID-2005 Low-level (camera motion) feature task

Wessel Kraaij

TNO

& Tzveta Ianeva NIST

slide-2
SLIDE 2

TRECVID 2005 2

Task definition

TRECVID 2005 pilot task Ability to detect camera movement features:

q Pan (left or right ) or track q Tilt (up or down) or boom q Zoom (in or out) or dolly

slide-3
SLIDE 3

TRECVID 2005 3

Task definition ...

Camera movement features are usually combined

q Pan & Tilt q Pan & Zoom q Tilt & Zoom

slide-4
SLIDE 4

TRECVID 2005 4

Task definition ...

q Pan & Tilt & Zoom

Submissions provide complete judgments for test set by specifying all shots identified as positive by the system No Training data provided by NIST Tool to create development data developed by Werner Bailer at Joanneum Researh

slide-5
SLIDE 5

TRECVID 2005 5

Ground truth creation at NIST

Watch randomly chosen subset of test data (~5000 shots) Keep only shots with “clear” examples of (no) motion (~2226) No-motion shots seem to more clearly exhibit no motion than shots with motion features exhibit motionŁ #FP will tend to be small, #FN will tend to be high Define test subset for each feature by combining

shots exhibiting the feature shots exhibiting no motion (same for all features)

No adjustments to subset sizes or true:false ratios

Pan 587:1159 Tilt 210:1159 Zoom 511:1159

slide-6
SLIDE 6

TRECVID 2005 6

Truth data distribution (number of shots)

Pan Tilt Zoom No motion

1159 587 210 511

slide-7
SLIDE 7

TRECVID 2005 7

Truth and evaluation issues

Why feature groups? Perceptual limits in truth creation Cost of creating truth data Many shots with lots of small camera movement – not what’s wanted when user asks for a “pan”, etc.

Implications of test set construction on measures

Lack of randomness makes generalization hard Varying true:false ratios make precision harder for tilt than pan and zoom Greater clarity of no-motion shots would make false positive less likely then false negatives and higher precision easier to achieve than higher recall

slide-8
SLIDE 8

TRECVID 2005 8

No motion shots

slide-9
SLIDE 9

TRECVID 2005 9

Truth data costly to create – lot’s of shaky shots

Hard to judge Not what a user wants

slide-10
SLIDE 10

TRECVID 2005 10

12 Participating Groups

Carnegie Mellon University ( CMU ) - USA City University of Hong Kong ( CUHK ) - China Fudan University ( FUDAN ) - China Institute for Infocomm Research ( IIR ) - Singapore JOANNEUM RESEARCH ( Joanneum ) - Austria KDDI & R&D Laboratories, Inc. ( KDDI ) - Japan LaBRI ( LaBRI ) - France Tsinghua University ( Tsinghua ) - China University of Central Florida / University of Modena ( UCF ) – USA/Italy University of Iowa ( Uiowa ) - USA University of Marburg ( MARBURG ) - Germany

  • Univ. of Amsterdam & TNO ( UvA ) - Netherlands
slide-11
SLIDE 11

TRECVID 2005 11

NIST baseline runs All features true for all shots (TrueForAllShots) Random run with true distribution of Pan, Tilt, Zoom as in truth data (TruthDataDistrib) Features randomly true/false for each shot

(Random)

slide-12
SLIDE 12

TRECVID 2005 12

Evaluation Measures

Precision = Recall =

# True positives # True positives + # False positives # True positives # True positives + # False negatives Given the imbalance in class properties, it’s easier to achieve a high precision than a high recall. The use of F =1 seems not appropriate

slide-13
SLIDE 13

TRECVID 2005 13

Pan: recall and precision by system

1 1 1 1 1 1 2 2 2 3 3 3 3 2 3 4 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 1 1 1 4 4 4 4 1 1 1 1 1 1 1 1 2 2 2 2 3 3 3 3 3 3 3 3 3 2 3 3 2 4 4 4 4 4

  • !"#
  • $%

&'("

  • )

%* + &",(- &"-(--. /#

NIST

slide-14
SLIDE 14

TRECVID 2005 14

Pan: recall and precision by system (zoomed)

1 1 1 1 1 1 2 2 2 3 3 3 3 2 3 4 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 1 1 1 4 4 4 4 1 1 1 1 1 1 1 1 2 2 2 2 3 3 3 3 3 3 3 3 3 2 3 3 2 4 4 4 4 4

  • !"#
  • $%

&'("

  • )

%* + &",(- &"-(--. /#

NIST

slide-15
SLIDE 15

TRECVID 2005 15

Tilt: recall and precision by system

1 1 1 1 1 1 2 2 2 3 3 3 3 2 3 4 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 1 1 1 4 4 4 4 1 1 1 1 1 1 1 1 2 2 2 2 3 3 3 3 3 3 3 3 3 2 3 3 2 4 4 4 4 4

  • !"#
  • $%

&'("

  • )

%* + &",(- &"-(-. /# NIST

slide-16
SLIDE 16

TRECVID 2005 16

Tilt: recall and precision by system (zoomed)

1 1 1 1 1 1 2 2 2 3 3 3 3 2 3 4 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 1 1 1 4 4 4 4 1 1 1 1 1 1 1 1 2 2 2 2 3 3 3 3 3 3 3 3 3 2 3 3 2 4 4 4 4 4

  • !"#
  • $%

&'("

  • )

%* + &",(- &"-(-. /# NIST

slide-17
SLIDE 17

TRECVID 2005 17

Zoom: recall and precision by system

1 1 1 1 1 1 2 2 2 3 3 3 3 2 3 4 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 1 1 1 4 4 4 4 1 1 1 1 1 1 1 1 2 2 2 2 3 3 3 3 3 3 3 3 3 2 3 3 2 4 4 4 4 4

  • !"#
  • $%

&'("

  • )

%* + &",(- &"-(--. /# NIST

slide-18
SLIDE 18

TRECVID 2005 18

Zoom: recall and precision by system (zoomed)

1 1 1 1 1 1 2 2 2 3 3 3 3 2 3 4 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 1 1 1 4 4 4 4 1 1 1 1 1 1 1 1 2 2 2 2 3 3 3 3 3 3 3 3 3 2 3 3 2 4 4 4 4 4

  • !"#
  • $%

&'("

  • )

%* + &",(- &"-(--. /# NIST

slide-19
SLIDE 19

TRECVID 2005 19

Mean recall and precision over all 3 features by system

1 1 1 1 1 1 2 2 2 3 3 3 3 2 3 4 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 1 1 1 4 4 4 4 1 1 1 1 1 1 1 1 2 2 2 2 3 3 3 3 3 3 3 3 3 2 3 3 2 4 4 4 4 4

  • !"#
  • $%

&'"

  • )

%* + &",(- &"-(--. /# NIST

slide-20
SLIDE 20

TRECVID 2005 20

Mean recall and precision over all 3 features by system (zoomed)

1 1 1 1 1 1 2 2 2 3 3 3 3 2 3 4 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 1 1 1 4 4 4 4 1 1 1 1 1 1 1 1 2 2 2 2 3 3 3 3 3 3 3 3 3 2 3 3 2 4 4 4 4 4

  • !"#
  • $%

&'"

  • )

%* + &",(- &"-(--. /# NIST

slide-21
SLIDE 21

TRECVID 2005 21

General points

  • NIST did not provide training data: some training

data was available from other sources and some training data was produced by participants

  • Input:

n MPEG motion vectors: optimal for compression, not

  • ptimal for modeling real motion

n Frame to frame motion analysis

  • Distinguish “jitter” from intended motion
slide-22
SLIDE 22

TRECVID 2005 22

CMU

  • Approach
  • 1. Probabilistic model (fitted using EM) based on MPEG

motion vectors

  • 2. Optical Flow model: extract the most consistent motion from

the optical flows (frame to frame)

  • !"#
  • $%

&'"

  • )

%* + &",(- &"-(--. /#

2 1

slide-23
SLIDE 23

TRECVID 2005 23

CUHK

  • Approach

n Motion features extracted from tracking image features in consecutive frames n Estimation of 6 parameter affine model, transformation in p,t,z vector for each set of adjacent frames n Rule based motion classification using empirical thresholds n Interesting failure analysis

  • !"#
  • $%

&'"

  • )

%* + &",(- &"-(--. /#

slide-24
SLIDE 24

TRECVID 2005 24

Fudan

  • Approach

n Motion vectors from MPEG,SVM, motion accumulation method to filter out imperceptable movements n Filter method seems to decrease precision though…

  • !"#
  • $%

&'"

  • )

%* + &",(- &"-(--. /#

slide-25
SLIDE 25

TRECVID 2005 25

Joanneum

  • presentation follows -
  • Approach

n Developed a training set , problems with annotation.. n Feature tracking, clustering trajectories, dominant cluster selection, camera motion detection, thresholding

  • !"#
  • $%

&'"

  • )

%* + &",(- &"-(--. /#

slide-26
SLIDE 26

TRECVID 2005 26

IIR

  • Approach

n Annotated 24 video files n Estimated affine camera model based on MPEG motion vectors n Transformation of model parametersŁ series of p,t,z values for each shot n Rule based classification of series using accumulation and thresholding

  • !"#
  • $%

&'"

  • )

%* + &",(- &"-(--. /#

slide-27
SLIDE 27

TRECVID 2005 27

LaBRI

  • presentation follows -
  • Approach

n Mpeg motion vector inputŁ 6 parameter affine model n Jitter suppression (statistical significance test) n Subshot segmentation (homogeneous motion) n Motion classification (using “a few annotated videos”)

  • !"#
  • $%

&'"

  • )

%* + &",(- &"-(--. /#

slide-28
SLIDE 28

TRECVID 2005 28

Marburg

  • Approach

n 3D camera model estimated from MPEG motion vectors n Cleaning necessary, + exlusion of center, frame border n Optimal thresholds estimated on tv2005 training set

  • !"#
  • $%

&'"

  • )

%* + &",(- &"-(--. /#

slide-29
SLIDE 29

TRECVID 2005 29

  • !"#
  • $%

&'"

  • )

%* + &",(- &"-(--. /#

Tsinghua

  • Approach

n Motion vector selection based spatial features, separating camera motion from object motion and accidental motion n 4 parameter camera model (Iterative Least Squares) parameter estimation n Rule based classification (FSA), using a range of thresholds for: 1.Continuous (speed) and noticable, 2,Minumum duration 3.Uninterrupted 4.Noticable in case in combination with other camera movement

slide-30
SLIDE 30

TRECVID 2005 30

Observations

This is clearly an easier task than the HLF task, though a high recall is hard to achieve. Truth data costly to create – lot’s of shaky shots

Many hard to judge Many not really what a user wants when s/he asks for a “pan” etc.

Hard to generalize from small, constructed test subset to larger, more realistic test set Given the definition of our task and test set characteristics, F measure not appropriate Concentrate on within-feature system comparisons