Annotation of Video and Film Matthias Zeppelzauer St. Plten - - PowerPoint PPT Presentation

Automated Analysis, Retrieval and Annotation of Video and Film

Matthias Zeppelzauer

• St. Pölten University of Applied Sciences

Symposium: Film Rechnen Computerbasierte Methoden in der Filmanalyse

July 3, 2017

Motivation

Manual annotation = time-consuming & tedious task

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Motivation

Manual annotation = time-consuming & tedious task Goal: Annotate content automatically!

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Automated Film Analysis

Outline

• Methods for automated film analysis
• Temporal segmentation
• Motion composition retrieval
• Visual composition retrieval
• Montage analysis
• Outline of future research

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Temporal Segmentation

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Step 1: Shot Segmentation

• Shot = continuous sequence of frames recorded from a

single camera

• Basic building block for high-level film analysis
• Motion analysis, montage patterns, rhythm analysis, …
• Shot boundary types
• Abrupt transitions (shot cuts)
• Gradual transitions (dissolves, fades. …)

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

... ...

Step 1: Shot Segmentation

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

time similarity of frame 400 and frame 250 400 250 time 7

Step 2: Scene Segmentation

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Step 2: Scene Segmentation

• Bottom-up approach
• Basis: shots
• Audio + visual similarity  links
• Fuse linked shots into scenes

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Scene 1 Scene 2

Motion Composition Retrieval

• Retrieve scenes with

particular compositions

• f camera and object motion
• Step 1: motion tracking and

segmentation

• Step 2: retrieval of motion

compositions

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Step 1: Motion Tracking and Segmentation

• Motion trajectories  clustering  motion components

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Time Y X

Step 2: Retrieve Motion Compositions

use segments for retrieval!

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Step 2: Retrieve Motion Compositions

• Goal: Search and retrieval of user-defined motion

compositions

• Input = Query: Sketch motion components as vectors
• Camera motion, object motion, groups of objects

pan/large object diagonal motion e.g. hammering

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Step 2: Retrieve Motion Compositions

people, horses, tractors move diagonally hammering, working people, trumpeter traveling right,

• bject to left,

group to left

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Motion Continuity Retrieval

• “Motion Continuity refers to the

matching of individual scenic elements from shot to shot so that details and actions, filmed at different times will edit together without error”

• Consistent screen direction
• Example: chasing scene
• Retrieve user-specified combinations of motion continuity

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Retrieval of Visual Composition

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Retrieval of Visual Composition

• Visual composition = spatial arrangement of visual

elements inside an image

• Manual search: time consuming + subjective
• Automatic retrieval possible?

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Retrieval of Visual Composition

• Global image analysis
• User-study for evaluation:
• User define composition to search for..

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Retrieval of Visual Composition

• Example results

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at) July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at) 20

Montage Analysis

• Synchronous Montage:
• Correlations between

soundtrack and cutting of movie

• Stylistic means to highlight

important scenes and events

• Goal: extract sequences

automatically

• Benefit: basis for video summarization, abstraction,

annotation…

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Extraction of Synchronous Montage Sequences

• Correlate shot boundaries with audio onsets

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Extraction of Synchronous Montage Sequences

• Step 1: Shot cut detection
• Step 2: Audio analysis
• Step 3: Correlation extraction
• Step 4: Sequence extraction

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Extraction of Synchronous Montage Sequences - Results

• Example
• “October”

(Eisenstein, 1928): protest on the street

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Extraction of Synchronous Montage Sequences

• Retrieved sequences: dialogue sequences, action scenes,

parallel montage  rich semantics

• Applications:
• highlight extraction
• scene segmentation
• summarization

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Conclusion

• Automatic film analysis has great potential, e.g.
• Temporal segmentation  extraction of shots / scenes
• Motion composition retrieval  retrieve typical camera / object

motions

• Visual composition retrieval  find similar image compositions /

framings

• Synchronous montage extraction  extract highlights / montage

patterns / cross-cutting

July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Thank you for your Attention!

Contact: Matthias.Zeppelzauer@fhstp.ac.at

Questions?