Condensed Movies: Story Based Retrieval with Contextual Embeddings - - PowerPoint PPT Presentation

▶

Aug 26, 2023 143 likes •284 views

Condensed Movies: Story Based Retrieval with Contextual Embeddings Max Bain, Arsha Nagrani, Andrew Brown, Andrew Zisserman Motivation Long term video understanding Context Billy reveals the truth to Louis about the Duke's bet which

SLIDE 1

Condensed Movies: Story Based Retrieval

with Contextual Embeddings

Max Bain, Arsha Nagrani, Andrew Brown, Andrew Zisserman

SLIDE 2

Motivation

Long term video understanding
Context

Billy reveals the truth to Louis about the Duke's bet which changed both their lives.

SLIDE 3

Motivation

Long term video understanding
Context

Billy reveals the truth to Louis about the Duke's bet which changed both their lives.

SLIDE 4

Motivation

Long term video & narrative understanding
Context

SLIDE 5

Text-Video Retrieval

SLIDE 6

Contextual Text-Video Retrieval

SLIDE 7

34K captioned clips, from 3.4K movies
1.3K hours video
Freely available
Pre extracted 6 expert features
400K facetracks, 8K characters
Ordered 2 minute clips.

1 2 3 … 8 9 10

Condensed Movies

SLIDE 8

Semantic Captions

SLIDE 9

Despite illegal moves from the Cobra Kai that bring him to the ground, Daniel (Ralph Macchio) delivers one final kick and wins the championship.

SLIDE 10

Overview

Context boosts retrieval
Contextual task utilizing long term narratives

SLIDE 11

Future work

Improved temporal aggregation for better 2-minute clip

embeddings

Other tasks to test long-term video understanding

SLIDE 12

Thank you

Paper, dataset, features and code can be found at:
https://www.robots.ox.ac.uk/~vgg/research/condensed-movies