Condensed Movies: Story Based Retrieval with Contextual Embeddings - PowerPoint PPT Presentation

Condensed Movies: Story Based Retrieval with Contextual Embeddings Max Bain, Arsha Nagrani, Andrew Brown, Andrew Zisserman

Motivation • Long term video understanding • Context Billy reveals the truth to Louis about the Duke's bet which changed both their lives.

Motivation • Long term video understanding • Context Billy reveals the truth to Louis about the Duke's bet which changed both their lives.

Motivation • Long term video & narrative understanding • Context ��������������������������� ��������������������������������� �������������������������

Text-Video Retrieval ������������������������ ������������ ������������������� ����������� ����������������� ������ ������� ������������������� ������� ������ ����������������

Contextual Text-Video Retrieval

Condensed Movies • 34K captioned clips, from 3.4K movies • 1.3K hours video • Freely available • Pre extracted 6 expert features • 400K facetracks, 8K characters • Ordered 2 minute clips. 1 2 3 … 8 9 10

Semantic Captions

Despite illegal moves from the Cobra Kai that bring him to the ground, Daniel (Ralph Macchio) delivers one final kick and wins the championship.

Overview • Context boosts retrieval • Contextual task utilizing long term narratives

Future work • Improved temporal aggregation for better 2-minute clip embeddings • Other tasks to test long-term video understanding

Thank you • Paper, dataset, features and code can be found at: • https://www.robots.ox.ac.uk/~vgg/research/condensed-movies