Stanford I2V: A News Video Dataset for Query-by-Image Experiments - - PowerPoint PPT Presentation

▶
stanford i2v a news video dataset for query by image
SMART_READER_LITE
LIVE PREVIEW

Stanford I2V: A News Video Dataset for Query-by-Image Experiments - - PowerPoint PPT Presentation

Jun 19, 2023 403 likes •671 views

Stanford I2V: A News Video Dataset for Query-by-Image Experiments Andr Araujo, J. Chaves, D. Chen, R. Angst, B. Girod Stanford University Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments. 1 Motivation Example:

slide-1
SLIDE 1

1 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Stanford I2V: A News Video Dataset for Query-by-Image Experiments

André Araujo, J. Chaves, D. Chen, R. Angst, B. Girod Stanford University

slide-2
SLIDE 2

2 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Retrieval System

NBC, 11/18/2014, 7:35:33 PM

Motivation

Logo or product Example: Brand Monitoring

slide-3
SLIDE 3

3 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Motivation

Retrieval System

KDTV, 01/18/2013, 6:41:45PM Example: Content Linking

slide-4
SLIDE 4

4 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Retrieval System

CS246, lecture 12 December 2, 2013 Presentation slide

Motivation

Example: Lecture search

slide-5
SLIDE 5

5 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Online demo http://videosearch.stanford.edu

slide-6
SLIDE 6

6 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Outline

  • Related Work
  • Stanford I2V Dataset
  • Dataset Construction
  • Baseline Experiments
slide-7
SLIDE 7

7 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Outline

  • Related Work
  • Stanford I2V Dataset
  • Dataset Construction
  • Baseline Experiments
slide-8
SLIDE 8

8 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Related Work: Visual Search Query Database

Image Video Videos Images

FV, Jégou et al., 2012 SVT, Nistér et al., 2006 SIFT, Lowe, 2004 TCD, Makar et al., 2012 Location Rec., Takacs et al., 2010 Frame Mat. + ST, Douze et al., 2010 TRECVID-CCD, Over et al., 2012

I2I: Traditional Visual Search V2I: Augmented Reality V2V: Content Tracking

BoW, Sivic et al., 2006

I2V: Video Search by Image

TRECVID-INS, Over et al., 2014 TAPS, Araujo et al., 2014

slide-9
SLIDE 9

9 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Related Work: Existing I2V Datasets

Dataset Size # Queries Sivic et al., Video-Google, 2006 2h 164

slide-10
SLIDE 10

10 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Related Work: Existing I2V Datasets

Dataset Size # Queries Sivic et al., Video-Google, 2006 2h 164 Over et al., TRECVID-INS, 2014 464h 30

slide-11
SLIDE 11

11 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Related Work: Existing I2V Datasets

Dataset Size # Queries Sivic et al., Video-Google, 2006 2h 164 Over et al., TRECVID-INS, 2014 464h 30 Araujo et al., CNN2h, 2014 2h 139

slide-12
SLIDE 12

12 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Related Work: Existing I2V Datasets

Dataset Size # Queries Sivic et al., Video-Google, 2006 2h 164 Over et al., TRECVID-INS, 2014 464h 30 Araujo et al., CNN2h, 2014 2h 139 Araujo et al., Stanford I2V, 2015 3,801h 229

slide-13
SLIDE 13

13 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Outline

  • Related Work
  • Stanford I2V Dataset
  • Dataset Construction
  • Baseline Experiments
slide-14
SLIDE 14

14 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Stanford I2V Dataset

Query images Database videos (selected frames)

slide-15
SLIDE 15

15 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Stanford I2V Dataset

Full version Light version 3.8k hours 1k hours 84k video clips 23k video clips 229 query images 78 query images 14M keyframes@1fps 3.8M keyframes@1fps 2.7 minutes/clip 2.65 minutes/clip

slide-16
SLIDE 16

16 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Evaluation Procedure

1st stage: Retrieval of Clips 2nd stage: Temporal Refinement Ranked retrieval measures:

  • Average Precision (AP)
  • Precision at 1 (p@1)

Unranked retrieval measure:

  • Temporal Jaccard Index

1 2 3 … … …

Query

System

slide-17
SLIDE 17

17 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Query/Annotation Viewer

Query image Clip 1 Clip 2

slide-18
SLIDE 18

18 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Outline

  • Related Work
  • Stanford I2V Dataset
  • Dataset Construction
  • Baseline Experiments
slide-19
SLIDE 19

19 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Dataset Construction: Video Collection

Video clips News Videos Recording Story Segmentation

Website

Daneshi et al., 2013

slide-20
SLIDE 20

20 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Dataset Construction: Query Set Collection

  • Collected images from news websites
  • Used the Internet Archive Wayback Machine
  • Collected 805 candidate images from dates between October 1st 2012

and September 30th 2013

  • Types of images:
  • Iconic images (events in the news)
  • Magazine covers (Time, Economist)
slide-21
SLIDE 21

21 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Dataset Construction: Annotation

Query image Query date

  • Jan. 7th, 2013

Select all videos within 1 week of query date Approve matches manually Global signature matching to entire database

Reject query if no approved matches

Feature-based matching + RANSAC

Accept query if there are approved matches

Select matches manually Match query against each frame individually Annotation

  • f video

sequences

slide-22
SLIDE 22

22 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Outline

  • Related Work
  • Stanford I2V Dataset
  • Dataset Construction
  • Baseline Experiments
slide-23
SLIDE 23

23 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Example: Evaluation of Standard Technique

  • SIFT descriptors + SCFV global signatures

[Lowe, 2004] [Duan et al., 2014]

  • Retrieval of Clips evaluation:
  • Compare query signature to video frames’ signatures (@1fps) from

entire database

  • Evaluate performance over top 100 ranked clips
  • Temporal Refinement evaluation:
  • Compare query signature to video frames’ signatures (@1fps) from

each correct matching video

  • Feature matching + RANSAC between query and top 50 frames

(consider a match if at least 8 inliers are found)

  • Evaluate Jaccard index between matches and ground-truth segments
slide-24
SLIDE 24

24 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Example: Evaluation of Standard Technique

10 20 30 40 50 60 25 30 35 40 45 50

mAP (%) mRetLatency (secs)

Light version Full version

Latency (secs) mAP (%)

30 ¡ 32 ¡ 34 ¡ 36 ¡ 38 ¡ 40 ¡ 42 ¡ 44 ¡ 128 ¡ 192 ¡ 256 ¡ 512 ¡

Light ¡ Full ¡ Number of Gaussians mJac (%) Retrieval of Clips: results Temporal Refinement results

slide-25
SLIDE 25

25 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Summary

  • Dataset for video retrieval using query images
  • 3.8k hours of video and 229 queries – largest dataset yet
  • First dataset to allow true large-scale experiments in this area
  • Experiments using standard image retrieval technique were

presented, serving as a baseline for future evaluations

slide-26
SLIDE 26

26 Araujo et al., Stanford I2V: A News Video Dataset for Query-by-Image Experiments.

Thank you! Questions? Dataset webpage:

http://blackhole1.stanford.edu/vidsearch/dataset/stanfordi2v.html

Online demo:

http://videosearch.stanford.edu

André Araujo

http://stanford.edu/~afaraujo afaraujo@stanford.edu