DAVVI and vESP: experimental systems for doing search in multimedia - - PowerPoint PPT Presentation

DAVVI and vESP:

experimental systems for doing search in multimedia collections Pål Halvorsen

TF-Media meeting, March 2011

University of Oslo

Video streaming is everywhere!!

TF-Media meeting, March 2011

University of Oslo

Streaming systems today...

• View a video sequentially
• Once produced, video never changes
• Make and distribute the same video to everybody

 Emerging:

− topic based composition using search − personalized playlists − recommendation − integration with social networking − anytime, everywhere...

timeline

TF-Media meeting, March 2011

University of Oslo

Topic based composition

Query:

• premier league
• April 2010
• Liverpool
• Goal by Steven Gerrard
• 30 seconds duration

Users select events from multiple sources, played out as one video

TF-Media meeting, March 2011

University of Oslo

Personalized video:

Personalized ordering

Query:

• premier league
• April 2010
• Liverpool
• Goal by Steven Gerrard
• 30 seconds duration

Users select events from multiple sources, arbitrary ordered, played out as one video

TF-Media meeting, March 2011

University of Oslo

Recommendations and social networking

Personalized video:

• Enabling user (re)publishing:

− create directory service

(user generated content)

− social network

• Recommendations:

− recommend personalized content − user interest profile stored − match user profile against

interesting content

Streaming Solution

Segmented Adaptive HTTP Streaming

TF-Media meeting, March 2011

University of Oslo

Torrent-like HTTP streaming

• For load-balancing and scaling

multiple servers, taking the best from several worlds….

• Downloads segments
• Tracker manages information

about segment locations

• The user contacts the tracker

for segment locations

• Users send HTTP GET requests to

download video segments

Video object:

TF-Media meeting, March 2011

University of Oslo

Torrent-like HTTP streaming

playout time quality

• Based on experiments,

we use 2-second segments

(2-hour movie  3600+ + small, indexed videos)

• FFMPEG encoded:

− H.264 (GOP = IP48I) − MP3 − Custom made container

• To support adaptation to

available resources, each segment is coded in many quality levels

DAVVI

search and delivery of soccer events

TF-Media meeting, March 2011

University of Oslo

DAVVI: Idea

Present a 2-minutes video of highlights from last month games combined from

• goals by Dirk Kuyt
• sliding tackles
• tip over the bar
• …
• TV broadcasters, etc. have huge

repositories of sports content

• full videos, short events, highlights etc.
• should be searchable
• Multimedia search and delivery systems

still lacks precision and flexibility

TF-Media meeting, March 2011

University of Oslo

YouTube (or Google, Bing, …)

TF-Media meeting, March 2011

University of Oslo

VG Live

TF-Media meeting, March 2011

University of Oslo

Web-servers from BBC, Yahoo, VG, … video analysis

live-text crawling

live commentary DAVVI search / recommendation

feedback

transcoder/ chopper

HTTP Get Video segments

storage web-servers tracker

DAVVI: system architecture

TF-Media meeting, March 2011

University of Oslo

Annotation: sports event analysis

• Audio-video analysis is difficult - why so hard?

− Video data at 25 fps, an event may last 2,000 frames − Variation among & within sport broadcasts − Complex video quality, camera angles and on screen graphics − Many different events to detect, e.g., in the context of soccer

• yellow cards / red cards / goals / penalties / free kicks / fouls / corners / throw-ins / tackles / headings / passes / player numbers...

− Identify the beginning and end of the event − Find all the events – not miss any, no false positives − Computationally expensive

How can an event be identified??

TF-Media meeting, March 2011

University of Oslo

Annotation: sports event analysis

• Event identification figures:

− Huang et al. (U. Illinois) – text & video analysis (2000): 57% − Hanjalic et al. (Delft U.)– audio based analysis (2002): 52% − Sadlier et al. (DCU) – audio & video analysis (2005): 64%

• Initial evaluation of visual/aural approach we

developed for iAD: 67% - 83%

TF-Media meeting, March 2011

University of Oslo

Annotation: live text commentaries

• Many online TV-stations and newspapers

provide live text commentaries

• DAVVI uses a semi-automatic live-text crawler

and parser to improve the automatic annotations

uk.eurosport.yahoo.com: news.bbc.co.uk:

TF-Media meeting, March 2011

University of Oslo

Search and recommendation

• Solr/Lucene open-source search

engine which has indexed the videos

• Users can query for video clips using a rich set of keywords,

specifying values for tags or as free text

• Each result is returned as a playlist of video segments, and

playlists can be combined to make an topic-based, personalized video

• The playlist describing the personalized video can be submitted

to the social network

TF-Media meeting, March 2011

University of Oslo

DAVVI demo system

search box automatically generate an X-minute playlist search results – horizontally scrollable video quality indicator player controls playlist generated by drag-and-drop

• r automatically

generated each clip can be adjusted textual description of the event which can be expanded

vESP

search and delivery of talks/lectures

TF-Media meeting, March 2011

University of Oslo

vESP: Idea

Present a video of explanations about TCP congestion control techniques combining slides from talks/lectures given by

• Van Jacobson
• Vinton G. Cerf
• Mark Allman
• Jitendra Padhye
• …
• Companies/Schools/Universities/etc. have huge

repositories of presentation content

• presentations, training videos, etc.
• must be searchable
• part of the content is multimedia
• Enterprise multimedia search still lacks

precision and flexibility

TF-Media meeting, March 2011

University of Oslo

YouTube

TF-Media meeting, March 2011

University of Oslo

Altus vPresenter / vSearch

TF-Media meeting, March 2011

University of Oslo

TalkMiner

TF-Media meeting, March 2011

University of Oslo

transcoder/ chopper talk transcript indexing

HTTP Get Video segments

storage web-servers tracker search video analysis slide indexing

vESP platform

TF-Media meeting, March 2011

University of Oslo

vESP platform

Slid ide num number Sta tart End nd 1 0:00 2:34 2 2:34 5:43

Query: “Windows 7” Slides Transcript Slide timing Video of presentation Search index Video server Custom presentation Search results

TF-Media meeting, March 2011

University of Oslo

vESP custom video presentation

PPT-file A, slide 3 PPT-file D, slide 2

Video-file: A, seg. 161 - 198

PPT-file C, slide 4

Video-file: D, seg. 20 - 28 Video-file: C, seg. 40 - 61

TF-Media meeting, March 2011

University of Oslo

vESP demo system

slide playlist generated by drag-and-drop or automatically generated video presentation select slides for playlist document preview

TF-Media meeting, March 2011

University of Oslo

Summary

• DAVVI and vESP are scalable prototypes that

− give you a new way to access video content − integrate video streaming, search, personalization and recommendation

(with social networking potential)

− well evaluated by subjective assessment group

• Our next generation systems aim for

3D and free-view video experiences

TF-Media meeting, March 2011

University of Oslo

Questions?? Comments??

Contact information:

Pål Halvorsen

paalh@ifi.uio.no http://home.ifi.uio.no/paalh