DAVVI and vESP: experimental systems for doing search in multimedia - PowerPoint PPT Presentation

DAVVI and vESP: experimental systems for doing search in multimedia collections Pål Halvorsen

Video streaming is everywhere !! University of Oslo TF-Media meeting, March 2011

Streaming systems today...  View a video sequentially timeline  Once produced, video never changes  Make and distribute the same video to everybody  Emerging: − topic based composition using search − personalized playlists − recommendation − integration with social networking − anytime, everywhere... University of Oslo TF-Media meeting, March 2011

Topic based composition Query: - premier league - April 2010 - Liverpool - Goal by Steven Gerrard - 30 seconds duration Users select events from multiple sources, played out as one video University of Oslo TF-Media meeting, March 2011

Personalized ordering Query: - premier league - April 2010 Users select events - Liverpool from multiple sources, - Goal by Steven Gerrard - 30 seconds duration arbitrary ordered , played out as one video Personalized video: University of Oslo TF-Media meeting, March 2011

Recommendations and social networking  Enabling user (re)publishing: − create directory service (user generated content) − social network  Recommendations: − recommend personalized content − user interest profile stored − match user profile against interesting content Personalized video: University of Oslo TF-Media meeting, March 2011

Streaming Solution Segmented Adaptive HTTP Streaming

Torrent-like HTTP streaming Video object:  For load-balancing and scaling multiple servers, taking the best from several worlds….  Downloads segments  Tracker manages information about segment locations  The user contacts the tracker for segment locations  Users send HTTP GET requests to download video segments University of Oslo TF-Media meeting, March 2011

Torrent-like HTTP streaming  Based on experiments, we use 2-second segments (2-hour movie  3600+ + small, indexed videos)  FFMPEG encoded: − H.264 (GOP = IP 48 I) − MP3 playout time − Custom made container  To support adaptation to quality available resources, each segment is coded in many quality levels University of Oslo TF-Media meeting, March 2011

DAVVI search and delivery of soccer events

DAVVI: Idea Present a 2-minutes video of highlights from last month games combined from • goals by Dirk Kuyt • sliding tackles • tip over the bar • … • TV broadcasters, etc. have huge repositories of sports content o full videos, short events, highlights etc. o should be searchable • Multimedia search and delivery systems still lacks precision and flexibility University of Oslo TF-Media meeting, March 2011

YouTube (or Google, Bing, …) University of Oslo TF-Media meeting, March 2011

VG Live University of Oslo TF-Media meeting, March 2011

DAVVI: system architecture Web-servers from live commentary BBC, Yahoo, VG, … live-text crawling video analysis transcoder/ chopper search / recommendation tracker feedback storage web-servers HTTP Get Video segments DAVVI University of Oslo TF-Media meeting, March 2011

Annotation: sports event analysis How can an event be identified??  Audio-video analysis is difficult - why so hard? − Video data at 25 fps, an event may last 2,000 frames − Variation among & within sport broadcasts − Complex video quality, camera angles and on screen graphics − Many different events to detect, e.g., in the context of soccer • yellow cards / red cards / goals / penalties / free kicks / fouls / corners / throw-ins / tackles / headings / passes / player numbers... − Identify the beginning and end of the event − Find all the events – not miss any, no false positives − Computationally expensive University of Oslo TF-Media meeting, March 2011

Annotation: sports event analysis  Event identification figures: − Huang et al. (U. Illinois) – text & video analysis (2000): 57% − Hanjalic et al. (Delft U.)– audio based analysis (2002): 52% − Sadlier et al. (DCU) – audio & video analysis (2005): 64%  Initial evaluation of visual/aural approach we developed for iAD: 67% - 83% University of Oslo TF-Media meeting, March 2011

Annotation: live text commentaries news.bbc.co.uk:  Many online TV-stations and newspapers provide live text commentaries  DAVVI uses a semi-automatic live-text crawler and parser to improve the automatic annotations uk.eurosport.yahoo.com: University of Oslo TF-Media meeting, March 2011

Search and recommendation  Solr/Lucene open-source search engine which has indexed the videos  Users can query for video clips using a rich set of keywords, specifying values for tags or as free text  Each result is returned as a playlist of video segments, and playlists can be combined to make an topic-based, personalized video  The playlist describing the personalized video can be submitted to the social network University of Oslo TF-Media meeting, March 2011

DAVVI demo system automatically generate an X-minute playlist search box search results – horizontally scrollable textual playlist generated description of by drag-and-drop the event or automatically which can be generated expanded video each clip player quality can be controls indicator adjusted University of Oslo TF-Media meeting, March 2011

vESP search and delivery of talks/lectures

vESP: Idea Present a video of explanations about TCP congestion control techniques combining slides from talks/lectures given by • Van Jacobson • Vinton G. Cerf • Mark Allman • Jitendra Padhye • … • Companies/Schools/Universities/etc. have huge repositories of presentation content o presentations, training videos, etc. o must be searchable o part of the content is multimedia • Enterprise multimedia search still lacks precision and flexibility University of Oslo TF-Media meeting, March 2011

YouTube University of Oslo TF-Media meeting, March 2011

Altus vPresenter / vSearch University of Oslo TF-Media meeting, March 2011

TalkMiner University of Oslo TF-Media meeting, March 2011

vESP platform talk transcript indexing slide indexing video analysis transcoder/ chopper search tracker storage web-servers HTTP Get Video segments University of Oslo TF-Media meeting, March 2011

vESP platform Slides Query: “Windows 7” Search index Transcript Search results Slide timing Slid ide Sta tart End nd num number 1 0:00 2:34 2 2:34 5:43 Custom presentation Video server Video of presentation University of Oslo TF-Media meeting, March 2011

vESP custom video presentation PPT-file A, slide 3 PPT-file D, slide 2 PPT-file C, slide 4 Video-file: A, seg. 161 - 198 Video-file: D, seg. 20 - 28 Video-file: C, seg. 40 - 61 University of Oslo TF-Media meeting, March 2011

vESP demo system slide playlist generated by drag-and-drop or automatically generated document preview video presentation select slides for playlist University of Oslo TF-Media meeting, March 2011

Summary  DAVVI and vESP are scalable prototypes that − − give you a new way to access video content − integrate video streaming, search, personalization and recommendation (with social networking potential) − well evaluated by subjective assessment group  Our next generation systems aim for 3D and free-view video experiences University of Oslo TF-Media meeting, March 2011

Questions?? Comments?? Contact information: Pål Halvorsen paalh@ifi.uio.no http://home.ifi.uio.no/paalh University of Oslo TF-Media meeting, March 2011