Crowdsourced Automatic Zoom and Scroll for Video Retargeting Axel - PowerPoint PPT Presentation

Crowdsourced Automatic Zoom and Scroll for Video Retargeting Axel Carlier, Wei Tsang Ooi NUS (Singapore) Vincent Charvillat, Romulus Grigoras, Geraldine Morin IRIT (Toulouse, France) 1

iPhone 4 Video retargeting: Making a large video fit into a smaller screen and available with network capacities. 2

One simple way: Scale down the video to the resolution of the screen 3

Example 4

Drawback: Important details may not be visible 5

What makes a good video retargeting ? - Good comprehension of the video content - The video is aesthetically satisfying 6

Important region Zoom 7

Idea : Zoom and scroll to show only interesting regions of the video 8 8

Problem: how to find the region automatically ? 9

Approaches using Content Analysis Saliency map Motion detection Too many regions Liu, Gleicher MM 06 Avidan, Shamir Commun. ACM 09 10 10

Speech Recognition + Natural Language processing Object recognition

Our idea: Crowdsourcing Identifying regions of interest by gathering implicit input from users. 12 12

Viewer Retargeted traces video Passive Active Viewers Viewers

Use zoomable video 14 14

Interface 15 15

Example of user interaction 16 16

Crowdsourcing ● Tutorial: how to use the interface ? ● Magic videos : – HD Videos : 1920 × 1080 pixels – Fixed camera – Obvious ROIs : magician's hands, cards, dice... ● Between 7 and 12 viewers for each video ● 11,183 interaction events logged 17 17

Automatic Generation of Retargeted Video Heatmap ROIs Shot 2 Shots Shot 1 Final Video 18 18

Building Heatmaps 19 19

Building Heatmaps 20 20

Building Heatmaps 21 21

Analyzing Heatmaps Here draw gaussians in 3d with matlab GMM (Gaussian Mixture model) -K -w i Mean Shift -m i - ∑ i 22 22

Finding Modes Mean-Shift: Clustering algorithm (Comaniciu, ICCV 02) 23 23

Determining ROI size Minimum Covariance Determinant (MCD) 24 24

Building a ROI Dynamics Graph

Cutting the graph into shots

Shots selection Shot 1 Shot 3 Shot 2

Result video 28 28

Integrating Reframing techniques ● Bottom-up reframing ● Type of shot: fixed, zooming or dolly ● Shot level: stabilization according to its type ● Inter-shot level: transitions and reestablishing shots 29 29

Dolly shot Shot 3 30 30

Shots 31 31

Shot stabilization 32 32

Transitions 33 33

Reestablishing shots 34 34

Final Result 35 35

Results validation ● 3 poor videos: ● User interaction ( user ) ● Retargeted version without reframing techniques ( noRT ) ● Original version scaled down ( nozoom ) ● Retargeted version with reframing techniques ( crowdsourced ) ● Ground truth ( expert ) 36 36

Ground Truth 37 37

Protocol ● 48 participants divided into 3 categories ● User – crowdsourced – expert (18) ● NoRT – crowdsourced – expert (18) ● Nozoom – crowdsourced – expert (12) ● 3 questions were asked to the participants 38 38

Rate the video editing of the video 5 4 3 2 1 0 39 39 NoRT = retargeted version without reframing techniques

Is the video editing reasonable ? 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 40 40

Does the video manage to convey important information ? 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 41 41

Summary -Gather implicit input from users -No content analysis - In our examples : less than 12 viewers are enough to detect ROIs

Future work ● Explore alternative methods for intermediary steps: ● Modelling heatmaps not as a GMM ● Adding cinematographic rules ● Classify users into different profiles and generate a retargeted video for each profile 43 43

Questions ? 44 44

Results 45 45

Using aesthetics Liu, Chen, Wolf and Cohen-Or. Optimizing Photo Composition, Computer Graphic Forum Luo, Yi wen and Tang, Xiaoou. Photo and Video Quality Evaluation : Focusing on the Subject, ECCV 08 46

MCD Covariance 47

MCD Covariance 48

Gym Video Retargeted 49

Crowdsourcing Shamma, Shaw, Shafton, Liu. Watch what I watch, MIR 07 50

Overview ● Video retargeting ● Zoomable video ● Finding users' interests ● Creating shots ● Integrating reframing techniques ● Results validation 54 54

55 55 http://en.wikipedia.org/wiki/File:UHDV.svg

960 x 640 iPhone 4 56 56 http://en.wikipedia.org/wiki/File:UHDV.svg

57 57

58 58

Approaches using Content Analysis Saliency map Motion detection Liu, Gleicher MM 06 Avidan, Shamir Commun. ACM 09 59 59

Crowdsourcing Heatmap Hotspots Shot 2 Shots Shot 1 Final Video 60 60

Creating Heatmaps ● Modelization of ROIs as a GMM (Gaussian Mixture Model) 61 61

Creating Heatmaps ● Modelization of ROIs as a GMM (Gaussian Mixture Model) 62 62

Building a ROI dynamics graph 63 63

Minimal spanning tree 64 64

Cutting the tree into shots 65 65

Shots selection Shots are selected according to their popularity: 66 66