Crowdsourced Automatic Zoom and Scroll for Video Retargeting Axel - - PowerPoint PPT Presentation

▶
crowdsourced automatic zoom and scroll for video
SMART_READER_LITE
LIVE PREVIEW

Crowdsourced Automatic Zoom and Scroll for Video Retargeting Axel - - PowerPoint PPT Presentation

Oct 30, 2023 141 likes •790 views

Crowdsourced Automatic Zoom and Scroll for Video Retargeting Axel Carlier, Wei Tsang Ooi NUS (Singapore) Vincent Charvillat, Romulus Grigoras, Geraldine Morin IRIT (Toulouse, France) 1 iPhone 4 Video retargeting: Making a large video fit

slide-1
SLIDE 1

Crowdsourced Automatic Zoom and Scroll for Video Retargeting

Axel Carlier, Wei Tsang Ooi NUS (Singapore) Vincent Charvillat, Romulus Grigoras, Geraldine Morin IRIT (Toulouse, France)

1

slide-2
SLIDE 2

2

Video retargeting: Making a large video fit into a smaller screen and available with network capacities.

iPhone 4

slide-3
SLIDE 3

One simple way:

Scale down the video to the resolution

  • f the screen

3

slide-4
SLIDE 4

Example

4

slide-5
SLIDE 5

Drawback:

Important details may not be visible

5

slide-6
SLIDE 6

What makes a good video retargeting ?

  • Good comprehension of the video content
  • The video is aesthetically satisfying

6

slide-7
SLIDE 7

7 Important region Zoom

slide-8
SLIDE 8

Idea: Zoom and scroll to show only interesting

regions of the video

8 8

slide-9
SLIDE 9

Problem: how to find the region automatically ?

9

slide-10
SLIDE 10

Approaches using Content Analysis

Saliency map Motion detection 10 Liu, Gleicher MM 06 Avidan, Shamir Commun. ACM 09 10

Too many regions

slide-11
SLIDE 11

Speech Recognition + Natural Language processing Object recognition

slide-12
SLIDE 12

Our idea: Crowdsourcing

Identifying regions of interest by gathering implicit input from users.

12 12

slide-13
SLIDE 13

Viewer traces Retargeted video Active Viewers Passive Viewers

slide-14
SLIDE 14

Use zoomable video

14 14

slide-15
SLIDE 15

Interface

15 15

slide-16
SLIDE 16

Example of user interaction

16 16

slide-17
SLIDE 17

Crowdsourcing

  • Tutorial: how to use the interface ?
  • Magic videos :

– HD Videos : 1920 × 1080 pixels – Fixed camera – Obvious ROIs : magician's hands, cards, dice...

  • Between 7 and 12 viewers for each video
  • 11,183 interaction events logged

17 17

slide-18
SLIDE 18

18 Heatmap ROIs Shots Shot 1 Shot 2 Final Video 18

Automatic Generation of Retargeted Video

slide-19
SLIDE 19

Building Heatmaps

19 19

slide-20
SLIDE 20

Building Heatmaps

20 20

slide-21
SLIDE 21

Building Heatmaps

21 21

slide-22
SLIDE 22

Analyzing Heatmaps

22 22 Here draw gaussians in 3d with matlab

GMM (Gaussian Mixture model)

  • K
  • wi
  • mi
  • ∑i

Mean Shift

slide-23
SLIDE 23

Finding Modes

Mean-Shift: Clustering algorithm

(Comaniciu, ICCV 02)

23 23

slide-24
SLIDE 24

Determining ROI size

24

Minimum Covariance Determinant (MCD)

24

slide-25
SLIDE 25

Building a ROI Dynamics Graph

slide-26
SLIDE 26

Cutting the graph into shots

slide-27
SLIDE 27

Shots selection

Shot 1 Shot 3 Shot 2

slide-28
SLIDE 28

Result video

28 28

slide-29
SLIDE 29

Integrating Reframing techniques

  • Bottom-up reframing
  • Type of shot: fixed, zooming or dolly
  • Shot level: stabilization according to its type
  • Inter-shot level: transitions and reestablishing shots

29 29

slide-30
SLIDE 30

Dolly shot

30 30

Shot 3

slide-31
SLIDE 31

Shots

31 31

slide-32
SLIDE 32

Shot stabilization

32 32

slide-33
SLIDE 33

Transitions

33 33

slide-34
SLIDE 34

Reestablishing shots

34 34

slide-35
SLIDE 35

Final Result

35 35

slide-36
SLIDE 36

Results validation

  • 3 poor videos:
  • User interaction (user)
  • Retargeted version without reframing techniques

(noRT)

  • Original version scaled down (nozoom)
  • Retargeted version with reframing techniques

(crowdsourced)

  • Ground truth (expert)

36 36

slide-37
SLIDE 37

Ground Truth

37 37

slide-38
SLIDE 38

Protocol

  • 48 participants divided into 3 categories
  • User – crowdsourced – expert (18)
  • NoRT – crowdsourced – expert (18)
  • Nozoom – crowdsourced – expert (12)
  • 3 questions were asked to the participants

38 38

slide-39
SLIDE 39

Rate the video editing of the video

39 1 2 3 4 5 39

NoRT = retargeted version without reframing techniques

slide-40
SLIDE 40

Is the video editing reasonable ?

40 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 40

slide-41
SLIDE 41

Does the video manage to convey important information ?

41 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 41

slide-42
SLIDE 42

Summary

  • Gather implicit input from users
  • No content analysis
  • In our examples : less than 12

viewers are enough to detect ROIs

slide-43
SLIDE 43

Future work

  • Explore alternative methods for intermediary

steps:

  • Modelling heatmaps not as a GMM
  • Adding cinematographic rules
  • Classify users into different profiles and generate a

retargeted video for each profile

43 43

slide-44
SLIDE 44

Questions

?

44 44

slide-45
SLIDE 45

Results

45 45

slide-46
SLIDE 46

Using aesthetics

Liu, Chen, Wolf and Cohen-Or. Optimizing Photo Composition, Computer Graphic Forum Luo, Yi wen and Tang, Xiaoou. Photo and Video Quality Evaluation : Focusing on the Subject, ECCV 08

46

slide-47
SLIDE 47

MCD Covariance

47

slide-48
SLIDE 48

MCD Covariance

48

slide-49
SLIDE 49

Gym Video Retargeted

49

slide-50
SLIDE 50

Shamma, Shaw, Shafton, Liu. Watch what I watch, MIR 07

Crowdsourcing

50

slide-51
SLIDE 51

Overview

  • Video retargeting
  • Zoomable video
  • Finding users' interests
  • Creating shots
  • Integrating reframing techniques
  • Results validation

54 54

slide-52
SLIDE 52

http://en.wikipedia.org/wiki/File:UHDV.svg

55 55

slide-53
SLIDE 53

iPhone 4 960 x 640

http://en.wikipedia.org/wiki/File:UHDV.svg

56 56

slide-54
SLIDE 54

57 57

slide-55
SLIDE 55

58 58

slide-56
SLIDE 56

Approaches using Content Analysis

Saliency map Motion detection 59 Liu, Gleicher MM 06 Avidan, Shamir Commun. ACM 09 59

slide-57
SLIDE 57

60 Crowdsourcing Heatmap Hotspots Shots Shot 1 Shot 2 Final Video 60

slide-58
SLIDE 58

Creating Heatmaps

  • Modelization of ROIs as a GMM (Gaussian

Mixture Model)

61 61

slide-59
SLIDE 59

Creating Heatmaps

  • Modelization of ROIs as a GMM (Gaussian

Mixture Model)

62 62

slide-60
SLIDE 60

Building a ROI dynamics graph

63 63

slide-61
SLIDE 61

Minimal spanning tree

64 64

slide-62
SLIDE 62

Cutting the tree into shots

65 65

slide-63
SLIDE 63

66

Shots selection

Shots are selected according to their popularity:

66