Selectively De-Animating Video Jiamin Bai, Aseem Agarwala, Maneesh - - PowerPoint PPT Presentation

▶
selectively de animating video
SMART_READER_LITE
LIVE PREVIEW

Selectively De-Animating Video Jiamin Bai, Aseem Agarwala, Maneesh - - PowerPoint PPT Presentation

Aug 24, 2023 166 likes •998 views

Selectively De-Animating Video Jiamin Bai, Aseem Agarwala, Maneesh Agrawala, Ravi Ramamoorthi SIGGRAPH 2012 CS 448V: Computational Video Manipulation Inspiration http://cinemagraphs.com/ Cinemagraphs De-Animating Video Example Walkthrough

slide-1
SLIDE 1

Selectively De-Animating Video

Jiamin Bai, Aseem Agarwala, Maneesh Agrawala, Ravi Ramamoorthi SIGGRAPH 2012

CS 448V: Computational Video Manipulation

slide-2
SLIDE 2

Inspiration

http://cinemagraphs.com/

slide-3
SLIDE 3

Cinemagraphs

slide-4
SLIDE 4

De-Animating Video

slide-5
SLIDE 5

Example Walkthrough

slide-6
SLIDE 6

Example Walkthrough

slide-7
SLIDE 7

Cinemagraphs

slide-8
SLIDE 8

System Diagram

slide-9
SLIDE 9

System Diagram

slide-10
SLIDE 10

System Diagram

slide-11
SLIDE 11

Warping: Tracking

s = track index t = time (frame number) ta = reference frame

K (s, t) = set of tracks as a table of 2D coordinates KG (s, t) = subset of tracks that lie on the user indicated region K’G (s, t) = locations of tracks after warping K’G (s, t) = KG (s, ta) KG = KA ∪ KF

slide-12
SLIDE 12

Warping: Tracking

K (s, t) = set of tracks as a table of 2D coordinates KG (s, t) = subset of tracks that lie on the user indicated region K’G (s, t) = locations of tracks after warping K’G (s, t) = KG (s, ta) KG = KA ∪ KF

s = track index t = time (frame number) ta = reference frame

slide-13
SLIDE 13

Warping: Tracking

K (s, t) = set of tracks as a table of 2D coordinates KG (s, t) = subset of tracks that lie on the user indicated region K’G (s, t) = locations of tracks after warping K’G (s, t) = KG (s, ta) KG = KA ∪ KF

s = track index t = time (frame number) ta = reference frame

slide-14
SLIDE 14

Warping: Tracking

K (s, t) = set of tracks as a table of 2D coordinates KG (s, t) = subset of tracks that lie on the user indicated region K’G (s, t) = locations of tracks after warping K’G (s, t) = KG (s, ta) KG = KA ∪ KF

s = track index t = time (frame number) ta = reference frame

anchor tracks

slide-15
SLIDE 15

Warping: Tracking

s = track index t = time (frame number) ta = reference frame

K (s, t) = set of tracks as a table of 2D coordinates KG (s, t) = subset of tracks that lie on the user indicated region K’G (s, t) = locations of tracks after warping K’G (s, t) = KG (s, ta) KG = KA ∪ KF

slide-16
SLIDE 16

Warping: Tracking

s = track index t = time (frame number) ta = reference frame

K (s, t) = set of tracks as a table of 2D coordinates KG (s, t) = subset of tracks that lie on the user indicated region K’G (s, t) = locations of tracks after warping K’G (s, t) = KG (s, ta) KG = KA ∪ KF floating tracks

slide-17
SLIDE 17

System Diagram

slide-18
SLIDE 18

Warping: Initial Warp

E = Ea + ωEs

slide-19
SLIDE 19

Warping: Initial Warp

E = Ea + ωEs

main constraint

slide-20
SLIDE 20

Warping: Initial Warp

E = Ea + ωEs

shape-preserving

slide-21
SLIDE 21

Warping: Initial Warp

E = Ea + ωEs

slide-22
SLIDE 22

Warping: Initial Warp

E = Ea + ωEs

slide-23
SLIDE 23

Warping: Initial Warp

E = Ea + ωEs

K’A (s, t)

slide-24
SLIDE 24

Warping: Initial Warp

E = Ea + ωEs

weighting function

slide-25
SLIDE 25

System Diagram

slide-26
SLIDE 26

Warping: Refined Warp

E = Ea + Ef + ωEs

slide-27
SLIDE 27

Warping: Refined Warp

E = Ea + Ef + ωEs

K’F (s, ta) ???

slide-28
SLIDE 28

Warping: Refined Warp

E = Ea + Ef + ωEs

slide-29
SLIDE 29

Warping: Refined Warp

E = Ea + Ef + ωEs

K’F (s, t+1) K’F (s, t)

slide-30
SLIDE 30

Warping: Result

slide-31
SLIDE 31

System Diagram

slide-32
SLIDE 32

System Diagram

slide-33
SLIDE 33

Candidate Video Volumes

Labels L = W ∪ S dynamic: copies of warped video W(x, y, t) W = {Wi, Wj} (if loop seamlessly) static: still-frames from input video repeated to fill duration of output S = {Ib, I2b, … I5b}

  • r “clean plate”
slide-34
SLIDE 34

Candidate Video Volumes

Labels L = W ∪ S dynamic: copies of warped video W(x, y, t) W = {W} or {Wi, Wj} (if loop seamlessly) static: still-frames from input video repeated to fill duration of output S = {Ib, I2b, … I5b}

  • r “clean plate”
slide-35
SLIDE 35

Candidate Video Volumes

Labels L = W ∪ S dynamic: copies of warped video W(x, y, t) W = {W} or {Wi, Wj} (if loop seamlessly) static: still-frames from input video repeated to fill duration of output S = {Ib, I2b, … I5b}

  • r “clean plate”

b = time interval that evenly samples the input five times Ib = video where both frame of input video is repeated for duration of output

slide-36
SLIDE 36

Candidate Video Volumes

Labels L = W ∪ S dynamic: copies of warped video W(x, y, t) W = {W} or {Wi, Wj} (if loop seamlessly) static: still-frames from input video repeated to fill duration of output S = {Ib, I2b, … I5b}

  • r “clean plate”

b = time interval that evenly samples the input five times Ib = video where both frame of input video is repeated for duration of output

slide-37
SLIDE 37

Compositing: Graph-cut

Wi Wj

slide-38
SLIDE 38

Wi Wj

Compositing: Graph-cut

slide-39
SLIDE 39

Wi Wj

Compositing: Graph-cut

slide-40
SLIDE 40

Wi Wj

Compositing: Graph-cut

t = tj - 11 t = tj - 10

slide-41
SLIDE 41

Compositing: Labeling Constraints

slide-42
SLIDE 42

Compositing: Labeling Constraints

From user-drawn compositing strokes:

  • If v(x, y) = blue, λ(x, y, t) ∈ W
  • If v(x, y) = red, λ(x, y, t) ∈ S

For seamless looping:

  • λ(x, y, 0) ≠ Wi
  • λ(x, y, 20) ≠ Wj

v(x, y) = strokes {red, blue, NULL}

slide-43
SLIDE 43

Compositing: Labeling Constraints

From user-drawn compositing strokes:

  • If v(x, y) = blue, λ(x, y, t) ∈ W
  • If v(x, y) = red, λ(x, y, t) ∈ S

For seamless looping:

  • λ(x, y, 0) ≠ Wi
  • λ(x, y, 20) ≠ Wj

t = 0 t = 20 v(x, y) = strokes {red, blue, NULL}

slide-44
SLIDE 44

Compositing: Energy Function

slide-45
SLIDE 45

Compositing: Energy Function

RGB differences

slide-46
SLIDE 46

Compositing: Energy Function

edge strengths

slide-47
SLIDE 47

Compositing: Energy Function

RGB differences

slide-48
SLIDE 48

Compositing: Energy Function

color of pixel p2 in candidate video volume λ(p1)

slide-49
SLIDE 49

Compositing: Energy Function

edge strengths

slide-50
SLIDE 50

Compositing: Energy Function

edge strengths

  • nly consider dynamic

candidates for seams between dynamic and static

slide-51
SLIDE 51

Compositing: Energy Function

edge strengths

slide-52
SLIDE 52

Compositing: Energy Function

minimize

slide-53
SLIDE 53

System Diagram

slide-54
SLIDE 54

Results: Beer

slide-55
SLIDE 55

Results: Model K

slide-56
SLIDE 56

Results: Glass

slide-57
SLIDE 57

Results: Glass

slide-58
SLIDE 58

Results: Glass

slide-59
SLIDE 59

Results: Video Editing

slide-60
SLIDE 60

Results: Roulette

slide-61
SLIDE 61

Results: Roulette

slide-62
SLIDE 62

Results: Roulette

slide-63
SLIDE 63

Results: Video Editing

slide-64
SLIDE 64

Assumptions

slide-65
SLIDE 65

Assumptions

  • Input captured with a tripod (or previously stabilized)
  • Assume large-scale motions can be be de-animated

with 2D warps

  • Objects to de-animate shot in front of a defocused,

uniform, or uniformly-textured background

slide-66
SLIDE 66

Assumptions

  • Input captured with a tripod (or previously stabilized)
  • Assume large-scale motions can be be de-animated

with 2D warps

  • Objects to de-animate shot in front of a defocused,

uniform, or uniformly-textured background

slide-67
SLIDE 67

Assumptions

  • Input captured with a tripod (or previously stabilized)
  • Assume large-scale motions can be be de-animated

with 2D warps

  • Objects to de-animate shot in front of a defocused,

uniform, or uniformly-textured background

slide-68
SLIDE 68

Limitations: 3D Motion

slide-69
SLIDE 69

Limitations: Background

slide-70
SLIDE 70

Limitations

  • What happens if the input video is not stabilized?
slide-71
SLIDE 71

Follow-up

  • This system includes some manual annotation, how

would you automate the user input?

  • Specifically, what would you do for faces?
slide-72
SLIDE 72

Follow-up: Cinemagraph Portraits

“Automatic Cinemagraph Portraits” Bai et al. EGSR 2013

slide-73
SLIDE 73

Selectively De-Animating Video

Jiamin Bai, Aseem Agarwala, Maneesh Agrawala, Ravi Ramamoorthi SIGGRAPH 2012

CS 448V: Computational Video Manipulation

slide-74
SLIDE 74

Warping: Tracking

slide-75
SLIDE 75

Warping: Initial vs Refined

slide-76
SLIDE 76

Results: Existing Techniques

slide-77
SLIDE 77

Adapted Cost Function

Graph-cut

slide-78
SLIDE 78

User Input: De-animated Static

de-animate strokes compositing strokes

slide-79
SLIDE 79

User Input: De-animated Dynamic

compositing strokes de-animate strokes

slide-80
SLIDE 80

System Diagram

slide-81
SLIDE 81

System Diagram