Video Stabilization CS448V Computational Video Manipulation April - - PowerPoint PPT Presentation

▶
video stabilization
SMART_READER_LITE
LIVE PREVIEW

Video Stabilization CS448V Computational Video Manipulation April - - PowerPoint PPT Presentation

Dec 05, 2022 642 likes •1.85k views

Video Stabilization CS448V Computational Video Manipulation April 2019 Fundamental problem that became even more relevant in recent years Important for producing high quality video and as a first step of many algorithms Important for

slide-1
SLIDE 1

Video Stabilization

CS448V — Computational Video Manipulation April 2019

slide-2
SLIDE 2

Fundamental problem that became even more relevant in recent years

slide-3
SLIDE 3
slide-4
SLIDE 4

Important for producing high quality video and as a first step of many algorithms

slide-5
SLIDE 5

Important for producing high quality video and as a first step of many algorithms

“In forming a video loop, we assume that the input video has already been stabilized.” [Liao et al. ’15]

slide-6
SLIDE 6

Many ways to stabilize

slide-7
SLIDE 7

Many ways to stabilize

Both at capture time and in post

slide-8
SLIDE 8

Many ways to stabilize

Both at capture time and in post

Tripod

slide-9
SLIDE 9

Many ways to stabilize

Both at capture time and in post

Tripod OIS

slide-10
SLIDE 10

Many ways to stabilize

Both at capture time and in post

Tripod Gimbal OIS

slide-11
SLIDE 11

Many ways to stabilize

Both at capture time and in post

capture time

slide-12
SLIDE 12

Many ways to stabilize

Both at capture time and in post

capture time

slide-13
SLIDE 13

Many ways to stabilize

Both at capture time and in post

capture time post production

slide-14
SLIDE 14

Many ways to stabilize

Both at capture time and in post

capture time post production

manual automatic 2D 3D

slide-15
SLIDE 15

Many ways to stabilize

Both at capture time and in post

capture time post production

manual automatic 2D 3D [Liu et al. ’13]

slide-16
SLIDE 16

Recipe for video stabilization

slide-17
SLIDE 17

Recipe for video stabilization

Input frames

slide-18
SLIDE 18

Recipe for video stabilization

Input frames

Detect features

slide-19
SLIDE 19

Recipe for video stabilization

Input frames

Detect features

Raw pixels, 
 SURF , SIFT, …

slide-20
SLIDE 20

Recipe for video stabilization

Input frames

Detect features Calculate relation between photos

Raw pixels, 
 SURF , SIFT, …

slide-21
SLIDE 21

Recipe for video stabilization

Input frames

Detect features Calculate relation between photos

Raw pixels, 
 SURF , SIFT, … Homography, 
 3D camera location, …

slide-22
SLIDE 22

Recipe for video stabilization

Input frames

Detect features Calculate relation between photos

Raw pixels, 
 SURF , SIFT, … Homography, 
 3D camera location, …

Smooth relation between photos

slide-23
SLIDE 23

Recipe for video stabilization

Input frames

Detect features Calculate relation between photos

Raw pixels, 
 SURF , SIFT, … Homography, 
 3D camera location, …

Smooth relation between photos

Low pass filter, spline fitting, bilateral filter, …

slide-24
SLIDE 24

Recipe for video stabilization

Input frames

Detect features Calculate relation between photos

Raw pixels, 
 SURF , SIFT, … Homography, 
 3D camera location, …

Smooth relation between photos Create frames using smoothed relation

Low pass filter, spline fitting, bilateral filter, …

slide-25
SLIDE 25

Recipe for video stabilization

Input frames

Detect features Calculate relation between photos

Raw pixels, 
 SURF , SIFT, … Homography, 
 3D camera location, …

Smooth relation between photos Create frames using smoothed relation

Low pass filter, spline fitting, bilateral filter, … Warp frames, reconstruct from 3D, …

slide-26
SLIDE 26

Recipe for video stabilization

Input frames

Detect features Calculate relation between photos

Raw pixels, 
 SURF , SIFT, … Homography, 
 3D camera location, …

Smooth relation between photos Create frames using smoothed relation

Output frames

Low pass filter, spline fitting, bilateral filter, … Warp frames, reconstruct from 3D, …

slide-27
SLIDE 27

Recipe for video stabilization

Input frames

Detect features Calculate relation between photos

Raw pixels, 
 SURF , SIFT, … Homography, 
 3D camera location, …

Smooth relation between photos Create frames using smoothed relation

Output frames

Low pass filter, spline fitting, bilateral filter, … Warp frames, reconstruct from 3D, …

Toy example:

slide-28
SLIDE 28

Recipe for video stabilization

Input frames

Detect features Calculate relation between photos

Raw pixels, 
 SURF , SIFT, … Homography, 
 3D camera location, …

Smooth relation between photos Create frames using smoothed relation

Output frames

Low pass filter, spline fitting, bilateral filter, … Warp frames, reconstruct from 3D, …

Toy example:

SIFT

slide-29
SLIDE 29

Recipe for video stabilization

Input frames

Detect features Calculate relation between photos

Raw pixels, 
 SURF , SIFT, … Homography, 
 3D camera location, …

Smooth relation between photos Create frames using smoothed relation

Output frames

Low pass filter, spline fitting, bilateral filter, … Warp frames, reconstruct from 3D, …

Toy example:

SIFT 2D translation

slide-30
SLIDE 30

Recipe for video stabilization

Input frames

Detect features Calculate relation between photos

Raw pixels, 
 SURF , SIFT, … Homography, 
 3D camera location, …

Smooth relation between photos Create frames using smoothed relation

Output frames

Low pass filter, spline fitting, bilateral filter, … Warp frames, reconstruct from 3D, …

Toy example:

SIFT 2D translation Gaussian

slide-31
SLIDE 31

Recipe for video stabilization

Input frames

Detect features Calculate relation between photos

Raw pixels, 
 SURF , SIFT, … Homography, 
 3D camera location, …

Smooth relation between photos Create frames using smoothed relation

Output frames

Low pass filter, spline fitting, bilateral filter, … Warp frames, reconstruct from 3D, …

Toy example:

SIFT 2D translation Gaussian Warp

slide-32
SLIDE 32

2D vs. 3D

slide-33
SLIDE 33

2D vs. 3D

Input frames Output frames warp

2D

slide-34
SLIDE 34

2D vs. 3D

Input frames Output frames warp

2D

[Snavely et al. ’06]

Input frames Output frames

3D

slide-35
SLIDE 35

Bundled Camera Paths for Video Stabilization

Liu et al. SIGGRAPH 2013

slide-36
SLIDE 36

Detect features Calculate relation between photos Smooth relation between photos Create frames using smoothed relation

Input frames Output frames

slide-37
SLIDE 37

Detect features Calculate relation between photos Smooth relation between photos Create frames using smoothed relation

warping-based motion representation Input frames Output frames

slide-38
SLIDE 38

Detect features Calculate relation between photos Smooth relation between photos Create frames using smoothed relation

warping-based motion representation adaptive space-time path smoothing Input frames Output frames

slide-39
SLIDE 39

Warping-based motion representation

frame t frame t+1

slide-40
SLIDE 40

Warping-based motion representation

frame t frame t+1 p = [v1

p

v2

p

v3

p

v4

p]

w1

p

w2

p

w3

p

w4

p 4

∑

i=1

wi

p = 1

Given that:

slide-41
SLIDE 41

Warping-based motion representation

frame t frame t+1 p = [v1

p

v2

p

v3

p

v4

p]

w1

p

w2

p

w3

p

w4

p 4

∑

i=1

wi

p = 1

Given that: We would like: ̂ p = [ ̂ v1

p

̂ v2

p

̂ v3

p

̂ v4

p]

w1

p

w2

p

w3

p

w4

p

̂ p = ̂ Vpwp

slide-42
SLIDE 42

Warping-based motion representation

frame t frame t+1 p = [v1

p

v2

p

v3

p

v4

p]

w1

p

w2

p

w3

p

w4

p 4

∑

i=1

wi

p = 1

Given that: Data term: ∑

p

∥ ̂ Vpwp − ̂ p∥2 We would like: ̂ p = [ ̂ v1

p

̂ v2

p

̂ v3

p

̂ v4

p]

w1

p

w2

p

w3

p

w4

p

̂ p = ̂ Vpwp

slide-43
SLIDE 43

Warping-based motion representation

Shape-preserving term: Distance from similarity transform

slide-44
SLIDE 44

Warping-based motion representation

Shape-preserving term: Distance from similarity transform sounds familiar?…

slide-45
SLIDE 45

Warping-based motion representation

E( ̂ V) = Ed( ̂ V) + αEs( ̂ V)

data shape-preserving

slide-46
SLIDE 46

Warping-based motion representation

with without

Shape-preserving term

slide-47
SLIDE 47

Warping-based motion representation

with without

Shape-preserving term

slide-48
SLIDE 48

Warping-based motion representation

with without

Shape-preserving term

slide-49
SLIDE 49

Warping-based motion representation

frame t frame t+1

We now have a local homography Fi(t) for each cell i of frame t

slide-50
SLIDE 50

Extensions for robust estimation

slide-51
SLIDE 51

Extensions for robust estimation

Outlier rejection: dual-scale RANSAC

slide-52
SLIDE 52

Extensions for robust estimation

Outlier rejection: dual-scale RANSAC global homography Course discard outliers

  • ver large

threshold

slide-53
SLIDE 53

Extensions for robust estimation

Outlier rejection: dual-scale RANSAC global homography Course discard outliers

  • ver large

threshold Fine local homographies discard outliers

  • ver small

threshold

slide-54
SLIDE 54

Extensions for robust estimation

Adaptive regularization

E( ̂ V) = Ed( ̂ V) + αEs( ̂ V)

Calculate α per frame Fitting error: 
 average residual of feature matching Smoothness error: 
 L2 distance between neighboring homographies

slide-55
SLIDE 55

Estimate for different α and pick minimal error

slide-56
SLIDE 56

Benefits of regularization

slide-57
SLIDE 57

Detect features Calculate relation between photos Smooth relation between photos Create frames using smoothed relation

warping-based motion representation adaptive space-time path smoothing Input frames Output frames

slide-58
SLIDE 58
slide-59
SLIDE 59

Optimize one

slide-60
SLIDE 60

Optimize all Optimize one

slide-61
SLIDE 61

Optimizing a single path

slide-62
SLIDE 62

Optimizing a single path

t

slide-63
SLIDE 63

Optimizing a single path

Data term: 
 blue should match red t

slide-64
SLIDE 64

Optimizing a single path

Data term: 
 blue should match red t Smoothness term: 
 blue at time t should match the (60) frames around t

slide-65
SLIDE 65

Optimizing a single path

Data term: 
 blue should match red t

min ∑

t

∥P(t) − C(t)∥2 + λt ∑

r∈Ωt

ωt,r(C) ⋅ ∥P(t) − P(r)∥2

data term smoothness term Smoothness term: 
 blue at time t should match the (60) frames around t

slide-66
SLIDE 66

Detour: bilateral filter

Slides adapted from Sylvain Paris

slide-67
SLIDE 67

Objective of bilateral filtering

  • Smooth texture
  • Preserve edges
slide-68
SLIDE 68

Illustration in 1D

slide-69
SLIDE 69

Illustration in 1D

1D image = line of pixels

slide-70
SLIDE 70

Illustration in 1D

1D image = line of pixels

pixel intensity pixel position

Better visualized as a plot

slide-71
SLIDE 71

Definition

slide-72
SLIDE 72

Definition

Gaussian blur

Ip = ∑

q

Gσs(∥p − q∥)Iq

  • nly spatial distance, intensity ignored

space p q

slide-73
SLIDE 73

Definition

Gaussian blur

Ip = ∑

q

Gσs(∥p − q∥)Iq

  • nly spatial distance, intensity ignored

space p q

Bilateral filter

[Aurich 95, Smith 97, Tomasi 98]

spatial and range distances

Ip = 1 Wp ∑

q

Gσs(∥p − q∥)Gσr(|Ip − Iq|)Iq

space range p q

slide-74
SLIDE 74

Example on a real image

Ä Ä Ä

slide-75
SLIDE 75

Bilateral filter is not just for pixel values!

slide-76
SLIDE 76

Bilateral filter is not just for pixel values!

Back to stabilization…

slide-77
SLIDE 77

Optimizing a single path

min ∑

t

∥P(t) − C(t)∥2 + λt ∑

r∈Ωt

ωt,r(C) ⋅ ∥P(t) − P(r)∥2

data term smoothness term

slide-78
SLIDE 78

Optimizing a single path

min ∑

t

∥P(t) − C(t)∥2 + λt ∑

r∈Ωt

ωt,r(C) ⋅ ∥P(t) − P(r)∥2

data term smoothness term

ωt,r = Gt(∥r − t∥) ⋅ Gm(∥C(r) − C(t)∥)

slide-79
SLIDE 79

Optimizing a single path

min ∑

t

∥P(t) − C(t)∥2 + λt ∑

r∈Ωt

ωt,r(C) ⋅ ∥P(t) − P(r)∥2

data term smoothness term

ωt,r = Gt(∥r − t∥) ⋅ Gm(∥C(r) − C(t)∥)

distance between frames

slide-80
SLIDE 80

Optimizing a single path

min ∑

t

∥P(t) − C(t)∥2 + λt ∑

r∈Ωt

ωt,r(C) ⋅ ∥P(t) − P(r)∥2

data term smoothness term

ωt,r = Gt(∥r − t∥) ⋅ Gm(∥C(r) − C(t)∥)

distance between frames distance between camera poses

slide-81
SLIDE 81

Optimizing a single path

min ∑

t

∥P(t) − C(t)∥2 + λt ∑

r∈Ωt

ωt,r(C) ⋅ ∥P(t) − P(r)∥2

data term smoothness term setting the weights λt

slide-82
SLIDE 82

Optimizing a single path

min ∑

t

∥P(t) − C(t)∥2 + λt ∑

r∈Ωt

ωt,r(C) ⋅ ∥P(t) − P(r)∥2

data term smoothness term setting the weights λt Run optimization with global weight For each frame While too much cropping or distortion Decrease weight and re-run

slide-83
SLIDE 83

Optimizing bundled paths

slide-84
SLIDE 84

Optimizing bundled paths

min ∑

i

O({Pi(t)}) + ∑

t

∑

j∈N(i)

∥Pi(t) − Pj(t)∥2

single path smoothness between neighboring paths i N(i)

slide-85
SLIDE 85
slide-86
SLIDE 86
slide-87
SLIDE 87
slide-88
SLIDE 88
slide-89
SLIDE 89
slide-90
SLIDE 90

Detect features Calculate relation between photos Smooth relation between photos Create frames using smoothed relation

warping-based motion representation adaptive space-time path smoothing Input frames Output frames

slide-91
SLIDE 91

Evaluation & Results

slide-92
SLIDE 92

Comparison to previous methods

slide-93
SLIDE 93

Comparison to commercial products

slide-94
SLIDE 94

User study

slide-95
SLIDE 95
slide-96
SLIDE 96
slide-97
SLIDE 97
slide-98
SLIDE 98
slide-99
SLIDE 99
slide-100
SLIDE 100
slide-101
SLIDE 101
slide-102
SLIDE 102
slide-103
SLIDE 103
slide-104
SLIDE 104
slide-105
SLIDE 105
slide-106
SLIDE 106
slide-107
SLIDE 107
slide-108
SLIDE 108
slide-109
SLIDE 109
slide-110
SLIDE 110
slide-111
SLIDE 111

always check supplemental…

slide-112
SLIDE 112

always check supplemental…

slide-113
SLIDE 113

always check supplemental…

slide-114
SLIDE 114

always check supplemental…

slide-115
SLIDE 115

Recap

slide-116
SLIDE 116

Recap

  • Video stabilization is important!
slide-117
SLIDE 117

Recap

  • Video stabilization is important!
  • General recipe for stabilization

Detect features Calculate relation between photos Smooth relation between photos Create frames using smoothed relation Input Output

slide-118
SLIDE 118

Recap

  • Video stabilization is important!
  • General recipe for stabilization
  • [Liu et al. ’13]

Detect features Calculate relation between photos Smooth relation between photos Create frames using smoothed relation Input Output

slide-119
SLIDE 119

Recap

  • Video stabilization is important!
  • General recipe for stabilization
  • [Liu et al. ’13]
  • Bilateral filter

Detect features Calculate relation between photos Smooth relation between photos Create frames using smoothed relation Input Output

space range p q