Full-Frame Video Stabilization with Motion Inpainting Yasuyuki - - PowerPoint PPT Presentation

Full-Frame Video Stabilization with Motion Inpainting

Yasuyuki Matsushita, E l Of k Eyal Ofek Weina Ge Xiaoou Tang Heung-Yeung Shum g g IEEE Trans on PAMI, July 2006

Outline Outline

I t d ti

• Introduction
• Proposed Method
• Experimental results
• Quantitative Evaluation

Quantitative Evaluation

• Computation Cost

C l i

• Conclusion

2

Introduction Introduction

St bili ti

• Stabilization：

– Remove undesirable motion caused by i t ti l h k f h h d unintentional shake of a human hand.

• remove high frequency camera motion vs.

completely remove camera motion completely remove camera motion.

• full frame vs. trimming
• motion inpainting vs. mosaicing

p g g

3

Prior Work vs Now Prior Work vs. Now

4

Input Video Global Motion Estimation Motion Smoothing Local Motion Estimation Motion Inpainting Image Deblurring Completion

5

Output Video

Global Motion Estimation Global Motion Estimation

GM i ti t d b li i i i

• GM is estimated by aligning pair-wise

adjacent frames.

min( ( ) ( )) I Tp I p

–

• Hierarchical motion estimation

– construct an image pyramid

'

min( ( ) ( ))

t t t t T

I Tp I p −

– construct an image pyramid – start from the coarsest level

• By applying the parameter estimation for

By applying the parameter estimation for every pair of adjacent frames, a global transformation chain is obtained.

j i

T

6

i

H.-Y. Shum and R. Szeliski, “Construction of Panoramic Mosaics with Global and Local Alignment,” Int’l J. Computer Vision, vol. 36, no. 2, pp. 101-130, 2000.

Input Video Global Motion Estimation Motion Smoothing Local Motion Estimation Motion Inpainting Image Deblurring Completion

7

Output Video

Image Deblurring Image Deblurring

T f i h i i l

• Transferring sharper image pixels

from neighboring frames.

– evaluates the “relative blurriness”

1 b =

2 2 y

{(( )( )) (( )( )) }

t

t x t t t t p

b f I p f I p = ⊗ + ⊗

∑

– evaluates the “alignment error”

' ' '

( ) | ( ) ( ) |

t t t t t t t t t

E p I T p I p = −

8

( ) | ( ) ( ) |

t t t t t t

p p p

Image Deblurring Image Deblurring

Bl i l l d b i t l ti

• Blurry pixel are replaced by interpolating

shaper pixels.

฀

' ' '

( ) ( ) ( )

t t t t t t t t t

I p w p I T p + ∑

w is the eight factor hich consists of

฀

' ' '

( ) 1 ( )

t N t t t t t t N

I p w p

∈ ∈

= + ∑

• w is the weight factor which consists of

the pixel-wise alignment error and relative blurriness blurriness

' '

' ( )

1 ( )

t t t t t

b b t b t t b E p

if w p

• therwise

α α +

⎧ < ⎪ = ⎨ ⎪ ⎩

9

' ' (

)

t t t

b E p

• therwise

α +

⎪ ⎩

Image Deblurring Image Deblurring

10

Input Video Global Motion Estimation Motion Smoothing Local Motion Estimation Motion Inpainting Image Deblurring Completion

11

Output Video

Motion Smoothing Motion Smoothing

( )

i

S T G k = ⊗

∑

2 2

t / 2 1 2

( ) { : }, ( ) ,

t

t i N k t

S T G k N j t k j t k G k e k

σ πσ

σ

∈ −

⊗ = − ≤ ≤ + = =

∑

12

Motion Smoothing Motion Smoothing

13

Input Video Global Motion Estimation Motion Smoothing Local Motion Estimation Motion Inpainting Image Deblurring Completion

14

Output Video

Local Motion Estimation Local Motion Estimation

A id l i f L K d

• A pyramidal version of Lucas-Kanade
• ptical flow computation is applied to

bt i th l l ti fi ld

• btain the local motion field.

15

• J. Bouguet, “Pyramidal Implementation of the Lucas Kanade Feature Tracker: Description of the

Algorithm,” OpenCV Document, Intel, Microprocessor Research Labs, 2000.

Input Video Global Motion Estimation Motion Smoothing Local Motion Estimation Motion Inpainting Image Deblurring Completion

16

Output Video

Motion Inpainting Motion Inpainting

M i i ith i t t i t

• Mosaicing with consistency constraint.

' ' '

( ) ( ( )) ( )

t t t t t t t

if v p T midian I T p I p < ⎧ = ⎨

where

( )

t t

I p

• therwise

keep it as missing ⎨ ⎩

2 ' ' ' ' '

1 ( ) ( ( ) ( )) 1 1

t

t t t t t t t t t t t M

v p I T p I T p n

∈

= − − ∑

' ' ' ' '

1 ( ) ( )

t

t t t t t t t t t M

I T p I T p n

∈

=

∑

17

Motion Inpainting Motion Inpainting

18

• A. Telea, “An Image Inpainting Technique Based on the Fast Marching Method,” J. Graphics Tools,
• vol. 9, no. 1, pp. 23-34, 2004.

Motion Inpainting Motion Inpainting

Th ti l f i l i

• The motion value for pixel pt is

generated by a weighted average of th ti t f th i l H( ) the motion vectors of the pixels H(pt)

( )

( , ) ( | ) ( )

t t

t t t t q H p

w p q F p q F

∈∑

where

( ) ( )

( ) ( , )

t t t t

q H p t t t q H p

F p w p q

∈ ∈

=

∑

( ) ( ) ( ) ( )

( | ) ( ) ( )( ) ( )

x t x t y t y t

F q F q x y t t t t t t t F q F q x y

x F p q F q F q p q F q y

∂ ∂ ∂ ∂ ∂ ∂ ∂ ∂

⎡ ⎤ Δ ⎡ ⎤ ⎢ ⎥ = + ∇ − = + ⎢ ⎥ Δ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦

19

' ' ' '

1 1 ( , ) || || || ( ) ( ) ||

t t t t t t t t t t

w p q p q I q p q I q ε = − + − − +

Input Video Global Motion Estimation Motion Smoothing Local Motion Estimation Motion Inpainting Image Deblurring Completion

20

Output Video

Summary of the Algorithm Summary of the Algorithm

21

Experimental Results Experimental Results

30 id li ( b t 80 i t )

• 30 video clips (about 80 minutes)

with different types of scenes

• k = 6 for motion smoothing
• 5x5 filter for motion inpainting

p g

22

Experimental Results (1) Experimental Results (1)

23

Experimental Results (2) Experimental Results (2)

24

Experimental Results (3) Experimental Results (3)

25

Failure Cases – incorrect estimation of motion

26

Failure Cases – abrupt changes of motion

27

Quantitative Evaluation Quantitative Evaluation

D i ti f th G d T th

• Deviation from the Ground Truth.
• MAD of intensity

28

Quantitative Evaluation Quantitative Evaluation

E l ti f S ti T l S th

• Evaluation of Spatio-Temporal Smoothness.

– The normalized discontinuity measure D is defined as defined as

1 1 || ||

n n i i i i i I

D I I I n n

∂

= ∇ = ∇ ⋅∇ ⎡ ⎤ ⎡ ⎤

∑ ∑

( 1, , ) ( 1, , ) ( , 1, ) ( , 1, ) ( 1) ( 1)

I x I y I

I x y t I x y t I I x y t I x y t I x y t I x y t

∂ ∂ ∂ ∂ ∂

⎡ ⎤ + − − ⎡ ⎤ ⎢ ⎥ ⎢ ⎥ ∇ = ≈ + − − ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ + ⎣ ⎦ ⎣ ⎦

– The relative smoothness is evaluated by (D D )/(D D )

( , , 1) ( , , 1)

I t

I x y t I x y t

∂ ∂

⎢ ⎥ ⎢ ⎥ + − − ⎣ ⎦ ⎣ ⎦

29

(DM-DO)/(DM-DA) – 5.9%~23.5% smoother than mosaicing

Computation Cost Computation Cost

2 2 f / f 720 486 id ith

• 2.2 frames/s for 720x486 video with

P4 2.8GHz CPU

30

Conclusion Conclusion

Motion inpainting instead of cropping

• Motion inpainting instead of cropping.
• Deblurring without estimating PSFs.
• Spatial smoothness is indirectly

guaranteed by the smoothness of the extrapolated motion extrapolated motion.

• Temporal consistency on both static

and dynamic areas is given by optical and dynamic areas is given by optical flow from the neighboring frames.

31

Thank You

presented by 蕭志傑 蕭志傑 Hsiao, Chih-Chieh