Image Deblurring Seungyong Lee POSTECH 1 Contents Fast Motion - PowerPoint PPT Presentation

Image Deblurring Seungyong Lee POSTECH 1

Contents • Fast Motion Deblurring (Siggraph Asia 2009) • Non-uniform Motion Deblurring for Camera Shakes using Image Registration (Siggraph 2011 Talks) • Text Deblurring (an ongoing project) 2

Fast Motion Deblurring Sunghyun Cho Seungyong Lee POSTECH POSTECH 3

Motion blur • Camera jitters Latent sharp image Blurred image 4

Image formation model • Convolution – Motion blur kernel • Trace of a sensor Blurred image Latent sharp image Blur kernel * : convolution operator 5

Deblurring • Non-blind deconvolution – Ill-posed (Due to the loss of information caused by motion blur) Blurred image Latent image PSF • Blind deconvolution – Severely ill-posed Blurred image Latent image PSF 6

Blind deconvolution Possible solutions • Severely ill-posed problem – No unique solution Blurred image 7

Related work • Parametric kernels – Ex) 1D linear motion blur – [Yitzhakey et al. 1998], [Rav-Acha and Peleg 2005], [Cho et al. 2007], [Money and Kang 2008], … Blurred image Latent sharp image PSF 8

Related work • More complex motion blur – [Fergus et al. 2006], [Jia 2007], [Shan et al. 2008] – Excessive amount of computation Blurred image Latent sharp image PSF 9

Motivation • Computation time  Important for practical purpose • Previous methods are slow [Fergus et al. 2006] took 1 hr 25 min. [Shan et al. 2008] took 4 min 48 sec. Our method took 5.766 sec. in CPU and 0.734 sec. using GPU accel. Image size: 640 x 480 kernel size: 25 x 25 10

Contributions • Fast motion deblurring – Only a few sec. – Fast latent image estimation – Fast blur kernel estimation  40x ~ 60x faster than [Shan et al. 2008] – GPU acceleration  600x ~ 800x faster than [Shan et al. 2008] 11

Motion deblurring: Common framework • Iteratively solve 1. Estimate a PSF Blurred image Latent image PSF 2. Estimate a latent sharp image using a complex image prior Blurred image Latent image PSF 12

Motion deblurring: Common framework • Blur model * B L K N Blurred image B Latent image L Blur kernel K Noise N • Energy function 2 ( , ) * ( ) ( ) f L K B L K q L r K * : convolution operator q ( L ) , r ( K ) : regularization terms or priors for L, K 13

Motion deblurring: Common framework Blurred image Latent image estimation Kernel estimation 2 2 ' argmin * ( ) L B L K q L ' argmin * ( ) K B L K r K L K Deblurred result 14

Motion deblurring: Common framework Kernel estimation Blurred image 15

Motion deblurring: Common framework Kernel estimation 1 st latent image estimation 16

Motion deblurring: Common framework 1 st kernel estimation 1 st latent image estimation 17

Motion deblurring: Common framework 1 st kernel estimation 3 rd latent image estimation 18

Motion deblurring: Common framework 3 rd kernel estimation 3 rd latent image estimation 19

Motion deblurring: Common framework 3 rd kernel estimation 5 th latent image estimation 20

Motion deblurring: Common framework 5 th kernel estimation 5 th latent image estimation 21

Motion deblurring: Common framework 5 th kernel estimation 7 th latent image estimation 22

Motion deblurring: Common framework 7 th kernel estimation 7 th latent image estimation We analyzed and accelerated Both estimation steps are slow both estimation steps 23

Latent image estimation: Blurred image Deblurred result Latent image Kernel estimation estimation Analysis of prev. methods • Two important properties – Restoration of strong edges • Inspecting around strong edges, we can find a blur kernel – Noise suppression in smooth regions Blurry input Latent image • Avoids the effect of noise estimation of on kernel estimation [Shan et al. 2008] 24

Latent image estimation: Blurred image Deblurred result Latent image Kernel estimation estimation Analysis of prev. methods • Two important properties – Restoration of strong edges • Inspecting around strong edges, we can find a blur kernel – Noise suppression in smooth regions Blurry input Latent image • Avoids the effect of noise estimation of on kernel estimation [Shan et al. 2008] 25

Latent image estimation: Blurred image Deblurred result Latent image Kernel estimation estimation Analysis of prev. methods • Two important properties – Restoration of strong edges • Inspecting around strong edges, we can find a blur kernel – Noise suppression in smooth regions Blurry input Latent image • Avoids the effect of noise estimation of on kernel estimation [Shan et al. 2008] 26

Latent image estimation: Blurred image Deblurred result Latent image Kernel estimation estimation Analysis of prev. methods • Two important properties – Restoration of strong edges • Inspecting around strong edges, we can find a blur kernel – Noise suppression in smooth regions Blurry input Latent image • Avoids the effect of noise estimation of on kernel estimation [Shan et al. 2008] • Computationally expensive priors for q ( L ) 2 ' argmin * ( ) L B L K q L L 27

Latent image estimation: Blurred image Deblurred result Latent image Kernel estimation estimation Basic idea for acceleration Latent image estimation We divide… Simple Deconvolution Prediction Removes blur quickly Restores strong edges Low-quality results Removes noise Simple image processing tools 28

Latent image estimation: Blurred image Deblurred result Latent image Kernel estimation estimation Basic idea for acceleration Simple Deconvolution Prediction Current kernel Updated kernel 29

Kernel estimation: Blurred image Deblurred result Latent image Kernel estimation estimation Analysis of prev. methods • Previous methods estimate a blur kernel K by optimizing: 2 ' argmin * ( ) K B L K r K K • B : a blurred image • L : a latent sharp image • K : a blur kernel • r ( K ) : a regularization term for K 30

Kernel estimation: Blurred image Deblurred result Latent image Kernel estimation estimation Analysis of prev. methods • Previous methods estimate a blur kernel K by optimizing: 2 ' argmin * ( ) K B L K r K K • The simplest case: r ( K ) ≡ α | K | 2 ( α : a scalar value) • Then, K can be found by solving: L : a matrix rep. of L k : a vector rep. of K T T L Lk k L b b : a vector rep. of B • which can be solved by a conjugate gradient (CG) method • L T Lk is computed for each CG iteration 31

Kernel estimation: Blurred image Deblurred result Latent image Kernel estimation estimation Analysis of prev. methods • Computing L T Lk – Convolutions & correlations – Lk  L*K – L T Lk  L * correl ( L * K ) – Conv. & corr. can be accelerated using FFTs – 4 FFTs per CG iter. for computing L T Lk • A CG method needs to iterate… • 4 FFTs x 30 CG iters = 120 FFTs… 32

Kernel estimation: Blurred image Deblurred result Latent image Kernel estimation estimation Basic idea for acceleration • Energy function using derivative images: 2 2 ' argmin * K B L K a K K ∂ : partial differential operator • With deriv. images, we can avoid boundary problem of FFTs • ∂ L T ∂ Lk can be computed using 2 FFTs • CG iterations converge faster with derivative images 33

Deblurring process Blurred image Final Prediction Kernel estimation Deconvolution deconvolution Deblurred result * Deconvolution + prediction = latent image estimation 34

Results • Implementation – CPU version • C++, OpenCV, FFTW – GPU accelerated version • BSGP [Hou et al. 2008] – Easy GPGPU language • CUDA FFT library • Testing environment – PC running MS Windows XP 32 bit ver. – Intel Core2 Quad CPU 2.66 GHz – 3.25GB RAM – NVIDIA GeForce GTX 280 Graphics card 35

Results Blurry input Our result Blur kernel Processing time Processing time Image size Blur kernel size (CPU) (GPU) 1024 x 768 49 x 47 18.656 sec. 2.125 sec. 36

Results Blurry input Our result Blur kernel Processing time Processing time Image size Blur kernel size (CPU) (GPU) 972 x 966 65 x 93 18.813 sec. 5.766 sec. 37

Results Blur kernel Blurry input Our result Processing time Processing time Image size Blur kernel size (CPU) (GPU) 858 x 558 61 x 43 8.969 sec. 0.703 sec. 38

Comparison (quality) Blurry input [Yuan et al. 2007] [Shan et al. 2008] our method * This method uses two input images. 39

Comparison (processing time) • [Shan et al. 2008] vs. Our method Size Processing time (sec.) Image Shan et al. Our method Our method Image PSF (CPU) (CPU) (GPU) Picasso 800 x 532 27 x 19 360 7.485 0.609 Statue 903 x 910 25 x 25 762 15.891 0.984 Night 836 x 804 27 x 21 762 13.813 0.937 Red tree 454 x 588 27 x 27 309 4.703 0.438 * Processing times of our CPU code are updated from our paper 40

Demo video • Demo video 41

Conclusion and future work • Deblurring method fast enough for practical use – Efficient latent image estimation – Efficient kernel estimation • Limitation and future work – Sharp edge assumption – Noise and saturation – Non-uniform blur 42