Image Deblurring Seungyong Lee POSTECH 1 Contents Fast Motion - - PowerPoint PPT Presentation

▶
image deblurring
SMART_READER_LITE
LIVE PREVIEW

Image Deblurring Seungyong Lee POSTECH 1 Contents Fast Motion - - PowerPoint PPT Presentation

Dec 09, 2022 464 likes •1.35k views

Image Deblurring Seungyong Lee POSTECH 1 Contents Fast Motion Deblurring (Siggraph Asia 2009) Non-uniform Motion Deblurring for Camera Shakes using Image Registration (Siggraph 2011 Talks) Text Deblurring (an ongoing project) 2

slide-1
SLIDE 1

Seungyong Lee POSTECH

Image Deblurring

1

slide-2
SLIDE 2

Contents

  • Fast Motion Deblurring (Siggraph Asia 2009)
  • Non-uniform Motion Deblurring for Camera Shakes using Image

Registration (Siggraph 2011 Talks)

  • Text Deblurring (an ongoing project)

2

slide-3
SLIDE 3

Sunghyun Cho Seungyong Lee POSTECH POSTECH

Fast Motion Deblurring

3

slide-4
SLIDE 4

Motion blur

  • Camera jitters

Blurred image Latent sharp image

4

slide-5
SLIDE 5

Image formation model

  • Convolution

– Motion blur kernel

  • Trace of a sensor

Latent sharp image Blur kernel Blurred image * : convolution operator

5

slide-6
SLIDE 6

Deblurring

  • Non-blind deconvolution

– Ill-posed (Due to the loss of information caused by motion blur)

  • Blind deconvolution

– Severely ill-posed

Latent image PSF Blurred image Latent image PSF Blurred image

6

slide-7
SLIDE 7

Blind deconvolution

  • Severely ill-posed problem

– No unique solution

Blurred image Possible solutions

7

slide-8
SLIDE 8

Related work

  • Parametric kernels

– Ex) 1D linear motion blur – [Yitzhakey et al. 1998], [Rav-Acha and Peleg 2005], [Cho et

  • al. 2007], [Money and Kang 2008], …

Blurred image Latent sharp image PSF

8

slide-9
SLIDE 9

Related work

  • More complex motion blur

– [Fergus et al. 2006], [Jia 2007], [Shan et al. 2008] – Excessive amount of computation

Blurred image Latent sharp image PSF

9

slide-10
SLIDE 10

Motivation

  • Computation time  Important for practical purpose
  • Previous methods are slow

Image size: 640 x 480 kernel size: 25 x 25 [Fergus et al. 2006] took 1 hr 25 min. [Shan et al. 2008] took 4 min 48 sec. Our method took 5.766 sec. in CPU and 0.734 sec. using GPU accel.

10

slide-11
SLIDE 11

Contributions

  • Fast motion deblurring

– Only a few sec. – Fast latent image estimation – Fast blur kernel estimation  40x ~ 60x faster than [Shan et al. 2008] – GPU acceleration  600x ~ 800x faster than [Shan et al. 2008]

11

slide-12
SLIDE 12

Motion deblurring: Common framework

  • Iteratively solve
  • 1. Estimate a PSF
  • 2. Estimate a latent sharp image using a complex image prior

Latent image PSF Blurred image Latent image PSF Blurred image

12

slide-13
SLIDE 13

Motion deblurring: Common framework

  • Blur model
  • Energy function

N K L B *

2

( , ) * ( ) ( ) f L K B L K q L r K

* : convolution operator q(L), r(K) : regularization terms or priors for L, K

Latent image L Blur kernel K Blurred image B Noise N

13

slide-14
SLIDE 14

Motion deblurring: Common framework

Latent image estimation

2

argmin * ) ' (

L

B L K L L q

Kernel estimation

2

argmin * ) ' (

K

B L K K K r

Blurred image Deblurred result

14

slide-15
SLIDE 15

Motion deblurring: Common framework

Blurred image Kernel estimation

15

slide-16
SLIDE 16

1st latent image estimation Kernel estimation

Motion deblurring: Common framework

16

slide-17
SLIDE 17

1st latent image estimation 1st kernel estimation

Motion deblurring: Common framework

17

slide-18
SLIDE 18

3rd latent image estimation 1st kernel estimation

Motion deblurring: Common framework

18

slide-19
SLIDE 19

3rd latent image estimation 3rd kernel estimation

Motion deblurring: Common framework

19

slide-20
SLIDE 20

5th latent image estimation 3rd kernel estimation

Motion deblurring: Common framework

20

slide-21
SLIDE 21

5th latent image estimation 5th kernel estimation

Motion deblurring: Common framework

21

slide-22
SLIDE 22

7th latent image estimation 5th kernel estimation

Motion deblurring: Common framework

22

slide-23
SLIDE 23

7th latent image estimation 7th kernel estimation

Motion deblurring: Common framework

Both estimation steps are slow We analyzed and accelerated both estimation steps

23

slide-24
SLIDE 24

Latent image estimation: Analysis of prev. methods

Deblurred result Blurred image

  • Two important properties

– Restoration of strong edges

  • Inspecting around strong edges,

we can find a blur kernel

– Noise suppression in smooth regions

  • Avoids the effect of noise
  • n kernel estimation

Blurry input Latent image estimation of [Shan et al. 2008]

Latent image estimation Kernel estimation

24

slide-25
SLIDE 25

Blurry input Latent image estimation of [Shan et al. 2008]

Latent image estimation: Analysis of prev. methods

  • Two important properties

– Restoration of strong edges

  • Inspecting around strong edges,

we can find a blur kernel

– Noise suppression in smooth regions

  • Avoids the effect of noise
  • n kernel estimation

Deblurred result Blurred image Latent image estimation Kernel estimation

25

slide-26
SLIDE 26

Latent image estimation: Analysis of prev. methods

  • Two important properties

– Restoration of strong edges

  • Inspecting around strong edges,

we can find a blur kernel

– Noise suppression in smooth regions

  • Avoids the effect of noise
  • n kernel estimation

Blurry input Latent image estimation of [Shan et al. 2008]

Deblurred result Blurred image Latent image estimation Kernel estimation

26

slide-27
SLIDE 27

Latent image estimation: Analysis of prev. methods

  • Two important properties

– Restoration of strong edges

  • Inspecting around strong edges,

we can find a blur kernel

– Noise suppression in smooth regions

  • Avoids the effect of noise
  • n kernel estimation
  • Computationally expensive priors for q(L)

2

argmin * ) ' (

L

B L K L L q

Blurry input Latent image estimation of [Shan et al. 2008]

Deblurred result Blurred image Latent image estimation Kernel estimation

27

slide-28
SLIDE 28

Latent image estimation: Basic idea for acceleration

Simple Deconvolution

Removes blur quickly Low-quality results

Prediction

Restores strong edges Removes noise Simple image processing tools

We divide…

Latent image estimation

Deblurred result Blurred image Latent image estimation Kernel estimation

28

slide-29
SLIDE 29

Latent image estimation: Basic idea for acceleration

Simple Deconvolution Prediction Updated kernel Current kernel

Deblurred result Blurred image Latent image estimation Kernel estimation

29

slide-30
SLIDE 30

Kernel estimation: Analysis of prev. methods

  • Previous methods estimate a blur kernel K by optimizing:

Latent image estimation Kernel estimation

  • B : a blurred image
  • K : a blur kernel
  • L : a latent sharp image
  • r(K) : a regularization term for K

2

argmin * ) ' (

K

B L K K K r

Deblurred result Blurred image

30

slide-31
SLIDE 31

Kernel estimation: Analysis of prev. methods

  • Previous methods estimate a blur kernel K by optimizing:
  • The simplest case: r(K) ≡ α|K|2 (α: a scalar value)
  • Then, K can be found by solving:
  • which can be solved by a conjugate gradient (CG) method
  • LTLk is computed for each CG iteration

b L k Lk L

T T

L: a matrix rep. of L k: a vector rep. of K b: a vector rep. of B

Latent image estimation Kernel estimation Deblurred result Blurred image

2

argmin * ) ' (

K

B L K K K r

31

slide-32
SLIDE 32

Kernel estimation: Analysis of prev. methods

  • Computing LTLk

– Convolutions & correlations – Lk  L*K – LTLk  L *correl (L*K) – Conv. & corr. can be accelerated using FFTs – 4 FFTs per CG iter. for computing LTLk

  • A CG method needs to iterate…
  • 4 FFTs x 30 CG iters = 120 FFTs…

Latent image estimation Kernel estimation Deblurred result Blurred image

32

slide-33
SLIDE 33

Kernel estimation: Basic idea for acceleration

  • Energy function using derivative images:
  • With deriv. images, we can avoid boundary problem of FFTs
  • ∂LT∂Lk can be computed using 2 FFTs
  • CG iterations converge faster with derivative images

2 2

argmin * '

K

B a K L K K

∂: partial differential operator

Latent image estimation Kernel estimation Deblurred result Blurred image

33

slide-34
SLIDE 34

Deblurred result Blurred image

Deblurring process

Final deconvolution Kernel estimation Prediction Deconvolution

* Deconvolution + prediction = latent image estimation

34

slide-35
SLIDE 35

Results

  • Implementation

– CPU version

  • C++, OpenCV, FFTW

– GPU accelerated version

  • BSGP [Hou et al. 2008] – Easy GPGPU language
  • CUDA FFT library
  • Testing environment

– PC running MS Windows XP 32 bit ver. – Intel Core2 Quad CPU 2.66 GHz – 3.25GB RAM – NVIDIA GeForce GTX 280 Graphics card

35

slide-36
SLIDE 36

Results

Image size Blur kernel size Processing time (CPU) Processing time (GPU) 1024 x 768 49 x 47 18.656 sec. 2.125 sec. Blurry input Our result Blur kernel

36

slide-37
SLIDE 37

Results

Image size Blur kernel size Processing time (CPU) Processing time (GPU) 972 x 966 65 x 93 18.813 sec. 5.766 sec. Blurry input Our result Blur kernel

37

slide-38
SLIDE 38

Results

Image size Blur kernel size Processing time (CPU) Processing time (GPU) 858 x 558 61 x 43 8.969 sec. 0.703 sec. Blurry input Our result Blur kernel

38

slide-39
SLIDE 39

Comparison (quality)

Blurry input [Yuan et al. 2007]

* This method uses two input images.

[Shan et al. 2008]

  • ur method

39

slide-40
SLIDE 40

Comparison (processing time)

  • [Shan et al. 2008] vs. Our method

Image Size Processing time (sec.) Image PSF Shan et al. (CPU) Our method (CPU) Our method (GPU) Picasso 800 x 532 27 x 19 360 7.485 0.609 Statue 903 x 910 25 x 25 762 15.891 0.984 Night 836 x 804 27 x 21 762 13.813 0.937 Red tree 454 x 588 27 x 27 309 4.703 0.438

* Processing times of our CPU code are updated from our paper 40

slide-41
SLIDE 41

Demo video

  • Demo video

41

slide-42
SLIDE 42

Conclusion and future work

  • Deblurring method fast enough for practical use

– Efficient latent image estimation – Efficient kernel estimation

  • Limitation and future work

– Sharp edge assumption – Noise and saturation – Non-uniform blur

42

slide-43
SLIDE 43

Non-uniform Motion Deblurring for Camera Shakes using Image Registration

Sunghyun Cho1 Hojin Cho1 Wu-Ying Tai2 Seungyong Lee1

1POSTECH 2KAIST

slide-44
SLIDE 44

Camera Shakes

44

slide-45
SLIDE 45

Previous Methods Fail

  • Result of [Shan et al. 2008]

45

slide-46
SLIDE 46

Uniform Blur

46

slide-47
SLIDE 47

Non-uniform Blur

47

slide-48
SLIDE 48

Our Method

[Shan et al. 2008] Our method

48

slide-49
SLIDE 49

Related Work

  • Uniform motion blur

– Fergus et al., SIGGRAPH 2006 – Jia, CVPR 2007 – Shan et al., SIGGRAPH 2008 – Cho and Lee, SIGGRAPH ASIA

2009

– Etc…

49

slide-50
SLIDE 50

Related Work

  • Non-uniform motion blur

– Whyte et al. CVPR 2010

  • x, y, z rotations

50

slide-51
SLIDE 51

Related Work

  • Non-uniform motion blur

– Whyte et al. CVPR 2010

  • x, y, z rotations

– Gupta et al. ECCV 2010

  • x, y translations

+ in-plane rotation

51

slide-52
SLIDE 52

Related Work

  • Non-uniform motion blur

– Whyte et al. CVPR 2010

  • x, y, z rotations

– Gupta et al. ECCV 2010

  • x, y translations

+ in-plane rotation

Our method

  • x, y, z translations

+ x, y, z rotations

52

slide-53
SLIDE 53
  • Non-uniform motion deblurring
  • Non-uniform blur estimation

Contributions

Blurred image Blurred image Deblurred result Blurred image Latent image

53

slide-54
SLIDE 54

Blind Deblurring

Input blurry images Blur Estimation Latent Image Restoration Deblurred Result

54

slide-55
SLIDE 55
  • Widely used in previous works

Uniform Blur Model

Motion blur kernel Blurred image Latent image Convolution

  • perator

55

slide-56
SLIDE 56
  • Widely used in previous works

Uniform Blur Model

translation weight Blurred image Latent image

56

slide-57
SLIDE 57
  • Tai et al., PAMI, to appear

Non-uniform Blur Model

Homography Blurred image Latent image

57

slide-58
SLIDE 58
  • Tai et al., PAMI, to appear

Non-uniform Blur Model

How to estimate these?

58

slide-59
SLIDE 59

Blur Estimation

  • Homography Estimation

Residual image Weighted latent image

59

slide-60
SLIDE 60

Blur Estimation

  • Residual

image Weighted latent image

60

slide-61
SLIDE 61
  • Homography Estimation

Blur Estimation

Residual image Weighted latent image

61

slide-62
SLIDE 62

Blur Estimation

  • Weight Estimation: simple linear system

62

slide-63
SLIDE 63

Blur Estimation

Input Weight estimation Solve a linear system

Blurred image Latent image

63

slide-64
SLIDE 64

Estimated blur Input

Blur Estimation

64

slide-65
SLIDE 65

Results

Blurred input 1 Blurred input 2

65

slide-66
SLIDE 66

Results

Estimated blur 1 Estimated blur 2

66

slide-67
SLIDE 67

Results

Our method Uniform deblurring [Cho and Lee 2009]

67

slide-68
SLIDE 68

Results

Blurred input 1 Blurred input 2

68

slide-69
SLIDE 69

Results

Estimated blur 1 Estimated blur 2

69

slide-70
SLIDE 70

Results

Blurred input 1 Blurred input 2 Output

70

slide-71
SLIDE 71

Results

71

slide-72
SLIDE 72

Results

72

slide-73
SLIDE 73

Results

73

slide-74
SLIDE 74

Results

74

slide-75
SLIDE 75

Results

75

slide-76
SLIDE 76

Results

76

slide-77
SLIDE 77

Conclusion

  • We have developed

– Non-uniform blur estimation method – Blind deblurring system for non-uniform motion

blur

77

slide-78
SLIDE 78
  • Still ongoing…
  • Analysis and comparison

– Recent non-uniform deblurring methods

  • More application

– Video deblurring

  • More severe blur
  • Zooming blur

Future Work

78

slide-79
SLIDE 79

Text Deblurring

79

slide-80
SLIDE 80

Original image Synthetic blurred image Fast motion deblurring result Our result

80

slide-81
SLIDE 81

Results on Real Photographs (1/6)

Blurred image Our result

81

slide-82
SLIDE 82

Results on Real Photographs (2/6)

Blurred image Our result

82

slide-83
SLIDE 83

Results on Real Photographs (3/6)

Blurred image Our result

83

slide-84
SLIDE 84

Results on Real Photographs (4/6)

Blurred image Our result (very large blur: 105x105)

84

slide-85
SLIDE 85

Results on Real Photographs (5/6)

Blurred image Our result

85

slide-86
SLIDE 86

Results on Real Photographs (6/6)

Blurred image Our result

86

slide-87
SLIDE 87

Thank you! http://cg.postech.ac.kr

87