Richardson-Lucy Deblurring for Moving Light Field Cameras Donald - - PowerPoint PPT Presentation

▶
richardson lucy deblurring for moving light field cameras
SMART_READER_LITE
LIVE PREVIEW

Richardson-Lucy Deblurring for Moving Light Field Cameras Donald - - PowerPoint PPT Presentation

Jan 12, 2024 129 likes •602 views

Richardson-Lucy Deblurring for Moving Light Field Cameras Donald Dansereau 1 , Anders Eriksson 2 and Jrgen Leitner 2,3 1 Stanford University, 2 Queensland University of Technology, 3 ARC Centre of Excellence for Robotic Vision CVPR:LF4CV

slide-1
SLIDE 1

CVPR:LF4CV Workshop 2017 July 26

Richardson-Lucy Deblurring for Moving Light Field Cameras

Donald Dansereau1, Anders Eriksson2 and Jürgen Leitner2,3

1Stanford University, 2Queensland University of Technology, 3ARC Centre of Excellence for Robotic Vision

slide-2
SLIDE 2

2

ComputationalImaging.org

3D Motion Complicates Vision

Scene-dependent nonuniform apparent motion

slide-3
SLIDE 3

3

ComputationalImaging.org

3D is Easier in 4D

[Smith2009]

Video Stabilization

We have 6-DOF virtual camera control

http://pages.cs.wisc.edu/~lizhang/projects/lfstable/ [video]

slide-4
SLIDE 4

4

ComputationalImaging.org

3D is Easier in 4D

Per-pix ixel l stil still- l-ca camera meth thods

  • Change detection
  • Tracking/segmentation
  • Velocity & temporal filtering

[dansereau2016]

Closed-Form Change Detection

We can fix the camera’s position

http://dgd.vision/Projects/LFChangeDet/

slide-5
SLIDE 5

5

ComputationalImaging.org

3D is Easier in 4D

Per-pix ixel l stil still- l-ca camera meth thods

  • Change detection
  • Tracking/segmentation
  • Velocity & temporal filtering

[dansereau2016]

Closed-Form Change Detection

We can fix the camera’s position

http://dgd.vision/Projects/LFChangeDet/

slide-6
SLIDE 6

6

ComputationalImaging.org

3D is Easier in 4D

Per-pix ixel l stil still- l-ca camera meth thods

  • Change detection
  • Tracking/segmentation
  • Velocity & temporal filtering

[dansereau2016]

Closed-Form Change Detection

We can fix the camera’s position

http://dgd.vision/Projects/LFChangeDet/

slide-7
SLIDE 7

7

ComputationalImaging.org

3D is Easier in 4D

[Neumann2002, Dansereau2011, Dong2013]

Closed-form 6-DOF Odometry

Linearize Apparent Motion

Lx Ly Lz Ly Lx Lz

2 4 6 8 1 1 2 1 4 − 1 5 − 1 − 5 5 1 − 2 2 x ( m ) y ( m ) z ( m ) T r u e P l e n

  • p

t i c P

  • i

n t w i s e F e a t u r e

Lukas-Kanade optical flow generalizes to 6-DOF

slide-8
SLIDE 8

8

ComputationalImaging.org

3D is Easier in 4D

[Neumann2002, Dansereau2011, Dong2013]

Closed-form 6-DOF Odometry

Linearize Apparent Motion

Lx Ly Lz Ly Lx Lz

2 4 6 8 1 1 2 1 4 − 1 5 − 1 − 5 5 1 − 2 2 x ( m ) y ( m ) z ( m ) T r u e P l e n

  • p

t i c P

  • i

n t w i s e F e a t u r e

Lukas-Kanade optical flow generalizes to 6-DOF

slide-9
SLIDE 9

9

ComputationalImaging.org

Blur in 3D Scenes

Convolution models blurring in 2D… Can we replace convolution with LF rendering in 3D scenes?

slide-10
SLIDE 10

10

ComputationalImaging.org

Related Work

“Light Field Blind Motion Deblurring” [Srinivasan 2017]

  • 3-DOF
  • Insights on blur manifestation in LF
  • Blind
  • Modern optimization (ADAM)

LF-RL

  • Requires extension to be blind
  • 6-DOF
  • Proof of convergence to ML estimate (see paper)
  • New LF equiparallax regularizer
slide-11
SLIDE 11

11

ComputationalImaging.org

Related Work

“Richardson-Lucy Deblurring for Scenes under a Projective Motion Path” [Tai et al. 2011]

slide-12
SLIDE 12

12

ComputationalImaging.org

Richardson-Lucy Deblurring

IN

slide-13
SLIDE 13

13

ComputationalImaging.org

Still LF

Light Field Richardson-Lucy

BLUR FWD BLUR REV

IN

slide-14
SLIDE 14

14

ComputationalImaging.org

Still LF

Light Field Richardson-Lucy

BLUR FWD BLUR REV

IN

Simulated Motion

slide-15
SLIDE 15

15

ComputationalImaging.org

Still LF

Light Field Richardson-Lucy

BLUR FWD BLUR REV

IN

Simulated Motion Simulated Blur

slide-16
SLIDE 16

16

ComputationalImaging.org

Regularization

Anisotropic total variation Favour textural edges

[Goldluecke & Wanner 2013, Heber2013]

Equiparallax Favour equal slopes in s,u and t,v

slide-17
SLIDE 17

17

ComputationalImaging.org

Ori riginal Ori riginal Ori riginal

Rendered Results: Rot about y

slide-18
SLIDE 18

18

ComputationalImaging.org

Ori riginal Ori riginal Bl Blurre red Bl Blurre red Ori riginal Blurre red

Rendered Results: Rot about y

slide-19
SLIDE 19

19

ComputationalImaging.org

Ori riginal Ori riginal Bl Blurre red Bl Blurre red Ori riginal Blurre red

Rendered Results: Rot about y

Deblurre rred Deblurr rred Deblurr rred

slide-20
SLIDE 20

20

ComputationalImaging.org

rotz

Rendered Results: Rot about z

Bl Blurre rred Bl Blurr rred Bl Blurr rred

slide-21
SLIDE 21

21

ComputationalImaging.org

rotz

Rendered Results: Rot about z

Bl Blurre rred Deblurre red Bl Blurr rred Bl Blurr rred Deblurre rred Deblurre rred

slide-22
SLIDE 22

22

ComputationalImaging.org

Rendered Results: Trans along x

Bl Blurre rred Bl Blurr rred Bl Blurr rred

slide-23
SLIDE 23

23

ComputationalImaging.org

Rendered Results: Trans along x

Bl Blurre rred Deblurre red Bl Blurr rred Bl Blurr rred Deblurre rred Deblurre rred

slide-24
SLIDE 24

24

ComputationalImaging.org

rotz

Rendered Results: Rot about z

Bl Blurre rred Bl Blurr rred Bl Blurr rred

slide-25
SLIDE 25

25

ComputationalImaging.org

rotz

Rendered Results: Rot about z

Bl Blurre rred Deblurre red Bl Blurr rred Bl Blurr rred Deblurre rred Deblurre rred

slide-26
SLIDE 26

26

ComputationalImaging.org

Bl Blurr rred Bl Blurr rred

Rendered Results: Trans along z

Bl Blurre rred

slide-27
SLIDE 27

27

ComputationalImaging.org

Bl Blurr rred Bl Blurr rred Deblurre rred Deblurre rred

Rendered Results: Trans along z

Bl Blurre rred Deblurre red

slide-28
SLIDE 28

28

ComputationalImaging.org

Bl Blurr rred Bl Blurr rred Deblurre rred Deblurre rred

Rendered Results: Trans along z

Bl Blurre rred Deblurre red Without Regulariza zation Without Regulariza zation

slide-29
SLIDE 29

29

ComputationalImaging.org

Results: Captured

George Repeatable camera motion Isolated dimensions Known magnitudes Quantitative Evaluation

slide-30
SLIDE 30

30

ComputationalImaging.org

Still

Validating Calibration & Rendering

Camera calibration, rectification Metric blur simulation Metric robot motion Camera-to-robot calibration

slide-31
SLIDE 31

31

ComputationalImaging.org

Still Measured Blur

Validating Calibration & Rendering

Camera calibration, rectification Metric blur simulation Metric robot motion Camera-to-robot calibration

slide-32
SLIDE 32

32

ComputationalImaging.org

Still Measured Blur

Validating Calibration & Rendering

Camera calibration, rectification Metric blur simulation Metric robot motion Camera-to-robot calibration Simulated Blur

slide-33
SLIDE 33

33

ComputationalImaging.org

Still Measured Blur

Validating Calibration & Rendering

Camera calibration, rectification Metric blur simulation Metric robot motion Camera-to-robot calibration Simulated Blur Still

slide-34
SLIDE 34

34

ComputationalImaging.org

Results: Captured

No increase in noise Regularization is helping Large increase in sharpness

slide-35
SLIDE 35

35

ComputationalImaging.org

Results: Captured

Measured Blur

slide-36
SLIDE 36

36

ComputationalImaging.org

Results: Captured

Measured Blur Deblurred

slide-37
SLIDE 37

37

ComputationalImaging.org

Results: Captured

slide-38
SLIDE 38

38

ComputationalImaging.org

Results: Captured

slide-39
SLIDE 39

39

ComputationalImaging.org

Results: Captured

slide-40
SLIDE 40

40

ComputationalImaging.org

Results: Captured

slide-41
SLIDE 41

41

ComputationalImaging.org

Results: Captured

slide-42
SLIDE 42

42

ComputationalImaging.org

Results: Captured

slide-43
SLIDE 43

43

ComputationalImaging.org

Summary & Future Work

Next: Equiparallax regularization: applications Beyond 6-DOF, defocus Blind deblurring Generaliz lized co convolu lutio tional l blu lur usin sing LF Renderin ing Applied to RL deblurring 3D scenes, 6-DOF camera motion Proof of convergence to ML estimate Equiparallax regularization

slide-44
SLIDE 44

44

ComputationalImaging.org

Acknowledgments

QUT HPC Group George

slide-45
SLIDE 45

45

ComputationalImaging.org

Load Gantry and Lytro imagery Calibrate and rectify Lytro imagery Linear depth, volume fjlters Denoising: low-light, fog, dust, murky water Occluder removal: rain, snow, silty water

Light Field T

  • olbox for MATLAB

LF Synth: Bare-Bones Rendering