CS-ToF: High-resolution Compressive time-of-flight imaging Fengqiang - PowerPoint PPT Presentation

CS-ToF: High-resolution Compressive time-of-flight imaging Fengqiang Li, Chia-kai Yeh, Kuan He, Oliver Cossairt (Northwestern University) Huaijin Chen, Adithya Pediredla, Ashok Veeraraghavan (Rice University) fengqiang.li@u.northwestern.edu 1

Time of flight Image credit: Donald Griffin 2

Time-of-flight (ToF) imaging Pulsed laser Clock Object Detector Goyer et al., Bulletin of the American Meteorological Society, 1963 3

Time-of-flight (ToF) imaging Pulsed laser Clock Object Detector d Goyer et al., Bulletin of the American Meteorological Society, 1963 4

Time-of-flight (ToF) imaging Pulsed laser Clock Object Detector d d = 1 2 ⋅ c ⋅ t Goyer et al., Bulletin of the American Meteorological Society, 1963 5

Time-of-flight (ToF) imaging Pulsed laser Clock Object Detector d d = 1 2 ⋅ c ⋅ t Pulsed light based ToF q Raster scanning Goyer et al., Bulletin of the American Meteorological Society, 1963 6

Continuous-wave ToF !(#) # = 0 Laser diode !(#) Controller ToF camera %(#) r (#) Schwarte et al., SPIE Proceeding, 1997 7

Continuous-wave ToF !(#) # = 0 Laser diode !(#) Controller * ( p # ) ToF camera s (#) %(#) r (#) ( ) ( ) Schwarte et al., SPIE Proceeding, 1997 8

9

HCI Autonomous cars Robotics VR/AR 10

HCI Autonomous cars Non-Line-of-Sight imaging Robotics VR/AR Transient imaging 11

Low spatial resolution o 640 ✕ 480 pixels (0.3 mega) vs 120-megapixel CMOS 12

Low spatial resolution o 640 ✕ 480 pixels (0.3 mega) vs 120-megapixel CMOS o Extra circuits for each pixel o Limited wafer size 13

Previous work Work directly on ToF camera output: Edge guidance o Defocus debluring o Fuse ToF output with a second camera: RGB camera o Stereo, Photometric stereo, Shape from polarization o 14

Optical multiplexing Spatial light modulator-SLM ToF camera Objects (High resolution) (low resolution) q Multiple Pixels (e.g. 3 ✕ 3) on spatial light modulator projected on One Pixel of ToF camera 15

Motivation for optical multiplexing q Higher resolution compared to algorithm based method q No need to fusion with different image modalities q Resolution is dependent on SLM 16

Compressive sensing … … ToF output (y) Pattern on modulator (M) 17

Phasor Intensity (a) depth (d) q ToF output: intensity (a) and depth (d) q Phasor: a ∘ e $ %&' ( ) = a ∘ e $+ 18

Imaging forward model x = ! " ∘ e %& ' q High resolution scene projection on modulator (x) ( 19

Imaging forward model x = ! " ∘ e %& ' M q High resolution scene projection on modulator (x) q Spatial light modulator pattern ( M ) () 20

Imaging forward model C x = ! " ∘ e %& ' M q High resolution scene projection on modulator (x) q Spatial light modulator pattern ( M ) q Translation matrix from modulator to ToF camera ( C ) ()* 21

Imaging forward model C x = ! " ∘ e %& ' M y = ! - ∘ e %& . q High resolution scene projection on modulator (x) q Spatial light modulator pattern ( M ) q Translation matrix from modulator to ToF camera ( C ) q ToF camera output ( y ) ( = *+, 22

Imaging forward model ToF Output y . with modulation pattern M . q 0 12 3 45 67 ( " 0 >2 3 45 ?7 ( " ! " ( " 0 18 3 45 69 ( # 0 >8 3 45 @9 ( # ! # ( # 0 > 3 45 @ ( $ = = 0 1: 3 45 6; ( $ 0 >: 3 45 @; ! $ = ( $ ) . . . . . . . . . . ( & 0 1< 3 45 6= ( & 0 >< 3 45 @A ! & ( & * = +,) = () 23

Optimization + − -. / + 1Φ(x) x = arg min ! * / + G ; x 8 / Φ . = TV . = 7 G : x 8 8 24

Simulation q High resolution scene o Middlebury 3D Datasets o Size: 1140x912 q A t – simulated system A = CM t o M t – Hadamard multiplexing patterns on DMD at t -th measurement o C – mapping matrix, defined as spatial down-sampling by averaging q Low resolution measurements • Simulated through via y t = A t x • Size: 120 × 153 25

(e). HR reconstruction (15%) (c). HR reconstruction (50%) (d). HR reconstruction (25%) (a). Ground truth (b). Original LR ToF measurement Intensity Intensity Intensity Intensity Intensity 26

(e). HR reconstruction (15%) (c). HR reconstruction (50%) (d). HR reconstruction (25%) (a). Ground truth (b). Original LR ToF measurement Intensity Intensity Intensity Intensity Intensity (i). HR reconstruction (25%) (j). HR reconstruction (15%) (g). Original LR ToF measurement (h). HR reconstruction (50%) (f). Ground truth Phase Phase Phase Phase Phase 27

Ground truth Original LR ToF measurement HR reconstruction (15%) 28

Original LR ToF measurement HR reconstruction (15%) Ground truth 29

Prototype q Texas Instrument DLP 4500: 1140×912 pixels q ToF camera: Texas Instrument OPT 8241 320 × 240 (186 × 200 pixels) 30

Original LR ToF Measurement Pixel scanning 31

3D scene 32

Original LR ToF measurement HR Reconstruction (no compression) Depth in m Depth in m HR Reconstruction (0.25) HR Reconstruction (0.6) Depth in m Depth in m 33

No compression CS: 0.25 Native CS: 0.6 34

Original LR ToF measurement HR reconstruction (no compression) HR reconstruction (60%) HR reconstruction (25%) 35

No compression Native CS: 0.6 CS: 0.25 36

Depth resolution Depth in mm Ground truth depth Depth values with 25dB SNR q No depth resolution improvement q Depth resolution is better than bicubic interpolation 37

38

Project Page http://compphotolab.northwestern.edu/project/cs-tof-high-resolution-compressive-time-of-flight-imaging/ 39