April 4-7, 2016 | Silicon Valley LOCALLY NON-RIGID REGISTRATION FOR MOBILE HDR PHOTOGRAPHY Orazio Gallo, 04/06/2016 (work with Alejandro Troccoli, Jun Hu, Kari Pulli, and Jan Kautz)
WHAT IS “HIGH -DYNAMIC- RANGE”? Kinda depends on whom you ask… Displays Videogames Photography Photometry … 2
WHAT IS HDR? http://www.flickr.com/photos/lprowler/5704117093/ 3
WHAT IS HDR? 4
WHAT IS HDR? 6
WHY DO WE NEED REGISTRATION? 7
WHY DO WE NEED REGISTRATION? 8
WHY DO WE NEED REGISTRATION? 9
WHY DO WE NEED REGISTRATION? 10
RELATED WORK 11
FULLY NON-RIGID REGISTRATION Hu, Gallo, Pulli , Sun, “ HDR Deghosting: How to Deal with Saturation? ” IEEE CVPR 2013. 12
FULLY NON-RIGID REGISTRATION Hu, Gallo, Pulli , Sun, “ HDR Deghosting: How to Deal with Saturation? ” IEEE CVPR 2013. 13
FULLY NON-RIGID REGISTRATION Hu, Gallo, Pulli , Sun, “ HDR Deghosting: How to Deal with Saturation? ” IEEE CVPR 2013. 14
FULLY NON-RIGID REGISTRATION Hu, Gallo, Pulli , Sun, “ HDR Deghosting: How to Deal with Saturation? ” IEEE CVPR 2013. 15
FULLY NON-RIGID REGISTRATION Hu, Gallo, Pulli , Sun, “ HDR Deghosting: How to Deal with Saturation? ” IEEE CVPR 2013. 16
FULLY NON-RIGID REGISTRATION Hu, Gallo, Pulli , Sun, “ HDR Deghosting: How to Deal with Saturation? ” IEEE CVPR 2013. 17
RELATED WORK Speed Rigid registration Milliseconds (Ward ‘03, Tomaszewska and Mantiuk ‘07, …) Accelerated patch- based (Bao et al. ‘14) Seconds Rejection methods ( Gallo ‘09, Zhang and Cham ‘12, Oh et al. ‘15 …) Flow-based methods (Zimmer et al. ‘11, Zhang and Cham ‘12,…) Fully non-rigid registration Minutes (Hu et al. ‘12, Sen et al. ‘12, Hu et al. ‘13) Parallax Motion artifacts Little to no Quality and or artifacts 19 motion artifacts reduced dynamic range
WHAT’S THE CATCH? Original SIFT+warp Ours 25
A SPARSE-TO-DENSE APPROACH Compute the flow at sparse locations, Propagate the flow in an edge-aware fashion, and Merge the images in an error-tolerant way. 28
METHOD 29
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion 30
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion Reference Source 31
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion Reference Source 34
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion Reference Source 35
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion Reference 39
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion Source 40
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion Reference 41
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion Source 42
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion Reference 43
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion Source 44
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion Reference’s Luma Sparse flow 57
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion sparse samples latent signal 59
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion filtered sparse samples 60
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion sparse samples normalization map 61
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion filtered sparse samples reconstructed signal filtered normalization map 62
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion Luminance Pixel 67
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion Luminance Pixel 68
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion Luminance Pixel Gastal and Oliveira, “ Domain transform for edge-aware image and video processing ” ( SIGGRAPH '11) 69
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion 1 u v 1 Reference Luma, L Sparse flow, f Normalization map N 70
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion Reference 71
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion Source 72
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion Reference 73
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion Warped source 74
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion Warped source 75
Corners and Sparse-to- Error-tolerant Filter matches matches dense warp fusion 76
PERFORMANCE 77
Execution time for a pair of 5MP images 78
Related work Speed Rigid registration Milliseconds (Ward ‘03, Tomaszewska and Mantiuk ‘ 07, …) Accelerated patch-based (Bao et al. ‘14) Seconds Rejection methods (Gallo ‘09, Zhang and Cham ‘ 12, Oh et al. ‘ 15 …) Flow-based methods (Zimmer et al. ‘11, Zhang and Cham ‘ 12,…) Fully non-rigid registration Minutes (Hu et al. ‘12, Sen et al. ‘12, Hu et al. ‘ 13) Parallax Motion artifacts Quality No artifacts and or 79 motion artifacts reduced dynamic range
Related work Speed Milliseconds Accelerated patch-based (Bao et al. ‘14) Seconds (VGA resolution) Fully non-rigid registration Minutes (Hu et al. ‘12, Sen et al. ‘12, Hu et al. ‘ 13) Parallax Motion artifacts Quality No artifacts and or 80 motion artifacts reduced dynamic range
Related work Speed Milliseconds Accelerated patch-based (Bao et al. ‘14) Seconds (5MP resolution) Fully non-rigid registration Minutes (Hu et al. ‘12, Sen et al. ‘12, Hu et al. ‘ 13) Parallax Motion artifacts Quality No artifacts and or 81 motion artifacts reduced dynamic range
What about visually? Ours Bao et al.’s (2.5MP) 82
TO SUM UP… Contributions A fast registration algorithm >11x faster than the fastest published method We propose to use a sparse-to-dense approach CUDA-based sparse-to-dense propagation CUDA-based robust image fusion 83
MORE RESULTS Reference 85
MORE RESULTS Source 86
MORE RESULTS Reference 87
MORE RESULTS Warped source 88
MORE RESULTS HDR 89
MORE RESULTS Naïve fusion Our result 90
MORE RESULTS Reference 91
MORE RESULTS Source 92
MORE RESULTS Reference 93
MORE RESULTS Warped source 94
MORE RESULTS HDR 95
MORE RESULTS Naïve fusion Our result 96
MORE RESULTS Reference 97
MORE RESULTS Source 98
MORE RESULTS Reference 99
MORE RESULTS Warped source 100
MORE RESULTS HDR 101
MORE RESULTS Naïve fusion Our result 102
April 4-7, 2016 | Silicon Valley THAT’S ALL. O. Gallo A. Troccoli J. Hu K. Pulli J. Kautz
Recommend
More recommend
