Chapter 4 Multiframe Super-resolution Image Reconstruction 1 - PDF document

Chapter 4 Multiframe Super-resolution Image Reconstruction 1 Multi-frame SRIR (Video Enhancement) Low resolution video Our method LR image Source: https://users.soe.ucsc.edu/~milanfar/software/sr-datasets.html 2 1

Multi-frame SRIR (Video Enhancement) Input Video High-definition Video 3 Outline 1. High-resolution Image Reconstruction 1. High-resolution Image Reconstruction 2. Video Still Enhancement Models 2. Video Still Enhancement Models o Classical o Classical o Tight-frame o Tight-frame o Low-rank o Low-rank 3. Experiments 3. Experiments 4 2

Super-resolution Image Reconstruction (SRIR) Not one, but many lens— compound eyes 5 Super-resolution Image Reconstruction (SRIR) 6 3

Modeling of SRIR a b low-resolution pixel c d given intensity = (a+b+c+d)/4 LR 2 LR 1 high-resolution pixel d d LR 3 LR 4 7 Modeling of SRIR 8 4

Modeling of SRIR 9 Modeling of SRIR 10 5

Outline 1. High-resolution Image Reconstruction 2. Video Still Enhancement Models o Classical o Tight-frame o Low-rank 3. Experiments 11 11 Video Still Enhancement Tight-frame method Bilinear interpolation A 352-by-288 video using 21 frames from 1 frame from a video recorder [C., Shen, & Xia, ACHA 07] 30 frames/second frames are not aligned at exactly half-pixel length 12 12 12 6

Aligning the Frames Reference frame displacement t 13 Sub-pixel Displacement High- resolution pixels Ideal low- Displaced low- resolution pixel resolution position pixel 14 7

Modeling of Video Still Enhancement 15 Modeling of SRIR 16 8

Classical Approach [Tsai & Huang, 84] 17 Outline 1. High-resolution Image Reconstruction 2. Video Still Enhancement Models o Classical o Tight-frame o Low-rank 3. Experiments 18 18 9

Tightframe Approach [C 2 + S 2 , SISC (2003)] High- 1 1 1 resolution 4 2 4 Low- pixels 1 1 resolution 1 2 2 pixel 1 1 1 4 2 4 Averaging process = a lowpass filter with refinement mask 19 Piecewise Linear Tight Frame 20 10

From LR Images to HR Image 21 Key Observation 22 11

Tightframe SRIR 23 Outline 1. High-resolution Image Reconstruction 2. Video Still Enhancement Models o Classical o Tight-frame o Low-rank 3. Experiments 24 24 12

Low-rank Approach [ArXiv: 170406196] 25 Motion Matrices in 1D 26 13

Decomposition of the Motion Matrices 27 Creating Low Rank Structure 28 14

Variational Formulation 29 Motion Matrices in 2D 30 15

Creating Low Rank Structure Low-rank Super-resolution Model 32 16

Nuclear Norm [Candes, Recht, 09; Recht, Fazel, Parrilo, 10] 33 Motion Estimation 34 17

Local Motion Estimation [C. Gilliam and T. Blu, 2015] 35 Global Motion Estimation [C., Shen and Xia, ACHA, 2007] 36 18

Outline 1. High-resolution Image Reconstruction 2. Video Still Enhancement Models o Classical o Tight-frame o Low-rank 3. Experiments 37 37 Bookshelf Video Upsampled by 2 Single frame with 21 frames with bilinear interpolation TV regularization 21 frames with 21 frames with tightframe nuclear norm 38 38 19

Disk Video Upsampled by 2 LR video MAP (2015) tightframe (2007) reference frame sparse direct’l (‘10) nuclear 39 39 Source: https://users.soe.ucsc.edu/~milanfar/software/sr-datasets.html Text Video Upsampled by 2 LR video MAP tightframe reference frame sparse directional nuclear 40 40 Source: https://users.soe.ucsc.edu/~milanfar/software/sr-datasets.html 20

Alpaca Video Upsampled by 2 MAP tightframe LR video reference frame sparse directional nuclear 41 41 Source: https://users.soe.ucsc.edu/~milanfar/software/sr-datasets.html Alpaca Video Upsampled by 4 MAP tightframe LR video reference frame sparse directional nuclear 42 42 Source: https://users.soe.ucsc.edu/~milanfar/software/sr-datasets.html 21

Alpaca Image Upsampled by 4 reference image nuclear 43 43 EIA Video Upsampled by 4 LR video MAP tightframe reference frame sparse directional nuclear 44 44 Source: https://users.soe.ucsc.edu/~milanfar/software/sr-datasets.html 22

EIA Image Upsampled by 4 reference image nuclear sparse directional 45 45 EIA Image Upsampled by 4 L1-robust, Farsiu et al. (2004) nuclear 10 x 10 (.jpg) 46 46 23

Book-shelf Upsampled by 2 LR video MAP tightframe reference frame sparse directional nuclear 47 47 Bookshelf Video Upsampled by 2 tightframe MAP LR video reference frame nuclear sparse directional 48 48 24

Boat Video Upsampled by 2 20.88dB 18.66dB LR video MAP tightframe 25.76dB 26.94dB true image sparse directional nuclear 49 49 Single Frame Upsampling interpolate/inpaint t interpolate/inpaint interpolate/inpaint t 50 25

Input Video Qian et al. Siggraph 09: Upsampled by bicubic Level 6 Tightframe 51 Input Video Qian et al. Siggraph 09 Level 6 Tightframe Upsampled by bicubic No thresholding = completely parameter free 52 26

Concluding Remarks R. Zhao and R. Chan, arXiv:1704.06196 53 53 27