Multi-Modal Spectral Image Super-Resolution IVRL Prime Fayez - PowerPoint PPT Presentation

Multi-Modal Spectral Image Super-Resolution IVRL Prime Fayez Lahoud, Ruofan Zhou, Sabine Süsstrunk Image and Visual Respresentation Lab School of Computer and Communication Sciences École Polytechnique Fédérale de Lausanne 1

Multi-Modal Input ● Multi-Scale: different spatial resolutions Downsampled x3 (LR3) Downsampled x2 (LR2) 2

Multi-Modal Input ● Multi-Scale: different spatial resolutions ● Multi-Spectral: different spectral resolutions 14-channel spectral 3-channel RGB 3

Small Dataset ● Track 1 ○ 200 14-channel spectral images (LR2, LR3) ○ Solution: Upsampling + Stage-I ● Track 2 ○ 100 registered pairs ■ 14-channel spectral image (LR2, LR3) ■ 3-channel RGB image (HR) ○ Solution: Upsampling + Stage-I + Stage-II 4

Main Contributions ● LR2 + LR3 Upsampling Downsampled x2 High Resolution Candidate Downsampled x3 5

Main Contributions ● LR2 + LR3 Upsampling and Image Completion ● Transfer Learning Conv Net + Stage-I + Conv Net Stage-II 6

Nearest Neighbor and Image Completion 5 8 1 2 3 20 5 9 8 24 1 12 5 8 24 1 9 16 0 3 16 6 7 2 19 Downsampled x2 2 1 3 23 20 0 2 3 20 15 3 7 17 2 10 15 17 Downsampled x3 9 11 16 32 0 3 9 16 0 5 24 8 15 3 12 3 8 15 17 High Resolution Reconstruction 7

Nearest Neighbor and Image Completion Downsampled x2 Reconstruction Downsampled x3 8

Nearest Neighbor and Image Completion Downsampled x2 High Resolution Candidate Downsampled x3 R. Achanta, N. Arvanitopoulos, and S. Süsstrunk, "Extreme image completion," in the IEEE International 9 Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017.

Residual Learning Conv Net + High Resolution Candidate High Resolution Prediction ● Small model size ○ Stage-I: 1.6MB ○ Stage-II: 1.1MB ● Fast inference ● Low memory requirements 10

Transfer Learning Classical Learning Track1 Data Track2 Data Network 1 Network 2 Spectral Input Track 1 Origin 11 Color Input Track 2 Origin

Transfer Learning Spectral Image Classical Learning Super-Resolution Track1 Data Track2 Data Network 1 Network 2 Stage-I Spectral Input Track 1 Origin 12 Color Input Track 2 Origin

Transfer Learning Spectral Image Color Guided Classical Learning Super-Resolution Super-Resolution Track1 Data Track2 Data Network 1 Network 2 Stage-I Stage-II Spectral Input Track 1 Origin 13 Color Input Track 2 Origin

Transfer Learning Blind Residuals Conv Net + Stage-I High Resolution Candidate Track1 Prediction Color Guided Residuals + Conv Net Stage-II Track2 Prediction Color Guide 14

Transfer Learning: Example Output Stage-I Stage-II Output Error Histogram of Residuals 15

Comparative Results Metric Bicubic x2 EDSR Stage-I MRAE 0.11 0.10 0.08 SID 57.39 43.57 43.48 PSNR 36.07 37.27 37.44 Validation Track 1 16 Lim, B., Son, S., Kim, H., Nah, S. and Lee, K.M., "Enhanced Deep Residual Networks for Single Image Super-Resolution," in the IEEE conference on computer vision and pattern recognition (CVPR) workshops, 2017.

Comparative Results Metric Bicubic x2 EDSR Stage-I MRAE 0.11 0.10 0.08 SID 57.39 43.57 43.48 PSNR 36.07 37.27 37.44 Validation Track 1 Metric Bicubic x2 EDSR Stage-I Stage-II MRAE 0.13 0.16 0.10 0.09 SID 43.32 30.67 38.04 24.51 PSNR 36.48 37.13 37.02 39.17 Validation Track 2 17 Lim, B., Son, S., Kim, H., Nah, S. and Lee, K.M., "Enhanced Deep Residual Networks for Single Image Super-Resolution," in the IEEE conference on computer vision and pattern recognition (CVPR) workshops, 2017.

Conclusion ○ Multi-Modal Spectral Super Resolution ■ Use any signal you get your hands on! ■ Difficulty in obtaining new modalities can be overcome by transfer learning 18 https://github.com/IVRL/Multi-Modal-Spectral-Image-Super-Resolution

Thank you! https://github.com/IVRL/Multi-Modal-Spectral-Image-Super-Resolution {fayez.lahoud,ruofan.zhou,sabine.susstrunk}@epfl.ch 19