[PPT] - Multi-Modal Spectral Image Super-Resolution IVRL Prime Fayez PowerPoint Presentation

SLIDE 1

Multi-Modal Spectral Image Super-Resolution

IVRL Prime

Fayez Lahoud, Ruofan Zhou, Sabine Süsstrunk Image and Visual Respresentation Lab School of Computer and Communication Sciences École Polytechnique Fédérale de Lausanne

1

SLIDE 2

Multi-Modal Input

2

Multi-Scale: different spatial resolutions

Downsampled x3 (LR3) Downsampled x2 (LR2)

SLIDE 3

Multi-Modal Input

3

Multi-Scale: different spatial resolutions
Multi-Spectral: different spectral resolutions

14-channel spectral 3-channel RGB

SLIDE 4

Small Dataset

Track 1

○ 200 14-channel spectral images (LR2, LR3) ○ Solution: Upsampling + Stage-I

Track 2

○ 100 registered pairs ■ 14-channel spectral image (LR2, LR3) ■ 3-channel RGB image (HR) ○ Solution: Upsampling + Stage-I + Stage-II

4

SLIDE 5

5

Main Contributions

LR2 + LR3 Upsampling

Downsampled x2 Downsampled x3 High Resolution Candidate

SLIDE 6

6

Main Contributions

LR2 + LR3 Upsampling and Image Completion
Transfer Learning

Conv Net + Conv Net +

Stage-I Stage-II

SLIDE 7

Nearest Neighbor and Image Completion

7

5 9 8 24 1 12 3 16 6 7 2 19 2 1 3 23 20 15 3 7 17 2 10 9 11 16 32 3 8 15 3 12 3 8 5 8 1 2 3 20 9 16 5 24 15 17 5 8 24 1 2 3 20 15 17 9 16

Downsampled x2 Downsampled x3 High Resolution Reconstruction

SLIDE 8

8

Nearest Neighbor and Image Completion

Downsampled x2 Downsampled x3 Reconstruction

SLIDE 9

9

Nearest Neighbor and Image Completion

R. Achanta, N. Arvanitopoulos, and S. Süsstrunk, "Extreme image completion," in the IEEE International

Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017.

Downsampled x2 Downsampled x3 High Resolution Candidate

SLIDE 10

Residual Learning

Small model size

○ Stage-I: 1.6MB ○ Stage-II: 1.1MB

Fast inference
Low memory requirements

10

Conv Net +

High Resolution Candidate High Resolution Prediction

SLIDE 11

Transfer Learning

11

Classical Learning Track1 Data Track2 Data Network 1 Network 2

Spectral Input Color Input Track 1 Origin Track 2 Origin

SLIDE 12

Transfer Learning

12

Classical Learning Track1 Data Track2 Data Network 1 Network 2 Spectral Image Super-Resolution Stage-I

Spectral Input Color Input Track 1 Origin Track 2 Origin

SLIDE 13

Transfer Learning

13

Classical Learning Track1 Data Track2 Data Network 1 Network 2 Spectral Image Super-Resolution Color Guided Super-Resolution Stage-I Stage-II

Spectral Input Color Input Track 1 Origin Track 2 Origin

SLIDE 14

Transfer Learning

14

Conv Net + Conv Net +

Stage-I Stage-II

Blind Residuals Color Guided Residuals

High Resolution Candidate Track1 Prediction Track2 Prediction Color Guide

SLIDE 15

Transfer Learning: Example Output

15

Stage-II Stage-I Output Error Histogram

f

Residuals

SLIDE 16

Comparative Results

16

Metric Bicubic x2 EDSR Stage-I MRAE 0.11 0.10 0.08 SID 57.39 43.57 43.48 PSNR 36.07 37.27 37.44

Validation Track 1

Lim, B., Son, S., Kim, H., Nah, S. and Lee, K.M., "Enhanced Deep Residual Networks for Single Image Super-Resolution," in the IEEE conference on computer vision and pattern recognition (CVPR) workshops, 2017.

SLIDE 17

Comparative Results

17

Metric Bicubic x2 EDSR Stage-I Stage-II MRAE 0.13 0.16 0.10 0.09 SID 43.32 30.67 38.04 24.51 PSNR 36.48 37.13 37.02 39.17 Metric Bicubic x2 EDSR Stage-I MRAE 0.11 0.10 0.08 SID 57.39 43.57 43.48 PSNR 36.07 37.27 37.44

Validation Track 1 Validation Track 2

Lim, B., Son, S., Kim, H., Nah, S. and Lee, K.M., "Enhanced Deep Residual Networks for Single Image Super-Resolution," in the IEEE conference on computer vision and pattern recognition (CVPR) workshops, 2017.

SLIDE 18

Conclusion

18

https://github.com/IVRL/Multi-Modal-Spectral-Image-Super-Resolution

○ Multi-Modal Spectral Super Resolution ■ Use any signal you get your hands on! ■ Difficulty in obtaining new modalities can be

vercome by transfer learning

SLIDE 19

Thank you!

19

https://github.com/IVRL/Multi-Modal-Spectral-Image-Super-Resolution {fayez.lahoud,ruofan.zhou,sabine.susstrunk}@epfl.ch