Data-driven Optimal Camera Response Selection and Design for - - PowerPoint PPT Presentation

2018.07.11

Data-driven Optimal Camera Response Selection and Design for Spectral Reconstruction

Yinqiang Zheng (郑银强）

1. National Institute of Informatics 2. SOKENDAI （日本国立信息学研究所） （综合研究大学院大学）

Data-driven Filter Response Selection and Design VALSE

数据驱动、面向光谱重建的最优相机响应曲线 的自动选择和设计

2018.07.11

Main Contributors

1 Beijing Institute of Technology National Institute of Informatics/Saitama University

• Prof. Hua Huang
• Prof. Ying Fu
• Prof. Imari Sato

Shijie Nie Lin Gu Antony Lam Tao Zhang

2018.07.11

Color: RGB vs. Spectrum

2 3 Channels N Channels (N>>3)

2018.07.11

Electromagnetic Wave (Light)

3

https://study.com/academy/lesson/electromagnetic-waves-definition-sources-properties-regions.html

2018.07.11

Spectra of Ordinary Light Sources

4 Xenon Lamp（氙气灯） HID Lamp (高辉度放电灯)

2018.07.11

Applications of Hyperspectral Imaging

5

Remote Sensing Agriculture Medical Diagnostics Bioscience

2018.07.11

Hyperspectral Imaging Devices

6 Spectrometer-Single Point Line-Scan Spectral Camera Filter-Scan Spectral Camera Mechanical Electronic Hamamatsu LQT Quantum Pixelteq

2018.07.11

Single-Point Devices（单点分光仪）

7 Spectrometer-Single Point Hamamatsu C10082MD Diffraction Prism/Grating (分光棱镜、光栅） Hamamatsu S11510 (2048 x 64 pixels） Hitachi Fluorescence Spectrometer F-7100 Photomultiplier Tube (光电倍增管） Single-Pixel Camera

2018.07.11

Line Imaging Devices （线分光相机）

8 Line-Scan Spectral Camera LQT Quantum CCD Array DJI Phantom Static Object

2018.07.11

Major Drawbacks of Scan-based Spectral Cameras

9 Moving Object： Rolling Shutter畸变（果冻效应） High Cost Noisy Images

2018.07.11

Computational Reconstruction from Compressed Sensing Measurement

10 Random Sampling in the Spatial Domain Sparse Coding Shallow Network Deep Network Coded Aperture Representative Work:

• 1. Single disperser design for coded aperture snapshot spectral

imaging (CASSI), Applied Optics, 2008.

• 2. Single-shot compressive spectral imaging with a dual-

disperser architecture (DD-CASSI), Optical Express, 2007.

• 3. Spatial-spectral encoded compressive hyperspectral imaging,

TOG, 2014.

2018.07.11

Uniformly Downsampled Variants

11

A Prism-Mask System for Multispectral Video Acquisition，TPAMI, 2011. Hybrid-resolution spectral video system using low-resolution spectral sensor ， Optical Express, 2014.

2018.07.11

Low-Resolution Spectral Image and RGB/Gray Fusion

12

• 2. Acquisition of High Spatial and Spectral

Resolution Video with a Hybrid Camera System，IJCV, 2013.

• 1. High-resolution Hyperspectral Imaging via

Matrix Factorization，CVPR, 2011.

• 3. High speed hyperspectral video with a dual-camera

architecture, CVPR, 2015.

• 4. Optics and methods for hybrid resolution

spectral imaging, Applied Optics, 2015.

2018.07.11

Challenges in Hybrid Fusion – Image Alignment

13 Physical Alignment – Hard, but can be done with efforts. Algorithmic Alignment – Good if you know the transformation and the resolution is high. Extremely Low Resolution

2018.07.11

RGB Camera Color Imaging Mechanism

14 1 Filter Arry+1 Sensor 3 Filters+3 Sensors

2018.07.11

Computational Reconstruction from RGB/Multispectral Images

15 Sparse Coding Shallow Network Deep Network RGB Image Material Reflectance Spectra Ordinary Illumination Spectra Major Advantage: 1. Variation in spectral domain is much less than in space domain.

• 2. To capture multispectral images is fast.

2018.07.11

RGB-to-Spectrum via Manifold based Mapping

16 Jia et al., From RGB to Spectrum for Natural Scenes via Manifold-Based Mapping, ICCV, 2017.

2018.07.11

RGB-to-Spectrum via CNN

17 Ying Fu, Tao Zhang, Yinqiang Zheng, Hua Huang, Joint Camera Spectral Sensitivity Selection and Hyperspectral Image Recovery, ECCV, 2018, accepted.

2018.07.11

Comparison with State-of-the-Art

18 RBF: Training based spectral reconstruction form a single RGB image, ECCV, 2014. SR: Sparse recovery of hyperspectral images from natural RGB images, ECCV, 2016. MM: From RGB to Spectrum for Natural Scenes via Manifold-Based Mapping, ICCV, 2017.

2018.07.11

RGB Spectral Response Database

19

Jiang et al., What is the Space of Spectral Sensitivity Functions for Digital Color Cameras? WACV, 2013

2018.07.11

Sensitivity to RGB Response

20

• Using our CNN method for reconstruction
• Conducting experiment using all camera responses one by one with the same setting

Ref: Filter selection for hyper-spectral estimation, ICCV, 2017. Ref: Sparse recovery of hyperspectral images from natural RGB images, ECCV, 2016.

2018.07.11

Camera is Doing Convolution

21 Filter+CCD/CMOS Sensor is actually a convolution operator along the spectral axis, and the stride is 1 (sum of dot product). The convolutional kernel is the filter response function. Convolution Kernel Response Function

2018.07.11

Convolutional Neural Network

22 Here is an example of 2D convolutional kernel in the spatial domain. Note that, convolution has been implemented in all existing deep learning toolboxes. So, we can reuse them for our filter designing purpose.

2018.07.11

To Select the Best Response via CNN

23 Selection CNN Spectral Reconstruction CNN Response Database

1 1 2 2 28 28 1 1 2 2 28 28 1 28 1 28 1 1 2 2 28 28

... ... ,..., 0,[ ,..., ]sparse. ...

r r r r g g g g b b b b

c c c c c c c c c c c c                           ，

2018.07.11

Selection Results

24 Sparsity Constraint + Nonnegative Constraint Huge Acceleration: 28 -> 1

2018.07.11

Go beyond Existing RGB Responses

25 Channel #1 Channel #2 Channel #3 Nonnegative Negative Smooth Shape Spiky Shape

2018.07.11

To Design Optimal Response

26 Shijie Nie, Ling Gu, Yinqiang Zheng, Antony Lam, Nobutaka Ono and Imari Sato, Deeply Learned Filter Response Functions for Hyperspectral Reconstruction, CVPR，2018.

2018.07.11

Synthetic Experiment Results

27 Training Loss on CAVE Designed Response Curves on CAVE Comparison of Recovered Spectra from Our Method and Existing Ones

2018.07.11

Synthetic Experiment Results

28

2018.07.11

Realized Filters

29 Blue: Filter 1 Dotted: Designed/Solid: Measured. Red: Filter 2 Dotted: Designed/Solid: Measured.

2018.07.11

Prototype Two-Band Camera

30 Grayscale Image 1 Grayscale Image 2

2018.07.11

Remaining Challenges 1: Limited Datasets

• Common Datasets:
• 1. ICVL, 201 images, Natural Illumination
• 2. NUS, 66 images, Mixed Illuminations
• 3. Harvard, 50 images, Outdoor
• 4. CAVE, 32 images, D65 (but normalized)
• The Metamerism Issue:

31

2018.07.11

Remaining Challenges 2: Are Hyper- Spectral Images Really Necessary?

32 GT RGB (97.07%) Recon.(98.36%) HIS (99.12%) Object Classification RGB Multi-Spectral Hyper-Spectral

2018.07.11

Remaining Challenges 3: From Spectral Reconstruction to High-Level Tasks

33 RGB Image Spectral Sensing: Low-Level Vision Task Classification Face Detection Detection/Recognition/Classification: High-Level Vision Task

2018.07.11

Open Question: Is Human Eye Perception Optimal?

34 vs. For spectral reconstruction, and on our limited datasets, the answer is negative.

2018.07.11

Thank you very much!

35