Xun Cao ( ) School of Electronic Science & Engineering Nanjing - - PowerPoint PPT Presentation

▶
xun cao
SMART_READER_LITE
LIVE PREVIEW

Xun Cao ( ) School of Electronic Science & Engineering Nanjing - - PowerPoint PPT Presentation

Mar 23, 2023 357 likes •1.02k views

High Resolution Spectral Video Capture & Computational Photography Xun Cao ( ) School of Electronic Science & Engineering Nanjing University caoxun@nju.edu.cn Dec 30th, 2015 Computational Photography Computational Photography

slide-1
SLIDE 1

High Resolution Spectral Video Capture & Computational Photography

Dec 30th, 2015

School of Electronic Science & Engineering Nanjing University

Xun Cao (曹汛)

caoxun@nju.edu.cn

slide-2
SLIDE 2

Computational Photography Computational Photography

2

Computational photography refers broadly to computational imaging techniques that enhance or extend the capabilities of digital photography. The output of these techniques is an ordinary photograph, but one that could not have been taken by a traditional camera. (Wikipedia) Computational photography is an emerging new field created by the convergence

  • f computer graphics, computer vision and photography. Its role is to overcome

the limitations of the traditional camera by using computational techniques to produce a richer, more vivid, perhaps more perceptually meaningful representation of our visual world. (CMU Course Introduction)

slide-3
SLIDE 3

Spatial Temporal Spectral（Color） Dynamic Range SD<720p HD 1920*1080 Gigapixel UHD 3840*2160

2D

Stereo Multiview Light Field Gray scale RGB Multispectral Hyperspectral

10Hz

30Hz 60Hz Ps.Fs 8 bit 10/12 bit 24 bit 120Hz

CP for various imaging dimensions

Depth & View （3D）

slide-4
SLIDE 4

Computational Imaging Technology & Engineering

  • Spectral Camera

– High Resolution Spectral Video Camera: PMIS

  • Multiview Stereo Camera Array

– High Accuracy 3D Reconstruction

  • Super-Resolution Camera

– Nano-Scale Pixel Camera – Gigapixel on Single Chip

The lab focuses on 3 kinds of computational cameras

slide-5
SLIDE 5

Grayscale Imaging

sensor light source scene

( ) I  

( ) R  ( ) S 

( ) ( ) K R S d     

K

slide-6
SLIDE 6

Color Imaging

light source scene sensor

( ) I  

( ) R  ( ) S 

( ) ( )

R

R R S d      ( ) ( )

G

G R S d      ( ) ( )

B

B R S d     

, , R G B （ ）

slide-7
SLIDE 7

Spectral Imaging

？ ？

light source scene Imaging system

( ) I  

( ) R  ( ) R 

slide-8
SLIDE 8

Spatially varying filter

  • Prof. S.Nayar

Columbia

  • Filtered Camera based Spectrum Imaging

[Kidono07] [Gat00][Yamaguchi06][Schechner02]…

Color Filter Array Filter wheel Programmable Filter Filter Scanning

Key idea: Trade time for spectrum Shortcomings:

  • Incapable of capturing dynamic

scenes

  • Low spectrum resolution

Related Work (1)

PAMI ’02

slide-9
SLIDE 9
  • Prof. D.Brady (Duke Univ.) CASSI: Applied Optics SPIE, JOSA’06-09

2D Imaging + reconstruction Spectrum resolution: 6 nm Spatial resolution: 256 x 248 Limited spatial resolution Limited accuracy Time-consuming reconstructing (20min / frame)

Key Idea: Coded Aperture

  • Coded Aperture Snapshot Spectral Imager (CASSI)

[Brady’06] [Willett’07] [Gehm’07] [Wagadarikar’08]

Related Work (2)

slide-10
SLIDE 10
  • E. Dereniak

Applied Optics SPIE, JOSA’95-08

different linear “projections”

  • f the spectral data cube

[JOSA’08]

  • Computed Tomography Imaging Spectrometer

[Descour95] [Descour01] [Vandervlugt07] [Hagen08]…

Shortcomings: Low Resolution Difficult to Calibrate High Computational Cost Key Ideas: CT Projections + Reconstruction

  • Prof. E. Dereniak

Arizona

Computed Tomographic Imaging Spectrometer

CTIS

Related Work (3)

slide-11
SLIDE 11

Our Spectral Video Camera - PMIS

2008~2010: Prism-Mask Imaging Spectrometer (PMIS1)

– Directly capture multispectral video – High spectra-resolution – Low cost – Easy setup and calibration

2011~2014: Hybrid-Camera PMIS2

– Both high spectral and spatial resolution – Real-time hyperspectral video capture

2014~now Scene-Adaptive PMIS3

– Space-time coded modulation – Spectral video capture with improved accuracy and efficiency

slide-12
SLIDE 12

A glance at PMIS1

capturing system mask Pointgrey grayscale camera 2248x2048 @15fps

slide-13
SLIDE 13

Prism

Occlusion Mask

Grayscale Camera Re-generated RGB Video

System Principle

Camera System

GIF source: Wiki

slide-14
SLIDE 14

A Typical Camera

sensor array lens camera

slide-15
SLIDE 15

Camera & Prism

Spectra Overlap! prism sensor array lens camera

slide-16
SLIDE 16

Camera & Mask

sensor array lens camera mask

slide-17
SLIDE 17

Camera & Mask & Prism

Spectra Overlap! sensor array lens camera mask

slide-18
SLIDE 18

Spectral Resolution

prism image plane mask aperture grayscale camera

a

 

 

sin ( ) sin ( ( )) sin ( ( )) sin ( ( )) n           

f

( ) W S

( ) (tan( ( ) ) tan( ( ) ))

e s

W S f a a          

s

e

( )

spec

W S R  

CCD cell size

slide-19
SLIDE 19

Spectral Resolution

prism image plane mask aperture

  • Tradeoff Spatial/Spectral Resolution

( ) (tan( ( ) ) tan( ( ) ))

e s

W S f a a          

f

slide-20
SLIDE 20

Spectral Resolution

  • Tradeoff Spatial/Spectral Resolution

( )

spec

W S R  

( ) (tan( ( ) ) tan( ( ) ))

e s

W S f a a          

prism image plane mask aperture

f

slide-21
SLIDE 21

Spectral Resolution

  • Tradeoff Spatial/Spectral Resolution

( )

spec

W S R  

( ) (tan( ( ) ) tan( ( ) ))

e s

W S f a a          

prism image plane mask aperture

f

slide-22
SLIDE 22

Spectral Resolution

  • Tradeoff Spatial/Spectral Resolution

( )

spec

W S R  

( ) (tan( ( ) ) tan( ( ) ))

e s

W S f a a          

prism image plane mask aperture

f

slide-23
SLIDE 23

Spatial Resolution

prism image plane mask aperture Spectra Overlap!

  • Small Mask Hole Distance
slide-24
SLIDE 24

prism mask aperture Unused Pixels image plane

Spatial Resolution

  • Large Mask Hole Distance
slide-25
SLIDE 25

Spatial Resolution

prism mask aperture Perfect Alignment image plane

In practice, we can use a uniform mask

 d D

(tan( ( )) tan( ( )))

e s

D d          

Design Mask Hole Distance

slide-26
SLIDE 26

Device Calibration

slide-27
SLIDE 27

Calibration Overview

Spectrum Calibration Geometry Calibration Radiance Calibration

Mapping Position to Wavelength Geometry Distortion caused by the prism (Smile Distortion) Non-constant CCDSensitivity

slide-28
SLIDE 28

Spectrum Calibration

Spectrum Calibration Geometry Calibration Radiance Calibration

slide-29
SLIDE 29

Spectrum Calibration

Ground truth fluorescent spectra

captured spectra

slide-30
SLIDE 30

Spectrum Calibration

target spectra

Ground truth fluorescent spectra

Warp

slide-31
SLIDE 31

Spectrum Calibration

prism image plane mask aperture

Non linear , but smooth curve !

a

f

x  sin ( ) tan arcsin( sin( arcsin( ))) x f a n n

 

           

  • Mapping Function : Wavelength <-> Position
slide-32
SLIDE 32

Geometry Calibration

Spectrum Calibration Geometry Calibration Radiance Calibration

slide-33
SLIDE 33

Geometry Calibration

Predefined mask pattern captured image geometry calibrated image

slide-34
SLIDE 34

Radiance Calibration

Spectrum Calibration Geometry Calibration Radiance Calibration

captured radiance genuine radiance

slide-35
SLIDE 35

Radiance Calibration

genuine radiance

wavelength sensitivity light input intensity

captured radiance locally constant assuming

( ) ( )

( ) ( ( ) ( ) )

b a

z z

I z g c l d

 

   

( ), ( ) c l  

1( ( ))

( ) ( )( )

b a

g I z l c    

 

slide-36
SLIDE 36

Application 1: Human Skin Detection

  • The ‘W’ pattern in human skin reflectance
  • [Angelopoulou01]
slide-37
SLIDE 37

Application 1: Human Skin Detection

slide-38
SLIDE 38

Application 2: Material Discrimination

RGB Image IR Image The differences in IR Our measurement

slide-39
SLIDE 39

PMIS1 Conclusions

  • Compared to Traditional Spectrometers
  • Passive Multispectral Video Capture
  • High spectral resolution
  • Tradeoff spectral and spatial resolution
  • Easy setup and calibration
  • Applications
  • Skin Detection
  • Material Recognition
  • Illumination Identification

PMIS1： A Prism-Mask System for Multispectral Video Acquisition,

IEEE Intl’ Conf. Computer Vision (ICCV), 2009 , Oral IEEE Trans. Pattern Anal. Mach. Intell. (PAMI), 2011 High Resolution Multispectral Image Capture，US Patent.20140085502

slide-40
SLIDE 40

PMIS1: limitations

  • Light throughput is limited by

– occlusion mask – relatively small aperture

  • Can NOT achieve both high spatial and spectral

resolution

– Limited CCD resolution – Spatial resolution (1000 pixels)

slide-41
SLIDE 41

Scene or Object

Prism

Occlusion Mask

Gray Camera RGB Camera

High-Spatial Low-Spectral Resolution Video Low-Spatial High-Spectral Resolution Video

PMIS2: Hybrid Camera System

slide-42
SLIDE 42

PMIS2: System Pipeline

Propagation

slide-43
SLIDE 43

PMIS2: System Implementation

slide-44
SLIDE 44

High Spatial Resolution RGB Video

RGB Camera Gray Camera

Low Spatial Resolution Multispectral Video

Propagation Algorithm

slide-45
SLIDE 45

Propagation Algorithm

, ,

( ) ( ) ( ) ( )

r s r s

RGB xy c k k k k k ij RGB xy c R G B k k k

G d G d G d G d

   

  

  

ms ms

/ , e.g. for red channel

k ij k

R R  

slide-46
SLIDE 46

Propagation Algorithm

Ground Truth Data Evaluation (11 datasets, .aix, .mat)

  • Spectral Database, University of Joensuu Color Group
slide-47
SLIDE 47

Propagation Algorithm

Temporal Enhancement Error 7.8% 4.3%

slide-48
SLIDE 48

PMIS2: Results & Applications

slide-49
SLIDE 49

Spatial Comparison

PMIS1: Prism-Mask Multispectral-Video Imaging System (ICCV’09, PAMI’2011) PMIS2: Hybrid Camera Multispectral Video Imaging System (CVPR’2011, IJCV’2014)

slide-50
SLIDE 50

Spatial Comparison

PMIS1: Prism-Mask Multispectral-Video Imaging System (ICCV’09, PAMI’2011) PMIS2: Hybrid Camera Multispectral Video Imaging System (CVPR’2011, IJCV’2014)

slide-51
SLIDE 51

Application 3: illumination recognition

slide-52
SLIDE 52

Application 3: Automatic White Balance

slide-53
SLIDE 53

Application 3: mixed illumination

Fluorescent light

Tungsten light

slide-54
SLIDE 54

Original Frame Fluorescent Light Tungsten Light Our Result

Application 3: mixed illumination

slide-55
SLIDE 55

Spectral Comparison Poster vs. Water Color

RGB Space Spectral Space

slide-56
SLIDE 56

Application 4: Tracking

slide-57
SLIDE 57

PMIS2 Conclusions

  • Spectral Resolution
  • (1~6nm, adjustable)
  • Spatial Resolution
  • (1024 ×1024 )
  • Temporal Resolution
  • (15fps, Real-Time Capture)
  • Additional Applications by PMIS2
  • Automatic White balance - Object Tracking

High Resolution

PMIS2： Acquisition of High Spatial & Spectral Resolution Video with a Hybrid Camera System,

IEEE Intl’ Conf. Computer Vision & Pattern Recognition. (CVPR), 2011 International Journal of Computer Vision (IJCV), 2014 A Computational Spectral Video Capture Device，China Patent. ZL201110212923.X

slide-58
SLIDE 58

PMIS3 Basic Idea

PMIS1 & PMIS2

Fixed-Pattern Mask

Can we dynamically change the mask adaptive to the scene content ?

slide-59
SLIDE 59

Accuracy improvement Targeted spectral acquisition by annotating regions of interest

PMIS3 Prototype and Capturing Results

Spatial Light Modulation

PMIS3: Content-Adaptive High-resolution Spectral Video Acquisition

Optics Letters, 39(15), pp.1464-1466, 2014 Optics Express, 22(16), pp.19348-19356, 2014 IEEE CVPR, pp. 1684-1692, 2015

slide-60
SLIDE 60
  • PMIS: High Resolution Spectral Video Camera

– vs Traditional Spectrometer: Snapshot Capability (Video) – vs CTIS / CASSI:

  • Real-Time Video Output
  • Low Reconstruction Error
  • Improved Resolution <-> Low Cost
  • PMIS Hyperspectral Video Datasets Available :

Summary

Spectra Viewing Software Ma C, Cao X, Dai, Q, et al. IJCV 2014

slide-61
SLIDE 61

Commercial Spectral Video Cameras

Camera Camera Wavelength Wavelength Range Range Spectral Spectral Resolution Resolution Temporal Temporal Resolution Resolution Spatial Resolution Spatial Resolution BaySpec 600-1000 nm 10 nm 8 fps 256*256 SoC 270-550 nm 18 nm 60 fps 320*256 PMIS 400-1000 nm 6 nm 15 fps 1024*1024 (1M) Light Gene PMIS

slide-62
SLIDE 62

Acknowledgements

  • NSF China
  • Prof. Qionghai Dai, Dr. Steve Lin, Dr. Xin
  • Dr. Yue Tao, Dr. Chenguang Ma

– Assistance in experimentation

  • Moshe Ben-Ezra, Yanxiang Lan

– Helpful discussions on implementation issues

slide-63
SLIDE 63

Welcome to visit CITE Lab @ Nanjing Univ.

Computational Imaging Technology & Engineering

http: tp://cite //cite.nju ju.ed .edu.cn u.cn

slide-64
SLIDE 64

The Optical Path

slide-65
SLIDE 65

Spectra of Illuminations