Xun Cao ( ) School of Electronic Science & Engineering Nanjing - PowerPoint PPT Presentation

High Resolution Spectral Video Capture & Computational Photography Xun Cao ( 曹汛 ) School of Electronic Science & Engineering Nanjing University caoxun@nju.edu.cn Dec 30th, 2015

Computational Photography Computational Photography Computational photography refers broadly to computational imaging techniques that enhance or extend the capabilities of digital photography. The output of these techniques is an ordinary photograph, but one that could not have been taken by a traditional camera. (Wikipedia) Computational photography is an emerging new field created by the convergence of computer graphics, computer vision and photography. Its role is to overcome the limitations of the traditional camera by using computational techniques to produce a richer, more vivid, perhaps more perceptually meaningful representation of our visual world. (CMU Course Introduction) 2

CP for various imaging dimensions Spectral （ Color ） Depth & View （ 3D ） Spatial Hyperspectral Light Field Gigapixel Temporal Multiview Multispectral Stereo RGB Ps.Fs UHD 3840*2160 120Hz Gray 2D 60Hz scale HD 1920*1080 30Hz Dynamic 10Hz SD<720p Range 24 bit 10/12 bit 8 bit

Computational Imaging Technology & Engineering The lab focuses on 3 kinds of computational cameras • Spectral Camera – High Resolution Spectral Video Camera: PMIS • Multiview Stereo Camera Array – High Accuracy 3D Reconstruction • Super-Resolution Camera – Nano-Scale Pixel Camera – Gigapixel on Single Chip

Grayscale Imaging light source K I  ( )       K R ( ) ( ) S d S  ( )  R  ( ) scene sensor 

Color Imaging light source （ ） R G B , , I  ( )       R R ( ) S ( ) d R      G R ( ) S ( ) d G      B R ( ) S ( ) d B S  ( ) R   ( ) scene sensor 

Spectral Imaging light source R  ( ) I  ( )   ？ ？ R  ( ) scene Imaging system 

Related Work (1)  Filtered Camera based Spectrum Imaging [Kidono07] [Gat00][Yamaguchi06][Schechner02 ]… Filter wheel Color Filter Array Programmable Filter Filter Scanning Key idea: Trade time for spectrum Shortcomings:  Incapable of capturing dynamic scenes  Low spectrum resolution Prof. S.Nayar Spatially varying filter Columbia PAMI ’02

Related Work (2)  Coded Aperture Snapshot Spectral Imager ( CASSI ) Key Idea: Coded Aperture [Brady’06] [Willett’07] [Gehm’07] [Wagadarikar’08] 2D Imaging + reconstruction Spectrum resolution: 6 nm Spatial resolution: 256 x 248 Limited spatial resolution Limited accuracy Time-consuming reconstructing (20min / frame) Prof. D.Brady (Duke Univ.) CASSI: Applied Optics SPIE , JOSA’06 -09

Related Work (3)  Computed Tomography Imaging Spectrometer [Descour95] [Descour01] [Vandervlugt07 ] [Hagen08]… Key Ideas: CT Projections + Reconstruction Shortcomings: Low Resolution Difficult to Calibrate Computed Tomographic Prof. E. Dereniak CTIS High Computational Cost Arizona Imaging Spectrometer different linear “projections” of the spectral data cube E. Dereniak Applied Optics SPIE, JOSA’95 -08 [JOSA’08 ]

Our Spectral Video Camera - PMIS 2008~2010: Prism-Mask Imaging Spectrometer (PMIS 1 ) – Directly capture multispectral video – High spectra-resolution – Low cost – Easy setup and calibration 2011~2014: Hybrid-Camera PMIS 2 – Both high spectral and spatial resolution – Real-time hyperspectral video capture 2014~now Scene-Adaptive PMIS 3 – Space-time coded modulation – Spectral video capture with improved accuracy and efficiency

A glance at PMIS 1 Pointgrey grayscale camera 2248x2048 @15fps mask capturing system

System Principle Grayscale Camera Occlusion Mask Prism GIF source: Wiki Camera System Re-generated RGB Video

A Typical Camera lens sensor array camera

Camera & Prism Spectra Overlap! lens sensor array prism camera

Camera & Mask lens sensor array mask camera

Camera & Mask & Prism Spectra Overlap! lens sensor array mask camera

Spectral Resolution W S ( ) a  R  spec CCD cell size grayscale camera f        e  W S ( ) s mask prism aperture image plane     sin ( ) sin ( ( ))             n  W S ( ) f (tan( ( ) a ) tan( ( ) a ))      e s sin ( ( )) sin ( ( ))

Spectral Resolution • Tradeoff Spatial/Spectral Resolution f mask prism aperture image plane           W S ( ) f (tan( ( ) a ) tan( ( ) a )) e s

Spectral Resolution • Tradeoff Spatial/Spectral Resolution f mask prism aperture image plane   W S ( )          W S ( ) f (tan( ( ) a ) tan( ( ) a )) R  spec e s

Spectral Resolution • Tradeoff Spatial/Spectral Resolution f mask prism aperture image plane   W S ( )          W S ( ) f (tan( ( ) a ) tan( ( ) a )) R  spec e s

Spectral Resolution • Tradeoff Spatial/Spectral Resolution f mask prism aperture image plane   W S ( )          W S ( ) f (tan( ( ) a ) tan( ( ) a )) R  spec e s

Spatial Resolution • Small Mask Hole Distance Spectra Overlap! mask prism aperture image plane

Spatial Resolution • Large Mask Hole Distance Unused Pixels mask prism aperture image plane

Spatial Resolution In practice, we can use a uniform mask  Design Mask Hole Distance d  Perfect D Alignment mask prism aperture image plane           D d (tan( ( )) tan( ( ))) e s

Device Calibration

Calibration Overview Mapping Position to Wavelength Spectrum Calibration Geometry Calibration Geometry Distortion caused by the prism Radiance (Smile Distortion) Non-constant Calibration CCDSensitivity

Spectrum Calibration Spectrum Calibration Geometry Calibration Radiance Calibration

Spectrum Calibration captured spectra Ground truth fluorescent spectra

Spectrum Calibration target spectra Warp Ground truth fluorescent spectra

Spectrum Calibration • Mapping Function : Wavelength <-> Position a  f x  aperture mask prism image plane    sin      Non linear , but   x ( ) f tan a arcsin( n sin( arcsin( )))    n smooth curve ! 

Geometry Calibration Spectrum Calibration Geometry Calibration Radiance Calibration

Geometry Calibration Predefined mask pattern captured image geometry calibrated image

Radiance Calibration Spectrum captured radiance genuine radiance Calibration Geometry Calibration Radiance Calibration

Radiance Calibration captured radiance genuine radiance   assuming c ( ), ( ) l locally constant   z ( )     b I z ( ) g ( c ( ) ( ) l d )  z ( ) a sensitivity intensity  1 ( ( )) g I z   l ( ) light input     wavelength c ( )( ) b a

Application 1: Human Skin Detection • The ‘W’ pattern in human skin reflectance • [Angelopoulou01]

Application 1: Human Skin Detection

Application 2: Material Discrimination RGB Image IR Image Our measurement The differences in IR

PMIS 1 Conclusions • Compared to Traditional Spectrometers • Passive Multispectral Video Capture • High spectral resolution • Tradeoff spectral and spatial resolution • Easy setup and calibration • Applications • Skin Detection • Material Recognition • Illumination Identification PMIS 1 ： A Prism-Mask System for Multispectral Video Acquisition, IEEE Intl’ Conf. Computer Vision (ICCV), 2009 , Oral IEEE Trans. Pattern Anal. Mach. Intell. (PAMI), 2011 High Resolution Multispectral Image Capture ， US Patent.20140085502

PMIS 1 : limitations • Light throughput is limited by – occlusion mask – relatively small aperture • Can NOT achieve both high spatial and spectral resolution – Limited CCD resolution – Spatial resolution (1000 pixels)

Low-Spatial High-Spectral Gray Camera Scene or Object Resolution Video Occlusion Mask Prism High-Spatial RGB Camera Low-Spectral PMIS 2 : Hybrid Camera System Resolution Video

PMIS 2 : System Pipeline Propagation

PMIS 2 : System Implementation

Propagation Algorithm Gray Camera RGB Camera High Spatial Low Spatial Resolution Resolution RGB Video Multispectral Video

Propagation Algorithm   RGB xy c G ( d ) G ( d )  ms     k k k k k r s  ms ij RGB xy G ( d ) G ( d )    c R G B , ,  k k k r s   e.g. for red channel R / R , k ij k

Propagation Algorithm Ground Truth Data Evaluation (11 datasets, .aix, .mat) - Spectral Database, University of Joensuu Color Group

Propagation Algorithm Temporal Enhancement Error 7.8% 4.3%

PMIS 2 : Results & Applications