Recovering Cam era Sensitivities using Target-based Reflectances - - PowerPoint PPT Presentation

Recovering Cam era Sensitivities using Target-based Reflectances Captured under m ultiple LED-I llum inations

Philipp Urban, Michael Desch, Kathrin Happel and Dieter Spiehl

Motivation: Spectral Estim ation

Can we calculate spectral reflectances from the camera image? 

Illumination Scene Camera Camera Response Spectral Image

?

If yes, we could render/ reproduce the image for any illuminant

Motivation: Spectral Estim ation

If yes, we could estimate pigments used by an artist

Y . Zhao 2008

Motivation: Spectral Estim ation

Camera Sensitivities Required

Spectral Estim ation Approaches

Training-based Methods Model-based Methods Spatio-Spectral Methods

c1  cm Signal Processing

Multipoint spectral Measurement Methods

Use information to calculate spectral reflectance factor from camera responses Additional Information



Model-based Methods

e.g. Distribution

• f noise,reflectances

Cam era Model ( Linear, Continuous)

Reflectance Illuminant Camera Sensitivities

X X

d

c1 =  = c cm l r [s1  sm]T



Camera Response (additive noise) Spectral stimulus

+

Required but usually unknown



Geometry factor* *Lambertian surface assumption

Diagonal matrix with illuminant as diagonal Reflectance vector i-th sensitivity vector i-th channel response Noise Geometry Factor

Cam era Model ( Linear, Discrete)

r = [ 50 65 85 80 60 58 55 50 45 40] T

Relative pow er\reflectance

Sensitivity Range Spectral representation by N-dimensional vector

Monochrom ator-based Training Stim uli

Monochromator Halon (r  1)

Wavelength Wavelength Sensitivity Sensitivity Wavelength Sensitivity

Sampling the Sensitivities with monochromatic light + Very high effective dimension* (~N) Very good sensitivity estimation without any additional assumptions *Hardeberg (2002)

Target-based Training Stim uli

• Measure m target reflectances
• Capture target for one illuminant

Low effective dimension of stimuli for typical targets (broadband) Additional assumptions required (smoothness, non-negativity,…) Large influence of assumptions

• n estimated sensitivities

LED+ Target-based Training Stim uli

• Measure m target reflectances
• Capture target for LED illuminant 1
• Capture target for LED illuminant k

…

High effective dimension of stimuli for typical targets (narrowband) Additional assumptions required (smoothness, non-negativity,…) Small influence of assumptions

• n estimated sensitivities

m x k x n equations for recovering s1,…,sn

Problem : Measurem ent Geom etry

 

45°/0° Different measurement geometries:

• Distance between camera and patches
• Angle between camera and patches
• Difference between spectrophotometer

and camera Geometry factor differs among patches and illuminants Non-linear equation system (sensitivities + geometry factors)

Solution: Use Chrom aticities

Divide each equation by the sum of channels (geometry factor can be reduced) Rearrange equations + consider noise is a (k · m · n) × (n · N) dimensional matrix that depends on Noise, assumed to be uncorrelated zero-mean Gaussian with covariance matrix where

W hat Assum ptions are Reasonable?

1

Sensitivities are non-negative

2

Sensitivities are likely to be smooth: Model wavelength-correlation using a Toeplitz-matrix

invalid Unlikely, but possible

3

Covariance matrix for can be modeled as block-diagonal (i.e. channel sensitivities are uncorrelated)

2 3

Likelihood model: Prior distribution: where

W hat Assum ptions are Reasonable?

Posterior distribution: where

2 3

Minimize: subject to:

Reconstruct Relative Sensitivities

1

Solve Constrained Maximum-A-Posteriori (MAP) Problem:

• The result are sensitivities that maximize the posterior distribution subject to

constraints

• Last constraint is a normalization – required to avoid zero sensitivities

Experim ental Setup

1

Camera: Sinar 54H-based RGB camera (IR cutoff removed) Custom made two-stage filter wheel with yellow + blue filter Six channel camera (12 bit, 22MP)

2

Target: Color Checker measured using X-Rite’s EyeOne

Experim ental Setup

3

JUST Normlicht LED Color Viewing Light (six different types of LED)

4

UV/IR cutoff filter added to limit the sensitivity range of the camera to 400 – 700 nm [EyeOne measurements do not cover the real sensitivity range] Spectroradiometer to measure LED spectral power distribution (SPD) LED viewing Booth: Konica-Minolta CS1000

5

Results

Experimental Setup Training Stimuli (24 CC* x 6 LED = 144) *CC = Color Checker patches Effective dimension: 15 Color Checker only: 9

Results

Reconstructed Sensitivities Predicted vs. real channel responses for the training stimuli [Errors: mean ~ 0.3%, 95th < 0.8%, max < 1.35%]* *Error percentage is relative to the max. channel response

Results ( Test Stim uli)

Capture the ESSER TE221 Target under 6 LED illuminant [1698 different test stimuli] *Error percentage is relative to the max. channel response Predict channel responses using the reconstructed sensitivities Prediction error histogram*

• Small mean errors < 0.5%*
• Small 95th errors < 1.3%*
• But large max errors ~7.5%*

Possible reason for large max error:

• Invalid Lambertian surface assumption
• Effective dimension of stimuli is still to small

Conclusion

Camera sensitivities are required for many reflectance estimation methods  If unknown they need to be measured/estimated Sensitivity reconstruction methods use assumptions (non-negativity, smoothness) and training stimuli: Different LED + Target  Higher effective dimension of stimuli  Smaller influence of assumptions Sensitivity reconstruction using Constrained Maximum-A-Posteriori (MAP) Test: 6-channel camera, 6-LED, Color Checker Results (Esser test target): small mean and 95th-, large max prediction errors

Thanks for your attention

Philipp Urban, Michael Desch, Kathrin Happel and Dieter Spiehl http: / / www.idd.tu-darmstadt.de/ color Thanks to Maria Fedutina, Tanja Kaulitz, Henri Kröling, Karsten Rettig, Manfred Jakobi for constructing, manufacturing and assembling the filter wheel and to the Deutsche Forschungsgemeinschaft for the sponsorship of the project