HDR images acquisition dr. Francesco Banterle - PowerPoint PPT Presentation

HDR images acquisition dr. Francesco Banterle francesco.banterle@isti.cnr.it

Current sensors • No sensors available to consumer for capturing HDR content in a single shot • Some native HDR sensors exist, HDRc by Omron, but some issues: • too much noise • low resolution (around 1024x768) • expensive to manufacture

Exposure bracketing • Capturing many LDR images (8-bit) of the same scene: • from the darkest area in the scene • to the brightest area in the scene • The scene has to be static!!!

Exposure bracketing t = 1/128s t = 1/32s t = 1/8s

Exposure bracketing • Required equipment: • camera with the possibility to vary the exposure • tripod (avoid camera shake) • Optional equipment: • luminance meter • colorchecker chart • remote control for the camera

How many exposures? • Brute force approach: • Select an exposure for the darkest/brightest area in the scene and take a shot • Double/half exposure and take a shot • Repeat until brightest/darkest are in the scene is captured

How many exposures? • Some issue with this approach: • time consuming, especially if the camera is not programmable • we are making micro movements at every click! • over-sampling, maybe there is no need

Exposure Metering • Exposure metering [Gallo et al. 2012]: • capturing histograms from the viewfinder (picture preview in a camera) - free! P n i =0 H ( i ) • computing CDF for each histogram F ( n ) = P N i =0 H ( i ) • obtaining the global CDF • differentiation —> HDR histogram

Exposure Metering: LDR Histograms 0.04 0.06 0.08 0.09 0.08 0.035 0.07 0.05 0.07 0.03 0.06 0.04 0.06 0.025 0.05 0.05 0.02 0.03 0.04 0.04 0.015 0.03 0.02 0.03 0.01 0.02 0.02 0.01 0.005 0.01 0.01 0 0 0 0 − 11 − 10 − 9 − 8 − 7 − 6 − 5 − 4 − 3 − 2 − 12 − 11 − 10 − 9 − 8 − 7 − 6 − 5 − 4 − 3 − 13 − 12 − 11 − 10 − 9 − 8 − 7 − 6 − 5 − 4 − 13 − 12 − 11 − 10 − 9 − 8 − 7 − 6 − 5 1 0.5 0.35 0.08 0.9 0.45 0.07 0.3 0.8 0.4 0.06 0.25 0.7 0.35 0.05 0.6 0.3 0.2 0.5 0.25 0.04 0.15 0.4 0.2 0.03 0.3 0.15 0.1 0.02 0.2 0.1 0.05 0.01 0.1 0.05 0 0 0 0 − 2 − 1 0 1 2 3 4 5 6 − 8 − 7 − 6 − 5 − 4 − 3 − 2 − 1 0 1 − 9 − 8 − 7 − 6 − 5 − 4 − 3 − 2 − 1 0 − 10 − 9 − 8 − 7 − 6 − 5 − 4 − 3 − 2 − 1 0.9 1 0.8 0.9 0.9 0.8 0.8 0.7 0.8 0.7 0.7 0.6 0.7 0.6 0.6 0.5 0.6 0.5 0.5 0.5 0.4 0.4 0.4 0.4 0.3 0.3 0.3 0.3 0.2 0.2 0.2 0.2 0.1 0.1 0.1 0.1 0 0 0 0 − 4 − 3 − 2 − 1 0 1 2 3 4 − 3 − 2 − 1 0 1 2 3 4 5 − 6 − 5 − 4 − 3 − 2 − 1 0 1 2 3 − 5 − 4 − 3 − 2 − 1 0 1 2 3 4

Exposure Metering: LDR CDFs 1 0.9 0.8 0.7 0.6 0.5 F 0.4 0.3 0.2 0.1 0 − 14 − 12 − 10 − 8 − 6 − 4 − 2 0 2 4 6 log 2 Irradiance

Exposure Metering Algorithm 2.1: C OMP F ULL CDF( { B k } , { b ˜ E i , j } , F L 1 , F L 2 ,..., F L J ) for k ← 0 to K − 1 do F H ( B k ) ← 0 for j ← 0 to J − 1 8 for i ← 0 to I − 2 > < ( for each k : B k ∈ ( b ˜ i , j , b ˜ E E i + 1 , j ] do do j ( b ˜ do F H ( B k ) ← max ( F H ( B k ) , F L E > i , j )) : return ( F H )

Exposure Metering: HDR CDF 1 0.9 0.8 0.7 0.6 0.5 F 0.4 0.3 0.2 0.1 0 − 14 − 12 − 10 − 8 − 6 − 4 − 2 0 2 4 6 log 2 Irradiance

Exposure Metering: HDR Histogram 0.03 0.025 0.02 0.015 P 0.01 0.005 0 − 14 − 12 − 10 − 8 − 6 − 4 − 2 0 2 4 6 log 2 Irradiance

Exposure Metering • Selection of exposure times based on: • HDR histogram • Noise model of the camera

Exposure Metering: sampling 0.03 0.025 0.02 0.015 P 0.01 0.005 0 − 14 − 12 − 10 − 8 − 6 − 4 − 2 0 2 4 6 log 2 Irradiance

Exposure Metering: sampling 0.03 0.025 0.02 0.015 P 0.01 0.005 exp 0 − 14 − 12 − 10 − 8 − 6 − 4 − 2 0 2 4 6 log 2 Irradiance

Exposure Metering: sampling 0.03 0.025 0.02 0.015 P 0.01 0.005 exp exp 0 − 14 − 12 − 10 − 8 − 6 − 4 − 2 0 2 4 6 log 2 Irradiance

Exposure Metering: sampling 0.03 0.025 0.02 0.015 P 0.01 0.005 exp exp exp 0 − 14 − 12 − 10 − 8 − 6 − 4 − 2 0 2 4 6 log 2 Irradiance

Exposure Metering: sampling 0.03 0.025 0.02 0.015 P 0.01 0.005 exp exp exp exp 0 − 14 − 12 − 10 − 8 − 6 − 4 − 2 0 2 4 6 log 2 Irradiance

Exposure Metering: sampling 0.03 0.025 0.02 0.015 P 0.01 0.005 exp exp exp exp exp 0 − 14 − 12 − 10 − 8 − 6 − 4 − 2 0 2 4 6 log 2 Irradiance

Linear Images • What is a linear value? I = a E • where: • I the value recorded by the sensor • E is the radiance of the real value • a is a constant

Linear Images • High-end or prosumer camera can save RAW: • advantage: storing linear values • disadvantage: a lot of memory; no compression and 12-14bit per color channel

meanwhile in the real-word…

Linear Images • Consumer cameras, smartphones, tables save typically JPEG at high quality (in the best case): • advantage: images are stored in little memory • disadvantage: • no linear values • images are stored applying an unknown function, f, called Camera Response Function (CRF)

Linear Images: example 250 200 Pixel value 150 identity γ = 2.2 100 50 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Relative camera response

Linear Images: example 250 200 Pixel value 150 red channel green channel blue channel 100 50 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Relative camera response

Estimating CRF • What can we do? • We can estimate the CRF or perform a radiometric calibration • What can we do? • Taking a photograph with colorchecker and controlled environment • Taking photographs at different exposure times

Estimating CRF

Estimating CRF function to minimize smoothing term

Estimating CRF • To minimize the objective function, a dense linear system needs to be solved using SVD: • (N exposures x N samples + D + 1) x (N samples + D + 1) • where D = 256 (discretization levels) • We cannot use all pixels in the image: • too large system

Estimating CRF • To minimize the objective function, a dense linear system needs to be solved using SVD • We cannot use all pixels in the image: • too large system

Estimating CRF • To minimize the objective function, a dense linear system needs to be solved using SVD • We cannot use all pixels in the image: • too large system

Estimating CRF: samples selection • Idea1 : sampling in spatial domain

Estimating CRF: samples selection • For each spatial sample ( i, j ) • Collect values at each exposure, , to obtain a Z i sample vector: [ Z 0 ( i, j ) , ..., Z n ( i, j )]

Estimating CRF: samples selection • Idea2 : in histogram domain, to randomly subsample the image 1 0.8 0.6 F 0.4 Exposure 1 0.2 Exposure 2 Exposure 3 0 0 50 100 150 200 250 Pixel value

Estimating CRF: samples selection • Idea2 : sampling in histogram domain 1 0.8 0.6 F 0.4 Exposure 1 0.2 Exposure 2 Exposure 3 0 0 50 100 150 200 250 Pixel value

Estimating CRF: samples selection • Idea2 : sampling in histogram domain 1 0.8 0.6 F 0.4 Exposure 1 0.2 Exposure 2 Exposure 3 0 0 50 100 150 200 250 Pixel value

Estimating CRF: samples selection • Idea2 : sampling in histogram domain 1 0.8 0.6 F 0.4 Exposure 1 0.2 Exposure 2 Exposure 3 0 0 50 100 150 200 250 Pixel value

Estimating CRF: samples selection • Idea2 : sampling in histogram domain 1 0.8 0.6 F 0.4 Exposure 1 0.2 Exposure 2 Exposure 3 0 0 50 100 150 200 250 Pixel value

Estimating CRF: samples selection • Idea2 : sampling in histogram domain 1 0.8 0.6 F 0.4 Exposure 1 0.2 Exposure 2 Exposure 3 0 0 50 100 150 200 250 Pixel value

Estimating CRF: samples selection • Idea2 : sampling in histogram domain 1 0.8 0.6 F 0.4 Exposure 1 0.2 Exposure 2 Exposure 3 0 0 50 100 150 200 250 Pixel value

Estimating CRF: samples selection • Idea2 : sampling in histogram domain 1 0.8 0.6 F 0.4 Exposure 1 0.2 Exposure 2 Exposure 3 0 0 50 100 150 200 250 Pixel value

Estimating CRF: samples selection • Idea2 : sampling in histogram domain 1 0.8 0.6 F 0.4 Exposure 1 0.2 Exposure 2 Exposure 3 0 0 50 100 150 200 250 Pixel value

Estimating CRF: weighting function • weighting function: • to avoid outliers during the estimate • shapes: tent, box with cut-off, Gaussian, etc. • outliers: • over-exposed pixels • under-exposed pixels

Estimating CRF: samples selection 1 0.8 0.6 w(x) 0.4 0.2 0 0 50 100 150 200 250 Pixel value

Estimating CRF: samples selection w ( x ) = 1 − (2 x − 1) 12 1 0.8 0.6 w(x) 0.4 0.2 0 0 50 100 150 200 250 Pixel value

Estimating CRF: samples selection − 4 ( x − 0 . 5)2 w ( x ) = e 2(0 . 5)2 1 0.9 0.8 0.7 0.6 w(x) 0.5 0.4 0.3 0.2 0.1 0 50 100 150 200 250 Pixel value