Hyperprior bayesian approach for inverse problems in imaging. - - PowerPoint PPT Presentation
Hyperprior bayesian approach for inverse problems in imaging. Application to single shot HDR. Julie Delon with Cecilia Aguerrebere, Andr es Almansa, Yann Gousseau and Pablo Mus e 1 / 32 Teaser : what is High Dynamic Range Imaging (HDR) ?
Julie Delon
with Cecilia Aguerrebere, Andr´ es Almansa, Yann Gousseau and Pablo Mus´ e
1 / 32
Capture a scene containing a large range of intensity levels...
2 / 32
... using a standard digital camera. Limited dynamic range of the camera ! loss of details in bright and/or dark areas.
2 / 32
... using a standard digital camera. Limited dynamic range of the camera ! loss of details in bright and/or dark areas.
2 / 32
... using a standard digital camera. Limited dynamic range of the camera ! loss of details in bright and/or dark areas.
2 / 32
... using a standard digital camera. Limited dynamic range of the camera ! loss of details in bright and/or dark areas.
2 / 32
... using a standard digital camera. Limited dynamic range of the camera ! loss of details in bright and/or dark areas.
2 / 32
Limited dynamic range of the camera ! loss of details in bright and/or dark areas.
2 / 32
Limited dynamic range of the camera ! loss of details in bright and/or dark areas.
2 / 32
Usual approach for HDR image generation : fusion of mutiple exposures. HDR generation Irradiance Map
(number of photons reaching each pixel per unit time)
3 / 32
4 / 32
ghosting effect camera + object motion
5 / 32
6 / 32
6 / 32
Original image
7 / 32
Noise
7 / 32
Missing pixels
7 / 32
˜ u = Au + n u reference image A is a diagonal operator Additive noise n may depend on u:
Exemple RAW data (shot noise and readout noise) n(x) ∼ N(0, α(x)u(x) + β(x))
8 / 32
˜ u = Au + n
Zi = pi(˜ u) (degraded patch of size d = f ⇥ f centered at i) Ci = pi(u) (unknown reference patch) Ni = pi(n) (additive noise patch) Di restriction of A to pi(u)
Zi = DiCi + Ni
8 / 32
Observed patch
9 / 32
Observed patch
Assumptions : D is known N ⇠ N(0, ΣN), eventually depends on C but Cov(N, C) = 0
9 / 32
Patch we seek to estimate
Gaussian prior for patches
Observed patch
Assumptions : D is known N ⇠ N(0, ΣN), eventually depends on C but Cov(N, C) = 0 C ⇠ N(µ, Σ) with µ and Σ unknown
9 / 32
1
Classical choice : MLE [NL-Bayes - Lebrun et al. 2013] Set of similar patches Z1, . . . , ZM, such that all the (unknown) Ci follow the same law N(µ, Σ). b µ = 1 M
M
X
i=1
Zi and b Σ = 1 M 1
M
X
i=1
[Zi b µ][Zi b µ]T
10 / 32
1
Classical choice : MLE [NL-Bayes - Lebrun et al. 2013] Set of similar patches Z1, . . . , ZM, such that all the (unknown) Ci follow the same law N(µ, Σ). b µ = 1 M
M
X
i=1
Zi and b Σ = 1 M 1
M
X
i=1
[Zi b µ][Zi b µ]T
10 / 32
1
Classical choice : MLE [NL-Bayes - Lebrun et al. 2013] Set of similar patches Z1, . . . , ZM, such that all the (unknown) Ci follow the same law N(µ, Σ). b µ = 1 M
M
X
i=1
Zi and b Σ = 1 M 1
M
X
i=1
[Zi b µ][Zi b µ]T
2
Gaussian Mixture Model prior on patches [EPLL - Zoran and Weiss 2011 ; PLE - Yu et al. 2012]
10 / 32
MAP with an hyperprior on (µ, Σ) argmax
{Ci},µ,Σ
p({Ci}i, µ, Σ | {Zi}i) = argmax
{Ci},µ,Σ
p({Zi} | {Ci}, µ, Σ) . p({Ci} | µ, Σ) . p(µ, Σ). Rappel Zi | Ci, µi, Σi ⇠ N(DiCi, ΣNi) Ci | µi, Σi ⇠ N(µ, Σ) (µ, Σ) ? Inclusion of hyperprior information compensates for missing pixels.
11 / 32
Conjugate prior for a multivariate normal distribution Normal prior on the mean (conditionnal on the covariance) µ | Σ ⇠ N(µ0, Σ/κ) / |Σ|− 1
2 exp
⇣ κ 2 (µ µ0)TΣ−1(µ µ0) ⌘ inverse Wishart prior on the covariance matrix Σ ⇠ IW(νΣ0, ν) / |Σ|− ν+d+1
2
exp ✓ 1 2trace[νΣ0Σ−1] ◆
12 / 32
Conjugate prior for a multivariate normal distribution Normal prior on the mean (conditionnal on the covariance) µ | Λ ⇠ N(µ0, Λ−1/κ) / |Λ|
1 2 exp
⇣ κ 2 (µ µ0)TΛ(µ µ0) ⌘ Wishart prior on the inverse covariance matrix Λ ⇠ W((νΣ0)−1, ν) / |Λ|
ν−d−1 2
exp ✓ 1 2trace[νΣ0Λ] ◆
13 / 32
{Zi} set of similar patches; µ, Λ fixed. argmax
{Ci}
p({Zi} | {Ci}) . p({Ci} | µ, Λ) . p(µ, Λ) = argmax
{Ci}
ΠM
i=1 (p(Zi | Ci) . p(Ci | µ, Λ))
= argmax
{Ci}
ΠM
i=1
⇣ g0,ΣNi (Zi DiCi) . gµ,Λ−1(Ci) ⌘ .
b Ci = Λ−1DT
i
| {z }
E(CiZ T
i )
(DiΛ−1DT
i + ΣNi
| {z }
E(ZiZ T
i )
)−1 | {z }
Wi
(Zi Diµ) + µ
14 / 32
{Ci}i fixed. argmax
µ,Λ
p({Zi} | {Ci}, µ, Λ) | {z }
HYP : independent of µ,Λ
. p({Ci} | µ, Λ) . p(µ, Λ) ' argmax
µ,Λ
p({Ci} | µ, Λ) . p(µ, Λ) = argmax
µ,Λ
ΠM
i=1 gµ,Λ−1(Ci) gµ0,Λ−1/κ(µ) wΛ0/ν,ν(Λ).
( b µ = MC+κµ0
M+κ
b Λ−1 = νΣ0+κ(b
µ−µ0)(b µ−µ0)T +PM
i=1( b
Ci−b µ)( b Ci−b µ)T ν+M−d
15 / 32
In the previous formula, ˆ Ci, ˆ µ and ˆ Λ depend on each other. Replacing ˆ Ci by its expression in (µ, Λ) and reinjecting this in the formula of (ˆ µ, ˆ Λ), we get b µ = @κId +
M
X
j=1
WjDj 1 A
−1 0
@
M
X
j=1
WjZj + κµ0 1 A (ν + M d) b Λ−1 =
M
X
j=1
(Wj(Zj Djµ))(Wj(Zj Djµ))T + κ(µ µ0)(µ µ0)T + νΣ0 with Wj = Λ−1DT
j (DjΛ−1DT j + ΣNj)−1.
16 / 32
Initialization : compute Oracle image Coracle For k = 1 : maxit For each Patch Z
1
Find patches similar to Z in Coracle
2
Compute µ0 and Σ0 from this set of similar patches in Coracle
3
Compute first b µ, b Σ with a small loop and then ˆ C.
Restore image from restored patches and update Coracle = restored image.
17 / 32
for isotropic patterns
(K-1) edges with different orientations
K predefined models :
18 / 32
Synthetic data, 70% missing pixels.
19 / 32
Synthetic data, 70% missing pixels.
19 / 32
Synthetic data, 70% missing pixels, gaussian noise σ = 10.
19 / 32
Synthetic data, 70% missing pixels, gaussian noise σ = 10.
19 / 32
Zoom on Real data. Left to right : Input low-resolution image, HBE, PLE, bicubic.
19 / 32
20 / 32
Spatially Varying pixel Exposures (SVE) [Nayar and Mitsunaga, 2000]
21 / 32
21 / 32
X No need for image alignment. X No need for motion detection. X No ghosting problems. X No large saturated regions to fill. ⇥ Resolution loss : unknown pixels to be restored (over and under exposed pixels). ⇥ Noise. ⇥ Need to modify the standard camera.
I Alternative without camera modification [Hirakawa and Simon, 2011]. 22 / 32
23 / 32
Zi = DiCi + Ni Di is a diagonal operator
I Dii = 0
⇒ over- or under-exposed pixel (ignored)
I Dii = 1
⇒ well-exposed pixel (kept)
Ci irradiance at pixel i (reference image) Noise model for RAW data (shot noise and readout noise) Ni ⇠ N(0, ΣNi) with diagonal covariance matrix ΣNi such that (ΣNi)k = αkCk + βk, with α and β known.
24 / 32
25 / 32
Ground-truth HPNLB PLEV Sch¨
Nayar-Mitsun Input Differences to ground-truth PSNR: 33.1dB 29.7dB 30.4dB 29.4dB
26 / 32
Ground-truth HPNLB PLEV Sch¨
Nayar-Mitsun Input Differences to ground-truth PSNR: 35.1dB 34.0dB 30.0dB 28.5dB
27 / 32
28 / 32
28 / 32
28 / 32
28 / 32
28 / 32
29 / 32
29 / 32
29 / 32
HPNLB PLEV Mask
30 / 32
31 / 32
31 / 32
31 / 32
HPNLB PLEV Mask
32 / 32