Edge-Adaptive Image Interpolation with Contour Stencils Pascal - - PowerPoint PPT Presentation

Edge-Adaptive Image Interpolation with Contour Stencils

Pascal Getreuer Dec 27, 2010

TV along Curves

Let u be an image. For C a smooth simple curve, define uTV(C) = T

• ∂

∂t u

• γ(t)
• dt,

γ : [0, T] → C. Strategy: Find approximate con- tours of u by finding curves C such that uTV(C) is small.

γ′(t)

C

Contour Stencils

A contour stencil is a function S : Z2 × Z2 → R describing edges between pixels, and TV is estimated as (S ⋆ [u])(k) :=

• m,n∈Z2

S(m, n) |uk+m − uk+n| ≈ uTV(C) where S describes edges that approximate C.

+n1 +n2

(0, 0)

(S ⋆ [u])(i, j) =

• |ui,j−1−ui+1,j|

+ |ui−1,j−1 − ui,j| + |ui,j − ui+1,j+1| + |ui−1,j−ui,j+1|

• .

Contour Stencils

Estimate the contours locally by finding a stencil with low TV, S⋆(k) = arg min

S∈Σ(S ⋆ [u])(k)

where Σ is a set of candidate stencils.

Σ =

Contour Stencils

Input Contour Stencils Sobel

For each pixel, S⋆(k) is determined to estimate the local contour orientation.

Why TV?

Total variation is invariant under diffeomorphisms on space. Consider the change of variables t = ϕ(s) dt = ϕ′(s) ds

u(t) u

• ϕ(s)
• and suppose that ϕ′(s) > 0, then

T |u′(t)| dt = ϕ−1(T)

ϕ−1(0)

• u′

ϕ(s)

• ϕ′(s) ds

= ϕ−1(T)

ϕ−1(0)

• ∂

∂s u

• ϕ(s)
• ds.

Interpolation Problem

Given discrete image v and point spread function h(x, y), find function u(x, y) such that vi,j = (h ∗ u)(i, j) for all i, j.

Input v Interpolation u

Edge Directed Interpolation

Theorem (Edge Directed Interpolation)

Consider approximating u(x) by ˆ u(x) = (1 − λ)u(a) + λu(b) a b x

C1 C2

and let C = C1 ∪ C2 be a curve passing through a, x, and b. Then the approximation error is bounded by |ˆ u(x) − u(x)| ≤ max

• |1 − λ| , |λ|
• uTV(C) .

Choosing the stencil with the smallest TV minimizes the estimated interpolation error: |ˆ u(x) − u(x)| ≤ uTV(C) ≈

1 |S⋆|(S⋆⋆[u])(k) = min S∈Σ 1 S(S⋆[u])(k).

Contour Stencil Windowed Zooming

Local reconstructions: uk(x) = vk +

• n∈N

cn ϕn

S⋆(k)(x − n),

θ

k + N

k if S =

k + N

k if S = where vk : kth pixel of input image N : neighborhood ϕn

S⋆(k) : function oriented with the

best-fitting stencil S⋆(k) cn : coefficients such that (h ∗ uk)(m) = vk+m for m ∈ N

Contour Stencil Windowed Zooming

Combine local reconstructions with overlapping windows u(x) =

• k∈Z2

w(x − k)uk(x − k), where w satisfies

• k w(x − k) ≡ 1

s.t. method reproduces constants

w(k) = 0 for k ∈ N

s.t. ↓(h ∗ u) ≈ v

w compact support

for computational efficiency

Example: Cubic B-spline w(x) = B(x1)B(x2) B(t) =

• 1 − |t| + 1

6 |t|3 − 1 3

• 1 − |t|
• 3+

Original Image (332×300) Input Image (83×75) Estimated Orientations Proposed Interpolation (PSNR 25.97, 0.125 s)

Cubic B-spline (PSNR 25.92, 0.011 s) Fourier (PSNR 25.34, 0.062 s) TV Minimization (PSNR 25.73, 0.784 s) Proposed Interpolation (PSNR 25.97, 0.125 s)

AQua-2 (PSNR 24.72, 0.016 s) Fractal Zooming (PSNR 24.65) Roussos (PSNR 25.87, 2.518 s) Proposed Interpolation (PSNR 25.97, 0.125 s)

Zooming Comparison

Average PSNR on the Kodak Image Suite

Zoom Factor 2× 3× 4× AQua-2 25.06 22.48 21.35 Fractal Zooming 29.00 27.20 25.25 Contour Stencils 29.87 27.77 25.93 Roussos 30.56 27.97 26.19

Computation Time (s) vs. Output Image Size

Image Size 128 × 128 256 × 256 512 × 512 AQua-2 0.0048 0.017 0.068 Contour Stencils 0.025 0.088 0.34 Roussos 0.23 2.22 8.64

Analysis of Contour Stencils

C ˜ C

˜ C ≈ C = ⇒ uTV(˜

C) ≈ uTV(C)

Curve Perturbation

Let C and ˜ C be smooth curves parameterized by γ : [0, T] → C and ˜ γ : [0, T] → ˜

• C. Then if u is twice

continuously differentiable,

• uTV(˜

C) − uTV(C)

• ≤
• |∇u|
• ∞
• |˜

γ′ − γ′|

• 1

+ ∇2u∞ |˜

C|+|C| 2

+ 1

4

• |˜

γ′ − γ′|

• 1
• |˜

γ − γ|

• ∞.

Analysis of Contour Stencils

u ∈ C 2 = ⇒ discrete TV is first-order accurate

TV Discretization

Suppose u ∈ C 2[0, T] and 0 = t0 < t1 < · · · < tN = T, and define hi = ti − ti−1. Then uTV −1

3T h2

max

havg u′′∞ ≤ N

• i=1

|u(ti) − u(ti−1)| ≤ uTV .

Analysis of Contour Stencils

Let S⋆2(k) denote the second best-fitting stencil and define the separation sepv(k) between the first and second best, S⋆2(k) := arg min

S∈Σ\S⋆(k)

(S ⋆ [v])(k), sepv(k) :=

• S⋆2(k) ⋆ [v]
• (k) −
• S⋆(k) ⋆ [v]
• (k).

Stability of the Best-Fitting Stencil

Suppose that for two images v and ˜ v sepv(k) > 2M v − ˜ v2 , M = max

S∈Σ

• m∈Z
• n∈Z
• S(m, n) + S(n, m)
• 21/2

. Then they have the same best-fitting stencil at k.

Contour Stencil Design

Let {f 1, . . . , f J}, f j : Z2 → R, be a set of image features. Example: f j(x) = x1 sin π

8j − x2 cos π 8j,

j = 0, . . . , 7 We want to design stencils S1, . . . , SJ that distinguish between these features.

Contour Stencil Design

We want stencil Sj to be the best-fitting stencil on f j, Want:

1 |Sj|(Sj ⋆ [f j])(0) < 1 |Si|(Si ⋆ [f j])(0)

for all i = j. Ignoring the

1 |S| normalizations, this condition becomes

(Sj ⋆ [f j])(0) < (Si ⋆ [f j])(0) = ⇒

• (Sj − Si) ⋆ [f j]
• (0) < 0.

We can try to satisfy this condition by minimizing min

S1,...,SJ J

• i=1

J

• j=1
• (Sj − Si) ⋆ [f j]
• (0) + γ

J

• j=1

Sj1 s.t. 0 ≤ Sj(m, n) ≤ 1

Contour Stencil Design

min

S1,...,SJ J

• i=1

J

• j=1
• (Sj − Si) ⋆ [f j]
• (0) + γ

J

• j=1

Sj1 s.t. 0 ≤ Sj(m, n) ≤ 1 The minimization has closed-form solution Sj(m, n) =

• 1

if |f j

m − f j n| < 1 J

J

i=1|f i m − f i n| − γ J ,

if |f j

m − f j n| > 1 J

J

i=1|f i m − f i n| − γ J .

Corner-Shaped Stencils

Example: Corner-shaped stencils designed from the features f j(x) = max{x1 cos π

4j − x2 sin π 4j,

x1 sin π

4j + x2 cos π 4j}.

3D Stencils

In d dimensions, a stencil S : Zd × Zd → R is applied at voxel k ∈ Zd as (S ⋆ [u])(k) :=

• m,n∈Zd

S(m, n) |uk+m − uk+n| . Small TV detects isosurfaces. Some 3D stencils:

3D Stencils

Example: Stencils applied to ui,j,k =

• i2 + j2 + k2

The results are visualized by assigning a color to the region of space having a particular best-fitting stencil.

3D Stencils

Example: Stencils applied to an MRI brain volume

Demosaicing

Original Mosaiced Demosaiced Bayer Grid

Demosaicing Stencils

Centered on a green pixel:

Axially-oriented Diagonally-oriented

π 8 -oriented

Centered on a red or blue pixel:

Axially-oriented Diagonally-oriented

π 8 -oriented

Demosaicing Stencils

Stencil orientation estimation on mosaiced data:

Demosaicing Stencils

Stencil orientation estimation on mosaiced data:

Preliminary Demosaicing Method

Let f be the given mosaiced image. We consider demosaicing by the minimization of arg min

u

• k∈{Y ,Cb,Cr}
• n∈Ω
• m∈N(n)

wm,n

• u(k)

m − u(k) n

• + λ

2

• k∈{R,G,B}
• n∈Ω(k)

(fn − u(k)

n )2

where wm,n : weights choosen according the best-fitting stencils N(n) : neighbors of pixel n λ : fidelity parameter Ω(k) : subset of Ω where kth channel is given

Exact Demosaiced Exact Demosaiced Exact Demosaiced

Exact Bilinear Malvar et al. Hamilton-Adams Gunturk et al. Li Zhang-Wu Proposed

Exact Bilinear Malvar et al. Hamilton-Adams Gunturk et al. Li Zhang-Wu Proposed

Exact Bilinear Malvar et al. Hamilton-Adams Gunturk et al. Li Zhang-Wu Proposed

Thanks!

Webpage

http://www.math.ucla.edu/∼getreuer/contours

Contact

getreuer@gmail.com