An introduction to mathematical models in image processing with a - - PowerPoint PPT Presentation

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system

An introduction to mathematical models in image processing with a focus on PDEs and optimization techniques

Yifei Lou Department of Mathematical Sciences, UTD October 10, 2014

1/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system

Outline

1

ABCs of image processing

2

Image and PDEs Forward/backward diffusion Imaging through turbulence

3

Image and optimization TV regularization Bregman iterations

4

Optimization and dynamic differential system Inverse scale space flow Forward-backward splitting

2/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system

Outline

1

ABCs of image processing

2

Image and PDEs Forward/backward diffusion Imaging through turbulence

3

Image and optimization TV regularization Bregman iterations

4

Optimization and dynamic differential system Inverse scale space flow Forward-backward splitting

3/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system

Image representation

Grayscale digital image 8 bits per sampled pixel = 256 different intensities Value 255 = white, 0 = black, and in-between is a shade of gray Resolution: number of detectors/CCD sensors

Figure: Decreasing resolution

4/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system

Some applications–denoising

5/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system

Some applications–deblurring

6/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system

Some applications–inpainting and segmentation

7/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system

1992 LA Riots

8/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system

Outline

1

ABCs of image processing

2

Image and PDEs Forward/backward diffusion Imaging through turbulence

3

Image and optimization TV regularization Bregman iterations

4

Optimization and dynamic differential system Inverse scale space flow Forward-backward splitting

9/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward/backward diffusion

Forward diffusion methods

Heat equation u(0) = u0, ut = △u, t > 0.

10/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward/backward diffusion

Forward diffusion methods

Heat equation u(0) = u0, ut = △u, t > 0.

10/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward/backward diffusion

Forward diffusion methods

Anisotropic diffusion (Rudin-Osher-Fatemi 1992) ut = ∇ · ∇u |∇u|

• Diffuse along edges ⇒ edge-preserving

10/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward/backward diffusion

Backward diffusion methods

Backward heat equation is unstable ut = −△u.

11/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward/backward diffusion

Backward diffusion methods

Backward heat equation is unstable ut = −△u. A balanced diffusion-sharpening diffusion by Alvarez and Mazorra in 1994 ut = uξξ − sign(uηη)|∇u|, for the gradient direction η, and normal direction ξ.

11/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward/backward diffusion

Sobolev gradient flow

In 2009, Calder-Mansouri-Yezzi propose Sobolev gradient diffusion.

12/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward/backward diffusion

Sobolev gradient flow

In 2009, Calder-Mansouri-Yezzi propose Sobolev gradient diffusion. The heat equation is the gradient descent on the functional E(u) = 1 2

• Ω

∇u2 , with respect to the L2 metric.

12/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward/backward diffusion

Sobolev gradient flow

They consider an inner product on the Sobolev space H1

0(Ω)

v, w − → gλ(v, w) = (1 − λ)v, wL2 + λv, wH1 , for any λ > 0. The gradient of E(u) w.r.t. gλ is given by ∇gλE|u = −△(Id − λ△)−1u , The PDE ut = ±∇gλE|u are stable for both forward and backward directions.

13/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward/backward diffusion

Sobolev gradient flow

The backward direction can be used for image sharpening. They propose the following energy Es(u) = 1 4

• Ω

∇u02

Ω ∇u2

• Ω ∇u02 − α

2 , where u0 is the initial value and α is a scale. The gradient descent to minimize such energy is ut =

Ω ∇u2

• Ω ∇u02 − α
• △(Id − △)−1u .

14/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward/backward diffusion

(a) input (b) ROF (c) AM (d) SOB

15/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Imaging through turbulence

Imaging through turbulence1

Two effects of turbulence blurry image frames temporal oscillations

1Lou-Kang-Soatto-Bertozzi, 2013 16/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Imaging through turbulence

Imaging through turbulence1

Two effects of turbulence ⇒ our approach blurry image frames ⇒ sharpen individual frame temporal oscillations ⇒ stabilize temporal direction

1Lou-Kang-Soatto-Bertozzi, 2013 16/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Imaging through turbulence

Imaging through turbulence1

Two effects of turbulence ⇒ our approach blurry image frames ⇒ sharpen individual frame temporal oscillations ⇒ stabilize temporal direction Sharpen individual frame (backward diffusion): shock filter/Sobolev gradient flow. Stabilize temporal direction (forward diffusion): linear/anisotropic diffusion.

1Lou-Kang-Soatto-Bertozzi, 2013 16/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Imaging through turbulence

Our stabilization model

Suppose uk(x, y) = u(x, y, k), u0 be the original video

• sequence. Then our model is

un+1

k

− un

k

dt =

• ∇uk2

∇u0

k2 − α

• △(Id − λ△)−1un

k

+µ

• un

k+1 + un k−1 − 2un+1 k

• ,

λ is a parameter in defining the Sobolev gradient. α > 1 controls spatial sharpness. µ balances between the spatial sharpening and the temporal smoothing.

17/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Imaging through turbulence

Analysis

We prove the local and global existence and uniqueness of the solution. The frequency approach yields an efficient algorithm. This is a weakly conditionally stable method. The stability condition only depends on dt, not on spatial grid resolution.

18/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Imaging through turbulence

Figure: Raw data.

19/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Imaging through turbulence

Figure: SOB.

20/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Imaging through turbulence

Figure: SOB+LAP .

21/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Imaging through turbulence

Figure: SOB+LAP .

22/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Imaging through turbulence

Turbulence motion analysis

(a) (b)

Figure: The positioning of the key points along a line. The key points are displayed as the blue dots on (a). (b) shows how these points are

• scillating as time t changes. The wave movement of the turbulence

happens in groups.

23/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system

Outline

1

ABCs of image processing

2

Image and PDEs Forward/backward diffusion Imaging through turbulence

3

Image and optimization TV regularization Bregman iterations

4

Optimization and dynamic differential system Inverse scale space flow Forward-backward splitting

24/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system

General Model

We consider a general model: f = Au + n, where u :

• riginal image

A : continuous linear operator n : additive white noise f : blurry noisy image GOAL: given f and A ⇒ Estimate ˜ u ≈ u.

25/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system TV regularization

[Rudin, Osher and Fatemi, 1992] ROF model: E(u) = |∇u| + λ 2Au − f2. The gradient descent gives ut = −∂E ∂u = ∇ · ∇u |∇u| + λAT(f − Au). The steady state gives an optimal solution to minimize E(u).

26/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system TV regularization

One drawback – loss of contrast

Example2: Let A = Id, f(x) =

• 1
• x2 + y2 R
• therwise

and an optimal solution ˜ u from solving ˜ u = arg min |∇u| + λ 2u − f2 is ˜ u =

• 1 − 2

λR λR 2

• therwise

2Meyer 2001, Strang-Chan 2003 27/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Bregman iterations

Remedy – Bregman iterations

Define Bregman divergence3 between two points u and v Dp

J(u, v) := J(u) − J(v) − p, u − v ,

where J(·) is a convex functional, p ∈ ∂J(v) is the subgradient

• f J at the point v.

p is a subgradient of J (not necessarily convex) at v if J(u) J(v) + p, u − v ∀u

3Note Bregman divergence is not a “distance”. 28/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Bregman iterations

Bregman iterations (cont.)

Recall the ROF model: min |∇u| + λ 2Au − f2, and Bregman divergence: Dp

J(u, v) := J(u) − J(v) − p, u − v .

Bregman iteration4: un+1 = argmin Dpn

J (u, un) + λ

2Au − f2

2 .

4Osher-Burger-Goldfarb-Xu-Yin, 2005 29/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Bregman iterations

Adding noise back

The Bregman iterations are equivalent to

• un+1 = arg min J(u) + 1

2Au − f n2

2

f n+1 = f n + f − Aun The “noise” at each step f − Aun is added back to the noisy image.

30/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Bregman iterations

Convergence

Let J(u) = |∇u|, H(u, f) = λ 2Au − f2

2, it is proved that

31/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Bregman iterations

Convergence

Let J(u) = |∇u|, H(u, f) = λ 2Au − f2

2, it is proved that

1

Monotonic decrease in H: H(un+1, f) H(un, f);

31/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Bregman iterations

Convergence

Let J(u) = |∇u|, H(u, f) = λ 2Au − f2

2, it is proved that

1

Monotonic decrease in H: H(un+1, f) H(un, f);

2

Convergence to the original in H with exact data: if ˆ u minimizes H(·, f) and J(ˆ u) < ∞, then H(un, f) H(ˆ u, f) + J(ˆ u)/n;

31/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Bregman iterations

Convergence

Let J(u) = |∇u|, H(u, f) = λ 2Au − f2

2, it is proved that

1

Monotonic decrease in H: H(un+1, f) H(un, f);

2

Convergence to the original in H with exact data: if ˆ u minimizes H(·, f) and J(ˆ u) < ∞, then H(un, f) H(ˆ u, f) + J(ˆ u)/n;

3

Convergence to the original in D with noisy data: Suppose H(ˆ u, f) δ2, H(ˆ u, g) = 0 (g is the noiseless data), then Dpn+1

J

(ˆ u, un+1) < Dpn

J (ˆ

u, un) as long as H(un+1, f) > δ2.

31/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Bregman iterations

Take-home messages for Bregman iterations

1

“Adding noise back” compensates the loss of contrast;

32/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Bregman iterations

Take-home messages for Bregman iterations

1

“Adding noise back” compensates the loss of contrast;

2

Bregman iteration is a way of solving constrained

• ptimization;

32/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Bregman iterations

Take-home messages for Bregman iterations

1

“Adding noise back” compensates the loss of contrast;

2

Bregman iteration is a way of solving constrained

• ptimization;

3

Each subproblem is still hard to solve, un+1 = arg min J(u) + λ 2Au − f n2

2.

Solution: Split Bregman [CAM 08-37].

32/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system

Outline

1

ABCs of image processing

2

Image and PDEs Forward/backward diffusion Imaging through turbulence

3

Image and optimization TV regularization Bregman iterations

4

Optimization and dynamic differential system Inverse scale space flow Forward-backward splitting

33/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Inverse scale space flow

Recall Bregman iterations un+1 = arg min

u J(u) − pn, u + λ

2Au − f2

2

From the Euler-Lagrange equation pn+1 − pn = λAT(f − Au). Now thinking of λ = △t, we arrive at the differential inclusion: ˙ p = AT(f − Au) p(t) ∈ ∂J(u(t)). This is an inverse scale space (ISS) flow5.

5Burger, Gilboa, Osher and Xu, 2006. 34/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Inverse scale space flow

Figure: Evolution of u towards a clean image f with different initial conditions: u = 0 (top), u = n (middle), and u = f (bottom).

35/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward-backward splitting

A general problem: ˜ u = arg min J(u) + H(u) where J(u) is convex but not necessarily smooth, and H(u) is differentiable, but not necessarily convex. The forward-backward algorithm goes as follows, 0 ∈ un+1 − un λ + ∂J(un+1) + ∇H(un), which can be interpreted as first-order differential inclusion: ˙ u + p + ∇H(u(t)) = 0 p ∈ ∂J(u(t)).

36/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward-backward splitting

Consider the problem, min

u J(u) + H(u)

and p ∈ ∂J(u) Optimization v.s. differential equations ˙ p + ∇H(u) = 0 Bregman/ISS ˙ u + p + ∇H(u) = 0 forward-backward ˙ p + δ ˙ u + ∇H(u) = 0 linearized Bregman ˙ p + ˙ u + p + ∇H(u) = 0 used in Neuron network

37/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward-backward splitting

Conclusion

1

Some image applications can be regarded as diffusion

• process. Other useful techniques include statistical

sampling, graph theory, and differential geometry.

38/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward-backward splitting

Conclusion

1

Some image applications can be regarded as diffusion

• process. Other useful techniques include statistical

sampling, graph theory, and differential geometry.

2

After modeling, most problems boil down to optimization: the goal is to seek an efficient, stable algorithm with convergence analysis.

38/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward-backward splitting

Conclusion

1

Some image applications can be regarded as diffusion

• process. Other useful techniques include statistical

sampling, graph theory, and differential geometry.

2

After modeling, most problems boil down to optimization: the goal is to seek an efficient, stable algorithm with convergence analysis.

3

It is interesting to relate optimization scheme to differential equations: higher order, delay system?

38/39

ABCs of image processing Image and PDEs Image and optimization Optimization and dynamic differential system Forward-backward splitting

Reference

Turbulence: CAM report 12-30 Bregman: CAM report 04-13, 07-37, 08-37 Inverse Scale Space methods: CAM report 05-66, 11-08

Thank you!

39/39