Random Graph Models for Image Patches Franois Meyer University of - - PowerPoint PPT Presentation

Random Graph Models for Image Patches

François Meyer University of Colorado at Boulder Joint work with Kye Taylor February 7, 2014 ICERM, Spring 2014; Research Cluster: Geometric analysis methods for graph algorithms

Outline

1 Introduction 2 Notations, definitions 3 Warm-up: a first look at the patch set 4 The graph models 5 A better distance on the graph 6 Commute time on the graph models 7 From the random walk to the eigenvectors

François Meyer | Random Graph Models for Image Patches 2/55

1 Introduction

François Meyer | Random Graph Models for Image Patches 3/55

A fresh perspective on image processing

• 1. break the image into pieces (jigsaw puzzle)
• 2. piece = patch
• 3. gather the pieces that look alike and do something with them
• 4. reconstruct the jigsaw puzzle

François Meyer | Random Graph Models for Image Patches 4/55

Patch-Graph idea: state-of-the art image processing Denoising

• nonlocal means algorithm [Buades et al., 2005]
• learn the local geometry of the set of patches using SVDs

[Elad and Aharon, 2006, Dabov et al., 2009, Guleryuz, 2007]

• denoise by applying a diffusion on the graph of “patches”

[Szlam et al., 2008, Hein and Maier, 2007, Bougleux et al., 2009]

• analysis of the Fokker-Planck diffusion operator

[Singer et al., 2009] Inpainting, Hyper-resolution

• fill in regions with similar patches

[Criminisi et al., 2004, Zontak and Irani, 2011]

François Meyer | Random Graph Models for Image Patches 5/55

Problem statement and contribution Goal: understand the geometry observed in general patch-graphs Line of attack:

• 1. prototypical graph models that epitomize this geometry
• patches within which the intensity varies smoothly,
• patches where the intensity exhibits very rapid changes
• 2. probabilistic analysis of the commute time metric
• 3. spectral decomposition of commute time

→ predict the shape of the eigenvectors of the graph Laplacian

François Meyer | Random Graph Models for Image Patches 6/55

2 Notations, definitions

François Meyer | Random Graph Models for Image Patches 7/55

Image patch Definition 1 Let u be a

√ N× √ N image. Let xn = (i, j) be a pixel with

linear index n = i ×

√ N + j.

We extract an m × m block, centered about xn,

   u(i − m/2, j − m/2) · · · u(i − m/2, j + m/2)

. . . . . .

u(i + m/2, j − m/2) · · · u(i + m/2, j + m/2)    ,

We identify the m × m matrix with a vector in Rm2, and we define the patch u(xn) =

   u1(xn)

. . .

um2(xn)    =    u(i − m/2, j − m/2)

. . .

u(i + m/2, j + m/2)    .

François Meyer | Random Graph Models for Image Patches 8/55

The patch-set Definition 2 The patch-set is defined as the set of patches extracted from the image u,

P = {u(xn), n = 1, 2, . . . , N}.

1 3 2 5 7 4 6 1 2 3 4 5 6 7

The patch-graph

François Meyer | Random Graph Models for Image Patches 9/55

Definition 3 The patch-graph, Γ, is a weighted graph defined as follows.

• 1. The vertices of Γ are the N patches u(xn), n = 1, . . . , N.
• 2. Each vertex u(xn) is connected to its ν nearest neighbors using

the metric

d(n, m) = u(xn) − u(xm) + βxn − xm.

• 3. The weight wn,m along the edge {u(xn), u(xm)} is given by

wn,m =    e−d2(n, m)/σ2

if xn is connected to xm,

• therwise.

François Meyer | Random Graph Models for Image Patches 10/55

3 Warm-up: a first look at the patch set

François Meyer | Random Graph Models for Image Patches 11/55

Local variance within a patch

H I H I

François Meyer | Random Graph Models for Image Patches 12/55

Projection of the patches in R3

H I

• smooth patches within which u(x) varies smoothly

are clustered along low dimensional curves and surfaces

• rough patches within which ∇u(x) is large

are shattered all over the patch-set

François Meyer | Random Graph Models for Image Patches 13/55

Mutual distances: weight matrix W apply a permutation τ of the patch indices: Var(xτ(1)) Var(xτ(2,) · · · Var(xτ(N)).

H I

• smooth patches are close to one another
• rough patches are at a large distance of one another

François Meyer | Random Graph Models for Image Patches 14/55

Distribution of distances between patches: more experimental evidence

[Zontak and Irani, 2011]

distance to the nearest neighbor of a patch: grows exponentially with the gradient within that patch

François Meyer | Random Graph Models for Image Patches 15/55

Distribution of distances between patches: more experimental evidence

• smooth patches within which u(x) varies smoothly are clustered

along low dimensional curves and surfaces

• rough patches within which ∇u(x) is large are shattered all over

the patch-set

• ... and yet rough patches and smooth patches to not talk to one

another “Smooth patches are all alike; every rough patch is rough in its own way”

François Meyer | Random Graph Models for Image Patches 16/55

Patch graph: community structure?

A B C D E F G H I 0.2 0.4 0.6 0.8 low to high low to low high to high

probability that an edge [n, m] connects patches xn and xm

• same variance: low or high
• different variance

→ two communities that are weakly connected.

François Meyer | Random Graph Models for Image Patches 17/55

We need a better distance principal component analysis of the patch set from the butterfly image

− − − − − − − − − − − −

...but where is the beautiful structure?

⇒ we seek a better distance on the patch-set Idea: when physical distances are very different, a notion of connectivity

may be more useful

François Meyer | Random Graph Models for Image Patches 18/55

Summary and Line of attack

• experimental observation about the mutual distance in the dataset

→ change the metric on the graph

• construct a prototypical graph model that epitomizes the experi-

mental patch-graphs – a smooth subgraph of smooth patches ∈ smooth regions – a rough subgraph of rough patches ∈ edges, texture

⇒ predict the shape of the eigenfunctions φk

François Meyer | Random Graph Models for Image Patches 19/55

4 The graph models

François Meyer | Random Graph Models for Image Patches 20/55

The prototypical graph models

• Goal: predict the shape of the eigenfunctions φk on the patch-set

from the knowledge of the commute time

• compute estimates of the commute time on simple graph models
• the graph models epitomize the characteristic features observed in

patch-graphs

• the graph model is composed of two subgraphs:

– a smooth subgraph of smooth patches ∈ smooth regions – a rough subgraph of rough patches ∈ edges, texture

François Meyer | Random Graph Models for Image Patches 21/55

The graph model Mixture of smooth and rough patches: combine a smooth and a rough subgraph of equal size Definition 4 The graph Γ ∗(N) is a weighted graph composed of a smooth subgraph S(N/2, B) and a rough subgraph R(N/2, p). Edges between S(N/2, B) and R(N/2, p) are created randomly and in- dependently with probability q and assigned the edge weight wc > 0.

S (N/2,p) F (N/2,L)

Γ ∗(N)

W

François Meyer | Random Graph Models for Image Patches 22/55

The smooth subgraph model

• smooth patches: large entry Wn,m when |n − m| is small
• spatial proximity ⇒ proximity in patch-space

Definition 5 The smooth graph S(N, B) is a weighted graph com- posed of N vertices, x1, . . . , xN. The weight on the edge {xn, xm} is defined by

wn,m =

• wS

if |n − m| B,

• therwise,

for

1 n, m N

• weight wS > 0: distance between two spatially adjacent patches
• B = thickness of the diagonal in W

François Meyer | Random Graph Models for Image Patches 23/55

The rough subgraph model

• rough patches: large entries in W scattered throughout the matrix
• spatial proximity proximity in patch space

Definition 6 The rough graph R(N, p) is a random weighted graph com- posed of N vertices, x1, . . . , xN. The weight on the edge {xn, xm} is defined by

wn,m =

• wR

with probability p, with probability 1 − p if 1 n < m N, and wn,m = 1 if

n = m.

• weighted Erdös-Renyi graph with self-connections.
• p controls the density of the edges

François Meyer | Random Graph Models for Image Patches 24/55

5 A better distance on the graph

François Meyer | Random Graph Models for Image Patches 25/55

How fast can one diffuse on the patch-set?

• random walk Zk on the vertices of the patch-graph with transition

probability matrix P Pn,m = Prob(Zk+1 = xm|Zk = xn)

wn,m

• l wn,l

= Wn,m

Dn,n

.

• start the random walk at xn and count the number of steps

necessary to reach xm: hitting-time

h(xn, xm) = En min{j 0 : Zj = xm},

• commute time: symmetric hitting time

κ(xn, xm) = h(xn, xm) + h(xm, xn).

François Meyer | Random Graph Models for Image Patches 26/55

Spectral decomposition of the commute time Consider the eigenvectors φ1, . . . , φN of D−1/2W D−1/2 with eigenvalues −1 < λN . . . λ2 < λ1 = 1. Commute time:

κ(xn, xm) =

N

• k=2

1 1 − λk φk(xn) √πn − φk(xm) √πm 2

with

πn =

N

• m=1

wn,m/

N

• j,l=1

wj,l = stationary distribution of the random walk

François Meyer | Random Graph Models for Image Patches 27/55

6 Commute time on the graph models

François Meyer | Random Graph Models for Image Patches 28/55

Commute time on the smooth graph

• if B = 1, then the smooth graph is a path of N vertices
• for a path without self-connections: κ(xn, xm) = 2(N − 1)|m − n|
• if B > 1 then the random walk can move forward by a distance B

at each time step → κ(xn, xn) ∼ O(N/B)? Definition 7 Let κS be the average commute time in the smooth graph

S(N, B) κS 2 N(N − 1)

• 1m<nN

κ(xn, xm).

Lemma 1 We have

(N(2B + 1) − B(B + 1)) 2 (N + 1) 3B2(B + 1) κS.

François Meyer | Random Graph Models for Image Patches 29/55

Commute time on the smooth graph

• consider a fixed pair of vertices in the smooth graph, xn0 and xm0,
• compute a lower bound on the commute time κ(xn0, xm0)

– standard tool to obtain lower bounds on commute time: Nash-Williams inequality [Lyons and Peres, 2005]

• compute the average of this lower bound over all the pairs of

vertices

François Meyer | Random Graph Models for Image Patches 30/55

Sketch of a proof Definition 8 Let V1 and V2 be two disjoint sets of vertices. A set of edges E is an edge-cutset separating V1 and V2 if every path that connects a vertex in V1 with a vertex in V2 includes an edge in E. Lemma 2 (Nash-Williams) If xm0 and xn0 are distinct vertices in a graph that are separated by disjoint edge-cutsets Ek, k = 1, . . ., then

V

• k

 

• {xn,xm}∈Ek

wn,m  

−1

κ(xm0, xn0),

(1) where {xm, xn} is an edge in the cutset Ek and where the volume of the graph is defined by V = N

i=1

N

j=1 wi,j.

François Meyer | Random Graph Models for Image Patches 31/55

Sketch of a proof Sequence of edge cutsets: E1, E2, . . .

E3 m 0 n

any path from m0 to n0 needs to use an edge of the edge-cutset E3

L n0 E2 E3 E4 m 0 E

1

W

François Meyer | Random Graph Models for Image Patches 32/55

Sketch of a proof

• n0−(m0+1)+1

B

• + 1 cutsets between xm0+1 and xn0
• each cut is B(B + 1)ws/2

We have

• k

 

• {n,m}∈Ek

wn,m  

−1

[N(2B + 1) − B(B + 1)] B(B + 1)

• 2n0 − m0

B

• .

also the volume of the smooth graph is

V = wSN + 2wS

• NB − B(B + 1)

2

• = wS(N(2B + 1) − B(B + 1))

François Meyer | Random Graph Models for Image Patches 33/55

Sketch of a proof ...putting everything together yields Lemma 3 The commute time between vertices xn0 and xm0 inside

S(N, B) satisfies κ(xm0, xn0) 2 [N(2B + 1) − B(B + 1)] B(B + 1) n0 − m0 B

• .

(2) Finally, by averaging over all possible pair of vertices we get

κS = 2 N(N − 1)

• 1m<nN

κ(xn, xm) (N(2B + 1) − B(B + 1)) 2 (N + 1) 3B2(B + 1)

François Meyer | Random Graph Models for Image Patches 34/55

Commute time on the rough graph

• if p = 1, then the rough graph is a complete graph, or clique

→ average commute time = O(N)

Definition 9 Let κR be the average commute time between vertices in the giant component of a realization of the rough graph R

κR 1 NG

• n<m

κ(xn, xm), NG = number of distinct pairs of vertices in the giant component.

We choose

p = 2B N − 1 − B(B + 1) N(N − 1),

with

B = C log(N), C > 1

The average degree is approximately 2B, and as N → ∞, Prob(rough graph is fully connected) = 1

François Meyer | Random Graph Models for Image Patches 35/55

Commute time on the rough graph Lemma 4

κR (N(2B + 1) − B(B + 1))  

log N log

• 2B

N N−1 − B(B+1) N−1

• 



with probability approaching 1 as N → ∞.

François Meyer | Random Graph Models for Image Patches 36/55

Sketch of a proof

• assign resistance rn,m = 1/wn,m to the edge {xn, xm}
• effective resistance Rm0,n0 = voltage necessary to maintain a one-

unit current between xm0 and xn0

• commute time between vertices xm0 and xn0 [Chandra et al., 1989]

is given by

κ(xm0, xn0) = VRm0,n0.

(3)

• Rm0,n0

1 wF geodesic distance between xm0 and xn0

we have

κ(xm0, xn0) V δ(xm0, xn0) wF .

(4)

François Meyer | Random Graph Models for Image Patches 37/55

Sketch of a proof

• volume V = wRN + 2wRM
• M = number of edges in the giant component
• M converges to 2p/[N(N − 1)] in probability
• the geodesic distance δ(xm0, xn0) converges to log N/ log(Np) in

probability [Durrett, 2007]

• we choose p =

2B N−1 − B(B+1) N(N−1)

Putting everything together we get

κR (N(2B + 1) − B(B + 1))  

log N log

• 2B

N N−1 − B(B+1) N−1

• 



François Meyer | Random Graph Models for Image Patches 38/55

Growth rate of the commute times Corollary 1 lower bound on κS ∼

• N

log N

2

upper bound on κR ∼ N(log N)2 log log N . numerical simulation: compute the average approximate commute time

κ′ = 2 N(N − 1)

• n<m

  

d′

• k=2

1 1 − λk φk(xn) √πn − φk(xm) √πm 2   

(5) we only keep the d′ ∼ log(N) largest terms in the expansion

François Meyer | Random Graph Models for Image Patches 39/55

Commute time on the fused graph

• numerical simulation
• random realization of the rough graph, random attachment of the

smooth graph

• N = 1024, B = ⌈2 log(N)⌉ , q = log(N)/N, p = 2q, and

wS = wF = wc = 1.

• expect the commute time to be constrained by the slow graph
• fused graph is similar to a lollipop

Maximum commute time: lost in the city...

• among all graphs with N vertices,

what is the graph with the largest κ(i, j) ?

• lollipop graph: path with (N − 1)/3 vertices,

complete subgraph with (2N + 1)/3 vertices

François Meyer | Random Graph Models for Image Patches 40/55

• κ(i, j) =

4 27N3 + O(N) [Jonasson, 2000]

• δ(i, j) = 2

3N,

(N−1)/3 i j (2N+1)/3

François Meyer | Random Graph Models for Image Patches 41/55

Commute time on the fused graph

10

2

10

3

10

4

10

2

10

4

10

6

N Avg Approx commute time A Slow Fast 10

2

10

3

10

4

10 10

1

10

2

10

3

10

4

N Avg Approx commute time B smooth to rough smooth to smooth rough to rough

smooth graph S and rough graph F fused graph Γ ∗.

• dynamics of the fused graph is enslaved by the smooth subgraph
• κF/κS → 0 when the smooth and rough graphs are
• considered separately (left), or
• components of the fused graph (right)

François Meyer | Random Graph Models for Image Patches 42/55

Probability distribution of the commute time κ smooth graph S and rough graph F fused graph Γ ∗.

10 10

5

0.05 0.1 Approx commute−time Normalized frequency A Slow Fast 10 10

1

10

2

10

3

0.1 0.2 0.3 0.4 0.5 Approx commute time Normalized frequency B smooth to rough smooth to slow rough to rough

κ(smooth subgraph) ≈ 102 × κ(rough subgraph)

• κ(smooth → smooth)

κ(smooth → rough) ≈ 102 × κ(rough → rough)

François Meyer | Random Graph Models for Image Patches 43/55

7 From the random walk to the eigenvectors

François Meyer | Random Graph Models for Image Patches 44/55

Smooth and rough subgraphs: spectra smooth graph model is a “fat” path

• spectrum of a path without self-connections

λk = cos [π(k − 1)/(N − 1)] , k = 1, 2, . . . , N.

• eigenvalues associated with the smooth graph will decay smoothly

away from λ1 = 1 for small k rough graph ∼ Erdös-Renyi graph

• λ1 = 1, other eigenvalues asymptotically follow the Wigner semi-

circle distribution

• eigenvalues associated with the rough graph drop quickly from one

for small k

François Meyer | Random Graph Models for Image Patches 45/55

Spectrum of the matrix D−1/2W D−1/2

200 400 600 800 1000 −0.5 0.5 1 k λk entire smooth rough 10 20 30 40 50 −0.5 0.5 1 0.05 0.1 0.15 0.2 0.25 λk Probability entire

• ugh

λk as a function of k

histogram of λk

• smooth subgraph has the largest influence on the first few

(small k) eigenvalues λk of the fused graph

François Meyer | Random Graph Models for Image Patches 46/55

A fundamental observation recall

κ(xn, xm) =

N

• k=2

1 1 − λk φk(xn) √πn − φk(xm) √πm 2

consider a graph where the spectral gap is very small

λ2 − λ1 ≈ 0

• r

1/(1 − λ2)

is very large. If the commute time between two nodes xn and xm is very small

κ(xn, xm) ≈ 0

then we have

⇒ φ2(xn) ≈ φ2(xm)

François Meyer | Random Graph Models for Image Patches 47/55

A fundamental observation More generally, if λk decays very slowly for small values of k, then the largest contribution in κ(xn, xm) comes from

1 1 − λk φk(xn) √πn − φk(xm) √πm 2

for small values of k. Consequently,

κ(xn, xm) ≈ 0 ⇒

φk(xn) ≈ φk(xm) Implication for the eigenvectors on the rough subgraph:

• κ(xn, xm) ≈ 0 if both vertices xn and xm are in the rough subgraph

⇒ if xn, xm ∈ rough subgraph then φk(xn) ≈ φk(xm),

for small values of k

François Meyer | Random Graph Models for Image Patches 48/55

Eigenvectors φk

φ2 φ8 φ16 φ32 φ2 φ8 φ16 φ32 φ2 φ8 φ16 φ32

smooth graph rough graph entire graph

François Meyer | Random Graph Models for Image Patches 49/55

Discussion, Open questions

• realistic graph models and probabilistic arguments to estimate the

commute time of random walks on graphs and predict the shape

• f the eigenvectors
• analysis of the fused graph relies on numerical simulations
• theoretical bounds on the commute time?
• patch-set of an image consists of more than two homogeneous

subsets: uniform patches, edge patches, and texture patches, etc.

• patch-set constructed from a library of images
• similar analysis can be applied to datasets where the corresponding

graph exhibits community structure: e.g. social networks,

• Papers, software: ecee.colorado.edu/~fmeyer

François Meyer | Random Graph Models for Image Patches 50/55

Questions?

Acknowledgments

• National Science Foundation (DMS 0941476, ECS-0501578)

François Meyer | Random Graph Models for Image Patches 52/55

References

[Bougleux et al., 2009] Bougleux, S., Elmoataz, A., and Melkemi, M. (2009). Local and nonlocal discrete regularization on weighted graphs for image and mesh processing. Int. J. Comput. Vision, 84(2):220–236. [Buades et al., 2005] Buades, A., Coll, B., and Morel, J. M. (2005). A review of image denoising algorithms, with a new one. Multiscale

• Model. Sim., 4:490–530.

[Chandra et al., 1989] Chandra, A., Raghavan, P., Ruzzo, W., and Smolensky, R. (1989). The electrical resistance of a graph captures its commute and cover times. In Proc. 21st ACM Symposium on Theory

• f Computing, pages 574–586. ACM.

[Criminisi et al., 2004] Criminisi, A., Pérez, P., and Toyama, K. (2004). Region filling and object removal by exemplar-based image inpainting. Image Processing, IEEE Transactions on, 13(9):1200–1212.

François Meyer | Random Graph Models for Image Patches 53/55

[Dabov et al., 2009] Dabov, K., Foi, A., Katkovnik, V., and Egiazarian,

• K. (2009).

BM3D image denoising with shape-adaptive principal component analysis. In Workshop on signal processing with adaptive sparse structured representations, 2009, pages 1–6. [Durrett, 2007] Durrett, R. (2007). Random Graph Dynamics. Cam- bridge. [Elad and Aharon, 2006] Elad, M. and Aharon, M. (2006). Image denois- ing via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image Processing, 15(12):3736–3745. [Guleryuz, 2007] Guleryuz, O. (2007). Weighted Averaging for Denois- ing With Overcomplete Dictionaries. IEEE Transactions on Image Processing, 16(12):3020–3034. [Hein and Maier, 2007] Hein, M. and Maier, M. (2007). Manifold de-

• noising. In Adv. Neur. In., pages 561–568.

François Meyer | Random Graph Models for Image Patches 54/55

[Jonasson, 2000] Jonasson, J. (2000). Lollipop graphs are extremal for commute times. Random Structures and Algorithms, 16(2):131–142. [Lyons and Peres, 2005] Lyons, R. and Peres, Y. (2005). Probability

• n trees and networks. In preparation. Available at http://http:

//mypage.iu.edu/~rdlyons/http://mypage.iu.edu/ rdlyons. [Singer et al., 2009] Singer, A., Shkolnisky, Y., and Nadler, B. (2009). Diffusion Interpretation of Nonlocal Neighborhood Filters for Signal

• Denoising. SIAM J. Imag. Sci., 2(1):118–139.

[Szlam et al., 2008] Szlam, A., Maggioni, M., and Coifman, R. (2008). Regularization on graphs with function-adapted diffsion processes. J.

• Mach. Learn. Res., 9:1711–1739.

[Zontak and Irani, 2011] Zontak, M. and Irani, M. (2011). Internal statistics of a single natural image. In IEEE Conf. CVPR, 2011, pages 977 –984.

François Meyer | Random Graph Models for Image Patches 55/55