Image and Shape The beginning Analogies The middle A few pictures - - PowerPoint PPT Presentation
Lecture overview Image and Shape The beginning Analogies The middle A few pictures Some more middle The transfer of color and motion The end Presented by: Lior Shapira, PhD student Find the analogies! Analogies An
Presented by: Lior Shapira, PhD student
The beginning The middle A few pictures Some more middle The end
“An analogy is a comparison between two different things, in order to highlight some form of similarity. “
Transfer the colors of one image to
another
Analyze images Find correspondence (which color goes
where?)
Perform a smooth transfer of the colors
Working in geometric coordinates
A regular image
Image pixels in RGB space
Image pixels in RGB space
To find analogies we’ll attempt to find
correspondences between clusters in the data
Clustering of data is a method by which
large sets of data is grouped together into clusters of smaller sets of similar data
A very simple way to cluster data
{ }
,
5 2 , 1 1 ,
arg min
j i j
n i j i j j i C m
m x C
= =
−
5 , 1
1 the j-th cluster the j-th cluster 1 any a single cluster
i i j i i j j i
x m x m x
=
∈ ⎧ = ⎨ ∉ ⎩ = → ∈
K-means
Start with a random
guess of cluster centers
Determine the
membership of each data points
Adjust the cluster
centers
K-means
Start with a random
guess of cluster centers
Determine the
membership of each data points
Adjust the cluster
centers
K-means
Start with a random
guess of cluster centers
Determine the
membership of each data points
Adjust the cluster
centers
1. Ask user how many clusters they’d like. (e.g. k= 5)
1.
Ask user how many clusters they’d like. (e.g. k= 5)
2.
Randomly guess k cluster Center locations
1.
Ask user how many clusters they’d like. (e.g. k= 5)
2.
Randomly guess k cluster Center locations
3.
Each datapoint finds
closest to. (Thus each Center “owns” a set of datapoints)
1.
Ask user how many clusters they’d like. (e.g. k= 5)
2.
Randomly guess k cluster Center locations
3.
Each datapoint finds
closest to.
4.
Each Center finds the centroid of the points it owns
1.
Ask user how many clusters they’d like. (e.g. k= 5)
2.
Randomly guess k cluster Center locations
3.
Each datapoint finds
closest to.
4.
Each Center finds the centroid of the points it owns
User inputs number of clusters Sensitive to initial guesses Converges to local minimum Problematic with clusters of
Differing sizes Differing densities Non globular shapes
Difficulty with outliers
Normal distribution (1D)
2 2
1 ( , ) exp 2 2 x f x µ µ σ σ σ π ⎛ ⎞ − = − ⎜ ⎟ ⎜ ⎟ ⎝ ⎠
2D Gaussians
d = 2 x = random data point (2D vector)
1
1 ( , ) exp 2 2 det( )
T d
x x f x µ µ µ π
−
⎛ ⎞ − Σ − Σ = − ⎜ ⎟ ⎜ ⎟ Σ ⎝ ⎠ Σ µ
The same equation holds for a 3D Gaussian
1
1 ( , ) exp 2 2 det( )
T d
x x f x µ µ µ π
−
⎛ ⎞ − Σ − Σ = − ⎜ ⎟ ⎜ ⎟ Σ ⎝ ⎠ µ Σ
2 2 1
( , ) cov( , ) 1 cov( , )
i i i N T w i i i h
x random vector w h w h x x N h w σ µ µ σ
=
= ⎛ ⎞ Σ = − − = ⎜ ⎟ ⎝ ⎠
⇒ Σ * *
T
V D V Σ = Σ
1 2
...
d
λ λ λ ≥ ≥ ≥
Σ
1 2
* * 1* 2*
T
V D V a v b v λ λ Σ = = = b a
( )( )
2 2 1 2
( , , ) cov( , ) cov( , ) 1 cov( , ) cov( , ) cov( , ) cov( , )
i r N T i i g i b
x r g b g r b r x x r g b g N r b g b σ µ µ σ σ
=
= ⎛ ⎞ ⎜ ⎟ Σ = − − = ⎜ ⎟ ⎜ ⎟ ⎝ ⎠
W H
Suppose we have 1000 data points in 2D space (w,h)
W H
Assume each data point is normally distributed Obviously, there are 5 sets of underlying gaussians
There are K components (Gaussians) Each k is specified with three parameters:
weight, mean, covariance matrix
The total density function is:
( )
( ) ( )
1 1 1 1
1 ( ) exp 2 2 det( ) { , , } 1
T K j j j j d j j K j j j j K j j j
x x f x weight j µ µ α π α µ α α α
− = = =
⎛ ⎞ − Σ − ⎜ ⎟ Θ = − ⎜ ⎟ Σ ⎝ ⎠ Θ = Σ = ≥ ∀ =
Raw data GMMs (K = 6) Total Density Function
i
Σ
i
µ
Objective:
Given N data points, find maximum likelihood estimation of :
Algorithm:
Based on , associate each data point with specific gaussian
Based on data points clustering, maximize
Θ
1
N
Θ
Θ Θ Θ
, 1
( , ) 1,..., 1,..., ( , )
j t j j t j K i t i i i
f x w j K t N f x α µ α µ
=
Σ = = = Σ
, 1 , 1 , 1 , 1 , 1
1 ( )( )
N new j t j t N t j t new t j N t j t N new new T t j t j t j new t j N t j t
w N w x w w x x w α µ µ µ
= = = = =
= = − − Σ =
Gaussian j data point t blue: wt,j
Suppose we cluster the points for 2 clusters
We want to label “close” pixels with the same
label
Proposed metric: label pixels from the same
gaussian with same label
Label according to max probability: Number of labels = K
,
( ) arg max( )
t j j
label t w =
The result in image space
A Problem: Noise Why? Pixel labeling is done independently for each pixel,
ignoring the spatial relationships between pixels!
Previous model for labeling: A new model for labeling. Minimize E:
f = Labeling function, assigns label fp for each pixel p Edata = Data Term Esmooth = Smooth Term Lamda is a free parameter
,
1,..., ( ) ( ) arg max( ) ( )
p p j j
j K gaussians f label p w p image pixels = = = ∈Ρ ( ) ( ) ( )
data smooth
E f E f E f λ = +
Labels Set: { j= 1,…,K } Edata:
Penalize disagreement between pixel and the GMM
Esmooth:
Penalize disagreement between two pixels, unless it’s a
natural edge in the image
dist(p,q) = normalized color-distance between p,q
,
( ) ( ) 1
p
p p p p p f p Pixels
D f D f w
∈
= −
, , ,
( , ) ( , ) 1 ( , )
p q p q p q p q p q p q neighbors
f f V f f V f f dist p q
= ⎧ = ⎨ − ⎩
( ) ( ) ( )
data smooth
E f E f E f λ = + Solving Min(E) is NP-hard It is possible to approximate the solution using
iterative methods
Graph-Cuts based methods approximate the
global solution (up to constant factor) in polynomial time
When using iterative methods, each iteration some of the
pixels change their labeling
Given a label α, a move from partition P (labeling f) to a
new partition P’ (labeling f’) is called an α-expansion move if:
Current Labeling One Pixel Move α-β-swap Move α-expansion Move
' '
l l
P P P P l
α α
α ⊂ ∧ ⊂ ∀ ≠
3.1 Find f’ = argmin(E(f’)) among f’ within one α-expansion
3.2 If E(f’) < E(f), set f = f’ and success = 1
How to find argmin(E(f’)) ?
Given two terminal nodes α and β in G= (V,E), a cut is a set
Also, no proper subset of C separates α from β in G’. The cost of a cut is defined as the sum of all the edge
weights in the cut. The minimum-cut of G is the cut C with the lowest cost.
The minimum-cut problem is solvable in practically linear
time.
⊂
Problem:
Find f’ = argmin(E(f’)) among f’ within one α-expansion of f
Solution:
Translate the problem to a min-cut problem on an appropriately defined graph.
p p p p
if nodet C f f if nodet C
α α
α ⎧ ∈ ⎪ = ⎨ ∈ ⎪ ⎩
Terminal α Terminal not(α) Cut C
1-1 correspondence between cut and labeling E(f) is minimized!
Each pixel gets a node
P1 P2 Pα
Add auxiliary nodes between pixel with different
labels
P1 P2 Pα Add two terminal nodes for α and not(α)
P1 P2 Pα
P1 P2 Pα
P1 P2 Pα
P1 P2 Pα
P1 P2 Pα
P1 P2 Pα
Once we cluster and label the pixels in
each picture we can match between them
source reference result
Color transfer between images/Reinhard, Ashikmin, Gooch & Shirley
source reference target
Source Reference Labeling Labeling
The dominant 3D object representation – Triangular meshes
Geometry Connectivity
1 2 3 (1.56, 2.76, 3) (0.17, 2.06, 1.5) (3.1, 2, 2.22) (x, y, z)
What can ‘analogies’ mean in 3D?
Shape Matching Global Registration Complete Correspondence Part Correspondence
Given a database of 3D models, and a
query model, retrieve all similar objects
Given two 3D models, find the best
matching alignment between them
Find for each triangle in the
first model, a corresponding triangle in the second model.
Sample application – deformation transfer
Deformation transfer for triangle meshes / Sumner, Popovic’. SIGGRAPH 2004
Attempt to ‘think’ like a human, match logical
parts
Body Body Tail Tail Leg Leg …
We’re looking for a way to find analogies
between 3D models
3D triangular meshes are surface
representations of volumetric objects
A strong link between surface and volume
is the Medial Axis Transform ( MAT)
We grow circles within the given shape, until we are blocked by the shape’s
Things get more complicated in 3D
Instead of circles we grow spheres The MAT is a complicated structure composed
The MAT is difficult to compute and hard to
handle
Things get more complicated in 3D
Medial Axis Local shape radius Local shape diameter We wish to define a
scalar function which links the mesh’s surface to its volume
Highlights similar parts in similar 3D shapes
Remains consistent through pose differences of the same
The SDF gives a good intuition by itself on
a logical partitioning, but not perfect
The rays in this part see a ‘larger’ part of the mesh
We find good partitioning values, which
divide the mesh into significant parts
A partitioning is defined, but it is noisy
We find good partitioning values, which
divide the mesh into significant parts
A mincut operation is performed to improve the boundaries between the different parts
Given a logical partitioning of different meshes,
its easier to find correspondence between different parts
Given a logical partitioning of different meshes,
its easier to find correspondence between different parts
matching the best matching parts between two models
neighboring parts of the first matched parts
recursively until all parts are matched (if possible)
Not all meshes have the same parts
In some cases we wish to find correspondence only between specific parts, possibly a one-to-many connection
Here we transfer motion between
analogous parts of multiple models
Here we transfer motion between
analogous parts of multiple models