Learning Two-View Stereo Matching Jianxiong Xiao Jingni Chen - - PowerPoint PPT Presentation
Learning Two-View Stereo Matching Jianxiong Xiao Jingni Chen Dit-Yan Yeung Long Quan Department of Computer Science and Engineering The Hong Kong University of Science and Technology The 10th European Conference on Computer Vision Jianxiong
Jianxiong Xiao Jingni Chen Dit-Yan Yeung Long Quan
Department of Computer Science and Engineering The Hong Kong University of Science and Technology
The 10th European Conference on Computer Vision
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 1 / 45
1
Introduction
2
Semi-supervised Matching Framework Local Label Preference Cost Regional Surface Shape Cost Global Epipolar Geometry Cost Symmetric Visibility Consistency Cost
3
Iterative MV Optimization
4
Learning the Symmetric Affinity Matrix
5
More Details
6
Experiments
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 2 / 45
Introduction
1
Introduction
2
Semi-supervised Matching Framework Local Label Preference Cost Regional Surface Shape Cost Global Epipolar Geometry Cost Symmetric Visibility Consistency Cost
3
Iterative MV Optimization
4
Learning the Symmetric Affinity Matrix
5
More Details
6
Experiments
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 3 / 45
Introduction
Input: two wide-baseline images taken from the same static scene, neither calibrated nor rectified. For more general applications, such as robust motion estimation from structure.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 4 / 45
Introduction
Small-baseline matching algorithm: cannot be extended easily when the epipolar lines are not parallel. Wide-baseline matching: depend heavily on the epipolar geometry which has to be provided, often by off-line calibration. Sparse matching: the fundamental matrix so estimated often fits to subsets of image, not the whole image. Region growing based methods: greedily, bad result for quite different pixel scales due to discrete growing. Learning techniques: the information learned from other irrelevant images is very weak, the quality of the result greatly depends on the training data.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 5 / 45
Introduction
Small-baseline matching algorithm: cannot be extended easily when the epipolar lines are not parallel. Wide-baseline matching: depend heavily on the epipolar geometry which has to be provided, often by off-line calibration. Sparse matching: the fundamental matrix so estimated often fits to subsets of image, not the whole image. Region growing based methods: greedily, bad result for quite different pixel scales due to discrete growing. Learning techniques: the information learned from other irrelevant images is very weak, the quality of the result greatly depends on the training data.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 5 / 45
Introduction
Small-baseline matching algorithm: cannot be extended easily when the epipolar lines are not parallel. Wide-baseline matching: depend heavily on the epipolar geometry which has to be provided, often by off-line calibration. Sparse matching: the fundamental matrix so estimated often fits to subsets of image, not the whole image. Region growing based methods: greedily, bad result for quite different pixel scales due to discrete growing. Learning techniques: the information learned from other irrelevant images is very weak, the quality of the result greatly depends on the training data.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 5 / 45
Introduction
Small-baseline matching algorithm: cannot be extended easily when the epipolar lines are not parallel. Wide-baseline matching: depend heavily on the epipolar geometry which has to be provided, often by off-line calibration. Sparse matching: the fundamental matrix so estimated often fits to subsets of image, not the whole image. Region growing based methods: greedily, bad result for quite different pixel scales due to discrete growing. Learning techniques: the information learned from other irrelevant images is very weak, the quality of the result greatly depends on the training data.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 5 / 45
Introduction
Small-baseline matching algorithm: cannot be extended easily when the epipolar lines are not parallel. Wide-baseline matching: depend heavily on the epipolar geometry which has to be provided, often by off-line calibration. Sparse matching: the fundamental matrix so estimated often fits to subsets of image, not the whole image. Region growing based methods: greedily, bad result for quite different pixel scales due to discrete growing. Learning techniques: the information learned from other irrelevant images is very weak, the quality of the result greatly depends on the training data.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 5 / 45
Introduction
Propose a semi-supervised prospective of the matching problem without training. Utilize all information in an optimization procedure: local, regional and global. More robust to noise: the label vector is affected not merely by one matched pair but by all pairs with weighted paths to it. Capable of handling real number labels which is the inherent requirement of sub-pixel accuracy matching.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 6 / 45
Semi-supervised Matching Framework
1
Introduction
2
Semi-supervised Matching Framework Local Label Preference Cost Regional Surface Shape Cost Global Epipolar Geometry Cost Symmetric Visibility Consistency Cost
3
Iterative MV Optimization
4
Learning the Symmetric Affinity Matrix
5
More Details
6
Experiments
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 7 / 45
Semi-supervised Matching Framework
Supervised Learning
Given that √ and ×. Now, whether is √ or × ?
Unsupervised Learning
Given , and , any interesting structure in them?
Semi-supervised Learning
× √ ?
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 8 / 45
Semi-supervised Matching Framework
Supervised Learning
Given that √ and ×. Now, whether is √ or × ?
Unsupervised Learning
Given , and , any interesting structure in them?
Semi-supervised Learning
× √ ?
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 8 / 45
Semi-supervised Matching Framework
Supervised Learning
Given that √ and ×. Now, whether is √ or × ?
Unsupervised Learning
Given , and , any interesting structure in them?
Semi-supervised Learning
× √ ?
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 8 / 45
Semi-supervised Matching Framework
Supervised Learning
Given that √ and ×. Now, whether is √ or × ?
Unsupervised Learning
Given , and , any interesting structure in them?
Semi-supervised Learning
× √ ?
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 8 / 45
Semi-supervised Matching Framework
For p = 1 or 2, q = 3−p: xp
(sp−1)×cp+tp: coordinate position (sp,tp) in the p-th image space,
sp ∈ {1,··· ,rp}, tp ∈ {1,··· ,cp},i = (sp −1)×cp +tp. X p: Input image with size np = rp ×cp pixels Xp =
1,xp 2,...,xp (sp−1)×cp+tp,...,xp np
T . xq
j :a matching point of xp i located at coordinate position (sq,tq) in
the q-th continuous image space, sq,tq ∈ R. Label Vector yp
i =
i ,hp i
T =
−
representing the position offset from the point in the first image to the point in the second image. Label Matrix Yp =
1,··· ,yp np
Tand Visibility Vector Op =
1 ,··· ,op np
T.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 9 / 45
Semi-supervised Matching Framework
For p = 1 or 2, q = 3−p: xp
(sp−1)×cp+tp: coordinate position (sp,tp) in the p-th image space,
sp ∈ {1,··· ,rp}, tp ∈ {1,··· ,cp},i = (sp −1)×cp +tp. X p: Input image with size np = rp ×cp pixels Xp =
1,xp 2,...,xp (sp−1)×cp+tp,...,xp np
T . xq
j :a matching point of xp i located at coordinate position (sq,tq) in
the q-th continuous image space, sq,tq ∈ R. Label Vector yp
i =
i ,hp i
T =
−
representing the position offset from the point in the first image to the point in the second image. Label Matrix Yp =
1,··· ,yp np
Tand Visibility Vector Op =
1 ,··· ,op np
T.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 9 / 45
Semi-supervised Matching Framework
For p = 1 or 2, q = 3−p: xp
(sp−1)×cp+tp: coordinate position (sp,tp) in the p-th image space,
sp ∈ {1,··· ,rp}, tp ∈ {1,··· ,cp},i = (sp −1)×cp +tp. X p: Input image with size np = rp ×cp pixels Xp =
1,xp 2,...,xp (sp−1)×cp+tp,...,xp np
T . xq
j :a matching point of xp i located at coordinate position (sq,tq) in
the q-th continuous image space, sq,tq ∈ R. Label Vector yp
i =
i ,hp i
T =
−
representing the position offset from the point in the first image to the point in the second image. Label Matrix Yp =
1,··· ,yp np
Tand Visibility Vector Op =
1 ,··· ,op np
T.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 9 / 45
Semi-supervised Matching Framework
For p = 1 or 2, q = 3−p: xp
(sp−1)×cp+tp: coordinate position (sp,tp) in the p-th image space,
sp ∈ {1,··· ,rp}, tp ∈ {1,··· ,cp},i = (sp −1)×cp +tp. X p: Input image with size np = rp ×cp pixels Xp =
1,xp 2,...,xp (sp−1)×cp+tp,...,xp np
T . xq
j :a matching point of xp i located at coordinate position (sq,tq) in
the q-th continuous image space, sq,tq ∈ R. Label Vector yp
i =
i ,hp i
T =
−
representing the position offset from the point in the first image to the point in the second image. Label Matrix Yp =
1,··· ,yp np
Tand Visibility Vector Op =
1 ,··· ,op np
T.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 9 / 45
Semi-supervised Matching Framework
For p = 1 or 2, q = 3−p: xp
(sp−1)×cp+tp: coordinate position (sp,tp) in the p-th image space,
sp ∈ {1,··· ,rp}, tp ∈ {1,··· ,cp},i = (sp −1)×cp +tp. X p: Input image with size np = rp ×cp pixels Xp =
1,xp 2,...,xp (sp−1)×cp+tp,...,xp np
T . xq
j :a matching point of xp i located at coordinate position (sq,tq) in
the q-th continuous image space, sq,tq ∈ R. Label Vector yp
i =
i ,hp i
T =
−
representing the position offset from the point in the first image to the point in the second image. Label Matrix Yp =
1,··· ,yp np
Tand Visibility Vector Op =
1 ,··· ,op np
T.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 9 / 45
Semi-supervised Matching Framework
IDEA: Nearby pixels are more likely to have similar label vectors. Smoothness Assumption = ⇒A graph G = V ,E . Two Images = ⇒Two Graphs G 1 =
and G 2 =
. N
i
i .
Affinity Matrix Wp, wp
ij is non-zero iff xp i and xp j are neighbors in E p.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 10 / 45
Semi-supervised Matching Framework
IDEA: Nearby pixels are more likely to have similar label vectors. Smoothness Assumption = ⇒A graph G = V ,E . Two Images = ⇒Two Graphs G 1 =
and G 2 =
. N
i
i .
Affinity Matrix Wp, wp
ij is non-zero iff xp i and xp j are neighbors in E p.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 10 / 45
Semi-supervised Matching Framework
IDEA: Nearby pixels are more likely to have similar label vectors. Smoothness Assumption = ⇒A graph G = V ,E . Two Images = ⇒Two Graphs G 1 =
and G 2 =
. N
i
i .
Affinity Matrix Wp, wp
ij is non-zero iff xp i and xp j are neighbors in E p.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 10 / 45
Semi-supervised Matching Framework
IDEA: Nearby pixels are more likely to have similar label vectors. Smoothness Assumption = ⇒A graph G = V ,E . Two Images = ⇒Two Graphs G 1 =
and G 2 =
. N
i
i .
Affinity Matrix Wp, wp
ij is non-zero iff xp i and xp j are neighbors in E p.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 10 / 45
Semi-supervised Matching Framework
IDEA: Nearby pixels are more likely to have similar label vectors. Smoothness Assumption = ⇒A graph G = V ,E . Two Images = ⇒Two Graphs G 1 =
and G 2 =
. N
i
i .
Affinity Matrix Wp, wp
ij is non-zero iff xp i and xp j are neighbors in E p.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 10 / 45
Semi-supervised Matching Framework
Many existing matching techniques such as SIFT have already been powerful enough to recover some sparse matched pairs accurately and robustly. Labeled Data
l ,Y1 l
l ,Y2 l
⇒Unlabeled Data
u,Y1 u
u,Y2 u
Semi-supervised learning on the graph representation tries to estimate a label matrix ˆ Yp that is consistent with:
◮ the initial incomplete label matrix, ◮ the geometry of the data manifold induced by the graph structure. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 11 / 45
Semi-supervised Matching Framework
Many existing matching techniques such as SIFT have already been powerful enough to recover some sparse matched pairs accurately and robustly. Labeled Data
l ,Y1 l
l ,Y2 l
⇒Unlabeled Data
u,Y1 u
u,Y2 u
Semi-supervised learning on the graph representation tries to estimate a label matrix ˆ Yp that is consistent with:
◮ the initial incomplete label matrix, ◮ the geometry of the data manifold induced by the graph structure. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 11 / 45
Semi-supervised Matching Framework
Many existing matching techniques such as SIFT have already been powerful enough to recover some sparse matched pairs accurately and robustly. Labeled Data
l ,Y1 l
l ,Y2 l
⇒Unlabeled Data
u,Y1 u
u,Y2 u
Semi-supervised learning on the graph representation tries to estimate a label matrix ˆ Yp that is consistent with:
◮ the initial incomplete label matrix, ◮ the geometry of the data manifold induced by the graph structure. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 11 / 45
Semi-supervised Matching Framework
Many existing matching techniques such as SIFT have already been powerful enough to recover some sparse matched pairs accurately and robustly. Labeled Data
l ,Y1 l
l ,Y2 l
⇒Unlabeled Data
u,Y1 u
u,Y2 u
Semi-supervised learning on the graph representation tries to estimate a label matrix ˆ Yp that is consistent with:
◮ the initial incomplete label matrix, ◮ the geometry of the data manifold induced by the graph structure. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 11 / 45
Semi-supervised Matching Framework
Many existing matching techniques such as SIFT have already been powerful enough to recover some sparse matched pairs accurately and robustly. Labeled Data
l ,Y1 l
l ,Y2 l
⇒Unlabeled Data
u,Y1 u
u,Y2 u
Semi-supervised learning on the graph representation tries to estimate a label matrix ˆ Yp that is consistent with:
◮ the initial incomplete label matrix, ◮ the geometry of the data manifold induced by the graph structure. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 11 / 45
Semi-supervised Matching Framework
Many existing matching techniques such as SIFT have already been powerful enough to recover some sparse matched pairs accurately and robustly. Labeled Data
l ,Y1 l
l ,Y2 l
⇒Unlabeled Data
u,Y1 u
u,Y2 u
Semi-supervised learning on the graph representation tries to estimate a label matrix ˆ Yp that is consistent with:
◮ the initial incomplete label matrix, ◮ the geometry of the data manifold induced by the graph structure. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 11 / 45
Semi-supervised Matching Framework
Given a configuration ˆ Yp, consistency with the initial labeling can be measured by C p
l
Yp,Op = ∑
xp
i ∈X p l
i
yp
i −yp i
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 12 / 45
Semi-supervised Matching Framework
Consistency with the geometry of the data in the image space, which follows from the smooth manifold assumption, motivates a penalty term of the form C p
s
Yp,Op = 1 2
xp
i ,xp j ∈X p
wp
ij φ
i ,op j
yp
i − ˆ
yp
j
, where φ
i ,op j
2
i
2 +
j
2 penalizes rapid changes in ˆ Yp between points that are close, and only enforces smoothness within visible regions.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 13 / 45
Semi-supervised Matching Framework Local Label Preference Cost
1
Introduction
2
Semi-supervised Matching Framework Local Label Preference Cost Regional Surface Shape Cost Global Epipolar Geometry Cost Symmetric Visibility Consistency Cost
3
Iterative MV Optimization
4
Learning the Symmetric Affinity Matrix
5
More Details
6
Experiments
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 14 / 45
Semi-supervised Matching Framework Local Label Preference Cost
The local cost is defined as: C p
d
Yp,Op = ∑
xp
i ∈X p
i ρp i
ˆ yp
i
i
i
Similarity cost function ρp
i (y) : similarity cost between the pixel xp i in
Penalty term τp
i = maxxp
j ∈N (xp i )
i −xp j
point tends to have zero visibility to escape from cost charging.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 15 / 45
Semi-supervised Matching Framework Regional Surface Shape Cost
1
Introduction
2
Semi-supervised Matching Framework Local Label Preference Cost Regional Surface Shape Cost Global Epipolar Geometry Cost Symmetric Visibility Consistency Cost
3
Iterative MV Optimization
4
Learning the Symmetric Affinity Matrix
5
More Details
6
Experiments
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 16 / 45
Semi-supervised Matching Framework Regional Surface Shape Cost
Shape Cues
The shapes of the 3D objects’ surface in the scene are very important cues for matching.
Intuitive Approach
To reconstruct the 3D surface based on two-view geometry. Unstable, especially when the baseline is not large enough.
Piecewise Planar Patch Assumption
Since two data points with high affinity relation are more likely to have similar label vectors, we assume that the label vector of a data point can be linearly approximated by the label vectors of its neighbors.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 17 / 45
Semi-supervised Matching Framework Regional Surface Shape Cost
Shape Cues
The shapes of the 3D objects’ surface in the scene are very important cues for matching.
Intuitive Approach
To reconstruct the 3D surface based on two-view geometry. Unstable, especially when the baseline is not large enough.
Piecewise Planar Patch Assumption
Since two data points with high affinity relation are more likely to have similar label vectors, we assume that the label vector of a data point can be linearly approximated by the label vectors of its neighbors.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 17 / 45
Semi-supervised Matching Framework Regional Surface Shape Cost
Shape Cues
The shapes of the 3D objects’ surface in the scene are very important cues for matching.
Intuitive Approach
To reconstruct the 3D surface based on two-view geometry. Unstable, especially when the baseline is not large enough.
Piecewise Planar Patch Assumption
Since two data points with high affinity relation are more likely to have similar label vectors, we assume that the label vector of a data point can be linearly approximated by the label vectors of its neighbors.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 17 / 45
Semi-supervised Matching Framework Regional Surface Shape Cost
The label of a data point can be linearly constructed by its neighbors: yp
i =
xp
j ∈N (xp i )
wp
ij yp j .
The reconstruction cost can be defined as Cr (Yp) =
xk
i ∈X k
i −
xp
j ∈N (xp i )
wp
ij yp j
= (I−Wp)Yp2
F
≈ tr
=
xp
i ,xp j ∈X p
ap
ij
i −yp j
, Ap = Wp +(Wp)T −Wp (Wp)T: adjacency matrix Dp is a diagonal matrix containing the row sums of Ap,Dp ≈ I Lp = Dp −Ap: un-normalized graph Laplacian matrix
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 18 / 45
Semi-supervised Matching Framework Regional Surface Shape Cost
To align the two 2D manifolds (image spaces) to one 2D manifold (visible surface). The labeled data (known matched pairs) are accounted for by constraining the mapped coordinates of matched points to coincide. X p
c = X p l ∪X p u ∪X q u ,
ˆ Yp
c =
ˆ Yp
l
ˆ Yp
u
ˆ Yq
u
, Op
c =
Op
l
Op
u
Oq
u
, Ap = Ap
ll
Ap
lu
Ap
ul
Ap
uu
Ap
c =
Ap
ll +Aq ll
Ap
lu
Aq
lu
Ap
ul
Ap
uu
Aq
ul
Aq
uu
, C p
r
Y1, ˆ Y2,O1,O2 =
xp
i ,xp j ∈X p c
(ap
c)ij φ
c )i ,(op c )j
yp
c)i −(ˆ
yp
c)j
.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 19 / 45
Semi-supervised Matching Framework Global Epipolar Geometry Cost
1
Introduction
2
Semi-supervised Matching Framework Local Label Preference Cost Regional Surface Shape Cost Global Epipolar Geometry Cost Symmetric Visibility Consistency Cost
3
Iterative MV Optimization
4
Learning the Symmetric Affinity Matrix
5
More Details
6
Experiments
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 20 / 45
Semi-supervised Matching Framework Global Epipolar Geometry Cost
For xp
i at position (sp,tp), the epipolar line:
i ,bp i ,cp i
pq.
Squared Euclidean distance in the image space of the other image: dp
i (y) =
i sq +bp i tq +cp i
2
i
2 +
i
2 , where y =
−
C p
g
Yp,Op = ∑
xp
i ∈X p
i dp i
ˆ yp
i
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 21 / 45
Semi-supervised Matching Framework Symmetric Visibility Consistency Cost
1
Introduction
2
Semi-supervised Matching Framework Local Label Preference Cost Regional Surface Shape Cost Global Epipolar Geometry Cost Symmetric Visibility Consistency Cost
3
Iterative MV Optimization
4
Learning the Symmetric Affinity Matrix
5
More Details
6
Experiments
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 22 / 45
Semi-supervised Matching Framework Symmetric Visibility Consistency Cost
If xp
i in one image has label to match with a point in the q-th image, then
there must exist some point in the q-th image to have label to match xp
i .
C p
v
Yq = β ∑xp
i ∈X p
i −γp i
Yq2 + 1
2 ∑xp
i ,xp j ∈X p wp
ij
i −op j
2 γ function indicates whether or not there exist one or more data points that match a point near xp
i from the other view according to ˆ
Yq. Last term enforces the smoothness of the occlusion. β controls the strength of the visibility constraint.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 23 / 45
Semi-supervised Matching Framework Symmetric Visibility Consistency Cost
For each point xq
j at position (sq,tq) in X q with label yq j =
j ,hq j
T : Place a 2D Gaussian ψ (s,t) on the p-th image centered at the matched position cj = (sp,tp)T . = ⇒ Mixture of Gaussian ∑xq
j ψcj (s,t) in the voted image space.
Truncate it γp (s,t) = min
xq
j ∈X q
ψcj (s,t)
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 24 / 45
Iterative MV Optimization
1
Introduction
2
Semi-supervised Matching Framework Local Label Preference Cost Regional Surface Shape Cost Global Epipolar Geometry Cost Symmetric Visibility Consistency Cost
3
Iterative MV Optimization
4
Learning the Symmetric Affinity Matrix
5
More Details
6
Experiments
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 25 / 45
Iterative MV Optimization
The optimization process has two steps:
1 M-step: estimate matching given visibility 2 V-step: estimate visibility given matching. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 26 / 45
Iterative MV Optimization
Visibility term Cv imposes two constraints on ˆ Y given O
1 Local Constraint : For each pixel xp
i in the p-th image, it should not
match the invisible (occluded) points in the other image.
2 Global Constraint: For each visible point in the q-th image, at least
In the M-step, we approximate the visibility term by considering only the local constraint, which can be incorporated into the similarity function ρp
i (y) in Cd.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 27 / 45
Iterative MV Optimization
Visibility term Cv imposes two constraints on ˆ Y given O
1 Local Constraint : For each pixel xp
i in the p-th image, it should not
match the invisible (occluded) points in the other image.
2 Global Constraint: For each visible point in the q-th image, at least
In the M-step, we approximate the visibility term by considering only the local constraint, which can be incorporated into the similarity function ρp
i (y) in Cd.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 27 / 45
Iterative MV Optimization
Let Y =
CM
Y
p=1,2
l +λsC p s +λdC p d +λrC p r +λgC p g
Y
where ε
Y
situations. For fixed O1 and O2, the cost minimization is done by setting the derivative with respect to ˆ Y to be zero since the second derivative is a positive definite matrix.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 28 / 45
Iterative MV Optimization
Let O =
CV (O) = ∑
p=1,2
l +λsC p s +λdC p d +λrC p r +λgC p g +λvC p v
For fixed ˆ Y1 and ^ Y2, the cost minimization is done by setting the derivative with respect to O to be zero since the second derivative is a positive definite matrix.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 29 / 45
Iterative MV Optimization
Solving
We can derive that by the way we defined Wp and the cost functions, the coefficient matrix is strictly diagonally dominant and positive definite. Hence, Gauss-Seidel and Conjugate Gradient iterations both converge to the solution of the linear system with theoretical guarantee. GPU is helpful to speed up.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 30 / 45
Learning the Symmetric Affinity Matrix
1
Introduction
2
Semi-supervised Matching Framework Local Label Preference Cost Regional Surface Shape Cost Global Epipolar Geometry Cost Symmetric Visibility Consistency Cost
3
Iterative MV Optimization
4
Learning the Symmetric Affinity Matrix
5
More Details
6
Experiments
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 31 / 45
Learning the Symmetric Affinity Matrix
Directly define the matrix: no reliable approach for model selection if
Learn the matrix: more reliable and stable.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 32 / 45
Learning the Symmetric Affinity Matrix
Smooth manifold and linear re-constructable assumption for the manifold in image space. The label space and the image space share the same local linear reconstruction weights. Linear construction weight matrix Wp by minimizing the energy function EWp = ∑xp
i ∈X p Exp i ,
Exp
i =
i −
xp
j ∈N (xp i )
wp
ij xp j
=
xp
j ,xp k∈N (xp i )
wp
ij Gi jkwp ik.
where Gi
jk =
i −xp j
T xp
i −xp k
To avoid the undesirable contribution of negative weights, we enforce
xp
j ∈N (xp i )
wp
ij = 1, wp ij ≥ 0.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 33 / 45
Learning the Symmetric Affinity Matrix
min
Wp
xp
i ∈X p
xp
j ,xp k∈N (xp i )
wp
ij Gi jkwp ik +κ∑ ij
ij −wp ji
2 (1) s.t. ∀xp
i ∈ X p,
xp
j ∈N (xp i )
wp
ij = 1, wp ij ≥ 0.
where ∑ij
ij −wp ji
2 is a penalty term to encourage wp
ij and wp ji to be
similar.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 34 / 45
More Details
1
Introduction
2
Semi-supervised Matching Framework Local Label Preference Cost Regional Surface Shape Cost Global Epipolar Geometry Cost Symmetric Visibility Consistency Cost
3
Iterative MV Optimization
4
Learning the Symmetric Affinity Matrix
5
More Details
6
Experiments
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 35 / 45
More Details
1 For each image, the occlusion boundaries and depth ordering in the
scene are identified using [Hoiem et al. 2007].
2 SIFT key points and nearest neighbor algorithm to achieve an initial
matching.
3 Enforce the constraints of the matching to be one-to-one cross
consistency.
4 Discrete region growing [Kannala and Brandt 2007]. 5 Interpolate the unmatched part by estimating the local homography
transformation.
6 Obtain the initial visibility matrix. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 36 / 45
More Details
1 For each image, the occlusion boundaries and depth ordering in the
scene are identified using [Hoiem et al. 2007].
2 SIFT key points and nearest neighbor algorithm to achieve an initial
matching.
3 Enforce the constraints of the matching to be one-to-one cross
consistency.
4 Discrete region growing [Kannala and Brandt 2007]. 5 Interpolate the unmatched part by estimating the local homography
transformation.
6 Obtain the initial visibility matrix. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 36 / 45
More Details
1 For each image, the occlusion boundaries and depth ordering in the
scene are identified using [Hoiem et al. 2007].
2 SIFT key points and nearest neighbor algorithm to achieve an initial
matching.
3 Enforce the constraints of the matching to be one-to-one cross
consistency.
4 Discrete region growing [Kannala and Brandt 2007]. 5 Interpolate the unmatched part by estimating the local homography
transformation.
6 Obtain the initial visibility matrix. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 36 / 45
More Details
1 For each image, the occlusion boundaries and depth ordering in the
scene are identified using [Hoiem et al. 2007].
2 SIFT key points and nearest neighbor algorithm to achieve an initial
matching.
3 Enforce the constraints of the matching to be one-to-one cross
consistency.
4 Discrete region growing [Kannala and Brandt 2007]. 5 Interpolate the unmatched part by estimating the local homography
transformation.
6 Obtain the initial visibility matrix. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 36 / 45
More Details
1 For each image, the occlusion boundaries and depth ordering in the
scene are identified using [Hoiem et al. 2007].
2 SIFT key points and nearest neighbor algorithm to achieve an initial
matching.
3 Enforce the constraints of the matching to be one-to-one cross
consistency.
4 Discrete region growing [Kannala and Brandt 2007]. 5 Interpolate the unmatched part by estimating the local homography
transformation.
6 Obtain the initial visibility matrix. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 36 / 45
More Details
1 For each image, the occlusion boundaries and depth ordering in the
scene are identified using [Hoiem et al. 2007].
2 SIFT key points and nearest neighbor algorithm to achieve an initial
matching.
3 Enforce the constraints of the matching to be one-to-one cross
consistency.
4 Discrete region growing [Kannala and Brandt 2007]. 5 Interpolate the unmatched part by estimating the local homography
transformation.
6 Obtain the initial visibility matrix. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 36 / 45
More Details
Our algorithm works with some label data in a semi-supervised manner by the consistent cost Cl The local cost Cd just works in an axillary manner. Unlike the traditional unsupervised matching, our framework does not heavily rely on the similarity function ρp
i (y).
For efficient computation, we just sample some values for some integer combination of h and v to compute ρp
i (y) = exp(−xp
i −xq j 2
2σ2
). We normalized the largest sampled value to be 1, and then fit ρp
i (y)
with continuous and differentiable quadratic function, i.e. ρp
i (y) = (v −vo)2 +(h −ho)2
2σ2 , where (vo,ho) and σ are the center and spread of the parabola for xp
i .
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 37 / 45
More Details
1 Compute the depth and occlusion boundary image and feature vector. 2 Compute sparse matching by SIFT and the confidence penalty τ, then
interpolate the results from sparse matching with depth information to achieve an initial solution.
3 Learn the affinity matrix W1 and W2. 4 while (cost change between two iterations ≥ threshold): 1
Estimate the fundamental matrix F, and reject outliers to achieve a subset as label data,
2
Compute the parameters for the similarity cost function ρ and epipolar cost function d,
3
Estimate matching given visibility,
4
Compute the γ map,
5
Estimate visibility given matching.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 38 / 45
More Details
1 Compute the depth and occlusion boundary image and feature vector. 2 Compute sparse matching by SIFT and the confidence penalty τ, then
interpolate the results from sparse matching with depth information to achieve an initial solution.
3 Learn the affinity matrix W1 and W2. 4 while (cost change between two iterations ≥ threshold): 1
Estimate the fundamental matrix F, and reject outliers to achieve a subset as label data,
2
Compute the parameters for the similarity cost function ρ and epipolar cost function d,
3
Estimate matching given visibility,
4
Compute the γ map,
5
Estimate visibility given matching.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 38 / 45
More Details
1 Compute the depth and occlusion boundary image and feature vector. 2 Compute sparse matching by SIFT and the confidence penalty τ, then
interpolate the results from sparse matching with depth information to achieve an initial solution.
3 Learn the affinity matrix W1 and W2. 4 while (cost change between two iterations ≥ threshold): 1
Estimate the fundamental matrix F, and reject outliers to achieve a subset as label data,
2
Compute the parameters for the similarity cost function ρ and epipolar cost function d,
3
Estimate matching given visibility,
4
Compute the γ map,
5
Estimate visibility given matching.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 38 / 45
More Details
1 Compute the depth and occlusion boundary image and feature vector. 2 Compute sparse matching by SIFT and the confidence penalty τ, then
interpolate the results from sparse matching with depth information to achieve an initial solution.
3 Learn the affinity matrix W1 and W2. 4 while (cost change between two iterations ≥ threshold): 1
Estimate the fundamental matrix F, and reject outliers to achieve a subset as label data,
2
Compute the parameters for the similarity cost function ρ and epipolar cost function d,
3
Estimate matching given visibility,
4
Compute the γ map,
5
Estimate visibility given matching.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 38 / 45
More Details
1 Compute the depth and occlusion boundary image and feature vector. 2 Compute sparse matching by SIFT and the confidence penalty τ, then
interpolate the results from sparse matching with depth information to achieve an initial solution.
3 Learn the affinity matrix W1 and W2. 4 while (cost change between two iterations ≥ threshold): 1
Estimate the fundamental matrix F, and reject outliers to achieve a subset as label data,
2
Compute the parameters for the similarity cost function ρ and epipolar cost function d,
3
Estimate matching given visibility,
4
Compute the γ map,
5
Estimate visibility given matching.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 38 / 45
More Details
1 Compute the depth and occlusion boundary image and feature vector. 2 Compute sparse matching by SIFT and the confidence penalty τ, then
interpolate the results from sparse matching with depth information to achieve an initial solution.
3 Learn the affinity matrix W1 and W2. 4 while (cost change between two iterations ≥ threshold): 1
Estimate the fundamental matrix F, and reject outliers to achieve a subset as label data,
2
Compute the parameters for the similarity cost function ρ and epipolar cost function d,
3
Estimate matching given visibility,
4
Compute the γ map,
5
Estimate visibility given matching.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 38 / 45
More Details
1 Compute the depth and occlusion boundary image and feature vector. 2 Compute sparse matching by SIFT and the confidence penalty τ, then
interpolate the results from sparse matching with depth information to achieve an initial solution.
3 Learn the affinity matrix W1 and W2. 4 while (cost change between two iterations ≥ threshold): 1
Estimate the fundamental matrix F, and reject outliers to achieve a subset as label data,
2
Compute the parameters for the similarity cost function ρ and epipolar cost function d,
3
Estimate matching given visibility,
4
Compute the γ map,
5
Estimate visibility given matching.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 38 / 45
More Details
1 Compute the depth and occlusion boundary image and feature vector. 2 Compute sparse matching by SIFT and the confidence penalty τ, then
interpolate the results from sparse matching with depth information to achieve an initial solution.
3 Learn the affinity matrix W1 and W2. 4 while (cost change between two iterations ≥ threshold): 1
Estimate the fundamental matrix F, and reject outliers to achieve a subset as label data,
2
Compute the parameters for the similarity cost function ρ and epipolar cost function d,
3
Estimate matching given visibility,
4
Compute the γ map,
5
Estimate visibility given matching.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 38 / 45
More Details
1 Compute the depth and occlusion boundary image and feature vector. 2 Compute sparse matching by SIFT and the confidence penalty τ, then
interpolate the results from sparse matching with depth information to achieve an initial solution.
3 Learn the affinity matrix W1 and W2. 4 while (cost change between two iterations ≥ threshold): 1
Estimate the fundamental matrix F, and reject outliers to achieve a subset as label data,
2
Compute the parameters for the similarity cost function ρ and epipolar cost function d,
3
Estimate matching given visibility,
4
Compute the γ map,
5
Estimate visibility given matching.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 38 / 45
Experiments
1
Introduction
2
Semi-supervised Matching Framework Local Label Preference Cost Regional Surface Shape Cost Global Epipolar Geometry Cost Symmetric Visibility Consistency Cost
3
Iterative MV Optimization
4
Learning the Symmetric Affinity Matrix
5
More Details
6
Experiments
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 39 / 45
Experiments
Set the parameters to be more favor Cl and Cg for M-step, and Cv for the V-step, tune the parameters manually. The intensity value is set to be the norm of the label vector, i.e. y. For visualization, we scale the intensity to the range between 0 and 200, and only visible matching is shown, i.e. o 0.5.
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 40 / 45
Experiments
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 41 / 45
Experiments
1 1 ←
→ 2 and 2 ← → 3.
2 1 ←
→ 2 ← → 3.
3 Projective reconstruction [Quan 1995]. 4 Metric upgrade and bundle adjustment [Hartley and Zisserman 2004]. 5 Feature tracks with large reprojection errors are considered as outliers. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 42 / 45
Experiments
1 1 ←
→ 2 and 2 ← → 3.
2 1 ←
→ 2 ← → 3.
3 Projective reconstruction [Quan 1995]. 4 Metric upgrade and bundle adjustment [Hartley and Zisserman 2004]. 5 Feature tracks with large reprojection errors are considered as outliers. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 42 / 45
Experiments
1 1 ←
→ 2 and 2 ← → 3.
2 1 ←
→ 2 ← → 3.
3 Projective reconstruction [Quan 1995]. 4 Metric upgrade and bundle adjustment [Hartley and Zisserman 2004]. 5 Feature tracks with large reprojection errors are considered as outliers. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 42 / 45
Experiments
1 1 ←
→ 2 and 2 ← → 3.
2 1 ←
→ 2 ← → 3.
3 Projective reconstruction [Quan 1995]. 4 Metric upgrade and bundle adjustment [Hartley and Zisserman 2004]. 5 Feature tracks with large reprojection errors are considered as outliers. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 42 / 45
Experiments
1 1 ←
→ 2 and 2 ← → 3.
2 1 ←
→ 2 ← → 3.
3 Projective reconstruction [Quan 1995]. 4 Metric upgrade and bundle adjustment [Hartley and Zisserman 2004]. 5 Feature tracks with large reprojection errors are considered as outliers. Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 42 / 45
Experiments
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 43 / 45
Experiments
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 44 / 45
Q & A
Jianxiong Xiao et al. (HKUST) Learning Two-View Stereo Matching ECCV 2008 45 / 45