Exact Camera Location Recovery by Least Unsquared Deviations Gilad - - PowerPoint PPT Presentation

Exact Camera Location Recovery by Least Unsquared Deviations

Gilad Lerman

University of Minnesota Joint work with Yunpeng Shi (University of Minnesota) and Teng Zhang (University of Central Florida)

Gilad Lerman 1 / 23

Content

1 Introduction

Structure from motion Camera location recovery

2 Previous Works 3 New theoretical guarantees 4 Conclusion

Gilad Lerman 2 / 23

Structure from motion (SfM)

❼ Input: 2D images of the same object from different views ❼ Output: 3D structure of the object

Demonstration by Snavely et al. (2006)

Gilad Lerman 3 / 23

Pipeline of Structure from Motion

❼ Keypoint matching ❼ Essential matrix estimation ❼ Camera orientation estimation ❼ Camera location estimation

Gilad Lerman 4 / 23

Camera Location Recovery

❼ Input: Possibly corrupted pairwise directions between some cameras ❼ Output: Camera locations ❼ ❼ ❼ ❼

Gilad Lerman 5 / 23

Camera Location Recovery

❼ Input: Possibly corrupted pairwise directions between some cameras ❼ Output: Camera locations ❼ Example: Recovery of 4 locations from 5 uncorrupted pairwise

directions

❼ ❼ ❼

Gilad Lerman 5 / 23

Camera Location Recovery

❼ Input: Possibly corrupted pairwise directions between some cameras ❼ Output: Camera locations ❼ Example: Recovery of 4 locations from 5 uncorrupted pairwise

directions

❼ The problem is defined up to shift and scale ❼ ❼

Gilad Lerman 5 / 23

Camera Location Recovery

❼ Input: Possibly corrupted pairwise directions between some cameras ❼ Output: Camera locations ❼ Example: Recovery of 4 locations from 5 uncorrupted pairwise

directions

❼ The problem is defined up to shift and scale ❼ Unique solution (up to shift and scale) may not exist ❼ Graphs whose vertex locations are recoverable from edge directions

are called Parallel Rigid

Gilad Lerman 5 / 23

❼ Example of non-parallel rigid graphs:

Gilad Lerman 6 / 23

Generative Graph Model for Camera Location Recovery

❼ The HLV model is due to Hand, Lee and Voroniski (2015) ❼ It has parameters n ∈ N, 0 ≤ p ≤ 1 and 0 ≤ ǫb ≤ 1 ❼

Gilad Lerman 7 / 23

Generative Graph Model for Camera Location Recovery

❼ The HLV model is due to Hand, Lee and Voroniski (2015) ❼ It has parameters n ∈ N, 0 ≤ p ≤ 1 and 0 ≤ ǫb ≤ 1 ❼ Step 1: Generate the following graph with vertices V ∶= {t∗

i }n i=1 ⊆ R3

i.i.d. ∼ N(0,I)

Gilad Lerman 7 / 23

Generative Graph Model for Camera Location Recovery

❼ Step 2: The set of edges E are drawn i.i.d. from {ij ∶ 1 ≤ i ≠ j ≤ n}

with probability p (Erd¨

• s-R´

enyi graph)

Gilad Lerman 8 / 23

Generative Graph Model for Camera Location Recovery

❼ Step 3: For each ij ∈ E, assign the true pairwise direction

γij = γ∗

ij ∶=

t∗

i − t∗ j

∥t∗

i − t∗ j ∥

Gilad Lerman 9 / 23

Generative Graph Model for Camera Location Recovery

❼ Step 4: Corrupt the generated graph

▸ Pick a subgraph Gb(V,Eb) such that Eb ⊆ E and the maximal degree

• f Gb < ǫbn. The set of uncorrupted edges is Eg ∶= E ∖ Eb

▸ For all ij ∈ Eb, replace γij by arbitrary unit vector Gilad Lerman 10 / 23

Generative Graph Model for Camera Location Recovery

❼ (Optional) Step 5: Add the noise

▸ For all ij ∈ Eg, let γij =

γij+σvij ∥γij+σvij∥, where σ > 0 is noise level and vij

iid ∼ N(0,I)

Gilad Lerman 11 / 23

Least Squares Camera Location Solvers

❼ Least Squares Solver (M. Brand and et al., 2004):

min

{ti}n

i=1⊂R3 ∑

ij∈E

∥Pγ⊥

ij(ti − tj)∥2 s.t.

⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ ∑n

i=1 ∥ti∥2 = 1

∑n

i=1 ti = 0

, where Pγ⊥

ij denotes the orthogonal projection onto the orthogonal

complement of γij

❼ Constrained Least Squares Solver (Tron and Vidal, 2009):

min

{ti}n i=1⊂R3 {αij}ij∈E⊂R

∑

ij∈E

∥ti − tj − αijγij∥2 s.t. ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ αij ≥ 1 ∑n

i=1 ti = 0

Gilad Lerman 12 / 23

Current Robust Location Solvers

❼ LUD: Least Unsquared Deviations (Ozyesil and Singer, 2015):

min

{ti}n i=1⊂R3 {αij}ij∈E⊂R

∑

ij∈E

∥ti − tj − αijγij∥ s.t. ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ αij ≥ 1 ∑n

i=1 ti = 0

❼ ShapeFit (Hand, Lee and Voroninski, 2015):

min

{ti}n

i=1⊂R3 ∑

ij∈E

∥Pγ⊥

ij(ti − tj)∥ s.t.

⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ ∑ij∈E⟨ti − tj,γij⟩ = 1 ∑n

i=1 ti = 0

, where Pγ⊥

ij denotes the orthogonal projection onto the orthogonal

complement of γij

Gilad Lerman 13 / 23

Empirical performance of LUD and ShapeFit

❼ Performance of LUD and ShapeFit for synthetic data with corruption

and noise

Gilad Lerman 14 / 23

Theoretical Guarantees

Theorem 0 (Hand, Lee and Voroninski, 2015)

There exist absolute constants n0, C0 and C1 such that for n > n0 and for {t∗

i }n i=1 ⊆ R3, E ⊆ [n] × [n] and {γij}ij∈E ⊆ R3 generated by the HLV

model with parameters n, p and ǫb satisfying C0n−1/5 log3/5 n ≤ p ≤ 1 and ǫb ≤ C1p5/log3 n, ShapeFit recovers {t∗

i }n i=1 up to shift and scale with

probability 1 − 1/n4.

Gilad Lerman 15 / 23

Theoretical Guarantees

Theorem 0 (Hand, Lee and Voroninski, 2015)

There exist absolute constants n0, C0 and C1 such that for n > n0 and for {t∗

i }n i=1 ⊆ R3, E ⊆ [n] × [n] and {γij}ij∈E ⊆ R3 generated by the HLV

model with parameters n, p and ǫb satisfying C0n−1/5 log3/5 n ≤ p ≤ 1 and ǫb ≤ C1p5/log3 n, ShapeFit recovers {t∗

i }n i=1 up to shift and scale with

probability 1 − 1/n4.

Theorem 1 (L, Shi and Zhang, 2017)

There exist absolute constants n0, C0 and C1 such that for n > n0 and for {t∗

i }n i=1 ⊆ R3, E ⊆ [n] × [n] and {γij}ij∈E ⊆ R3 generated by the HLV

model with parameters n, p and ǫb satisfying C0n−1/3 log1/3 n ≤ p ≤ 1 and ǫb ≤ C1p7/3/log9/2 n, LUD recovers {t∗

i }n i=1 up to shift and scale with

probability 1 − 1/n4.

Gilad Lerman 15 / 23

Part1 of Proof: Reformulation of LUD

❼ Recall LUD:

({ˆ ti}n

i=1,{ˆ

αij}ij∈E) = arg min

{ti}n i=1⊂R3 {αij}ij∈E⊂R

∑

ij∈E

∥ti − tj − αijγij∥ s.t. ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ αij ≥ 1 ∑i ti = 0

❼ Expression for ˆ

αij in two complementary cases: Case 1: ⟨ˆ ti − ˆ tj,γij⟩ > 1 γij ˆ ti − ˆ tj ˆ αijγij Here ˆ αij = ⟨ˆ ti − ˆ tj,γij⟩ Case 2: ⟨ˆ ti − ˆ tj,γij⟩ ≤ 1 γij = ˆ αijγij ˆ ti − ˆ tj Here ˆ αij = 1

Gilad Lerman 16 / 23

Reformulation of LUD

❼ Recall LUD:

({ˆ ti}n

i=1,{ˆ

αij}ij∈E) = arg min

{ti}n i=1⊂R3 {αij}ij∈E⊂R

∑

ij∈E

∥ti − tj − αijγij∥ s.t. ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ αij ≥ 1 ∑i ti = 0

❼ Expression for ˆ

αij: ˆ αij = ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ ⟨ˆ ti − ˆ tj,γij⟩, if ⟨ˆ ti − ˆ tj,γij⟩ > 1; 1, if ⟨ˆ ti − ˆ tj,γij⟩ ≤ 1

❼ Reformulation:

{ˆ ti}n

i=1 =

min

{ti}n

i=1⊂R3 ∑

ij∈E

fij(ti ,tj) subject to ∑n

i=1 ti = 0,

where fij(ti,tj) = ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ ∥Pγ⊥

ij(ti − tj)∥,

if ⟨ti − tj,γij⟩ > 1; ∥ti − tj − γij∥, if ⟨ti − tj,γij⟩ ≤ 1

Gilad Lerman 17 / 23

Part 2 of the Proof: Optimality Condition

❼ WLOG, assume that {t∗

i }n i=1 is already centered at 0

❼ Goal: Show that under a certain condition any perturbation from the

ground truth {c∗t∗

i }n i=1 increases the value of the objective function,

where c∗ = arg min

c∈R

∑

ij∈E

fij(ct∗

i ,ct∗ j )

Gilad Lerman 18 / 23

Adaptation of ShapeFit Analysis

❼ Observation: On long edges, i.e., ∥t∗

i − t∗ j ∥ > 1 c∗ , the objective

functions of ShapeFit and LUD coincide for the ground truth solution {ˆ ti}n

i=1 = {c∗t∗ i }n i=1

❼ Define the set of good and long edges,

Egl = {ij ∈ Eg∣ ∥t∗

i − t∗ j ∥ > 1/c∗}, and its complement Ec gl = E ∖ Egl

❼ The analysis of ShapeFit with Eg and Ec

g is replaced with Egl and Ec gl

Gilad Lerman 19 / 23

Good-Long-Dominance Condition

Definition 1

V = {t∗

i }n i=1, E ⊆ [n] × [n] and {γij}ij∈E satisfy the good-long-dominance

condition if for any perturbation vectors {ǫi}n

i=1 ∈ R3 such that ∑n i=1 ǫi = 0

and ∑n

i=1⟨ǫi,t∗ i ⟩ = 0,

∑

ij∈Egl

∥Pγ∗⊥

ij (ǫi − ǫj)∥ ≥ ∑

ij∈Ec

gl

∥ǫi − ǫj∥.

Theorem 2

If V = {t∗

i }n i=1, E ⊆ [n] × [n] and {γij}ij∈E satisfy the

good-long-dominance condition, then LUD exactly recovers the ground truth solution up to shift and scale.

Gilad Lerman 20 / 23

Idea of Proof of Theorem 2

❼ Assume that after perturbing the solution from the ground truth

{c∗t∗

i }n i=1 to {c∗t∗ i + ǫi}n i=1, the value of each term fij in the objective

function is changed by ∆fij, then ∑

ij∈Egl

∆fij = ∑

ij∈Egl

∥Pγ∗⊥

ij (ǫi − ǫj)∥

∑

ij∈Ec

gl

∆fij ≥ − ∑

ij∈Ec

gl

∥ǫi − ǫj∥.

❼ Therefore, the good-long-dominance condition implies that after any

perturbation from the ground truth solution ∑

ij∈E

∆fij > 0. This concludes Theorem 2.

Gilad Lerman 21 / 23

Roadmap for the Proof

Gilad Lerman 22 / 23

Conclusion

❼ We established exact camera location recovery by LUD under a

probabilistic model

❼ In comparison to ShapeFit, the new guarantees for LUD hold for

higher corruption and sparser graphs

❼ Our constraint on the perturbations is more faithful to the structure

• f the problem than an implicit constraint in the analysis of ShapeFit

❼ There are other interesting strategies for exact camera recovery and

we believe that the new theory can be used to establish them

Gilad Lerman 23 / 23