[PPT] - Efficient & Robust LK for Mobile Vision Instructor - Simon PowerPoint Presentation

SLIDE 1

Efficient & Robust LK for Mobile Vision

Instructor - Simon Lucey

16-623 - Designing Computer Vision Apps

SLIDE 2

Direct Method (ours) Indirect Method (ORB+RANSAC)

H. Alismail, B. Browning, S. Lucey “Bit-Planes: Dense Subpixel Alignment of Binary Descriptors.” ACCV 2016.

SLIDE 3

Direct Method (ours) Indirect Method (ORB+RANSAC)

H. Alismail, B. Browning, S. Lucey “Bit-Planes: Dense Subpixel Alignment of Binary Descriptors.” ACCV 2016.

SLIDE 4

Direct Method (ours) Indirect Method (ORB+RANSAC)

H. Alismail, B. Browning, S. Lucey “Bit-Planes: Dense Subpixel Alignment of Binary Descriptors.” ACCV 2016.

SLIDE 5

Today

Review - LK Algorithm
Inverse Composition
Robust Extensions

SLIDE 6

Review - LK Algorithm

Lucas & Kanade (1981) realized this and proposed a

method for estimating warp displacement using the principles of gradients and spatial coherence.

Technique applies Taylor series approximation to any

spatially coherent area governed by the warp .

4

W(x; p)

I(p + ∆p) ≈ I(p) + ∂I(p) ∂pT ∆p

SLIDE 7

Review - LK Algorithm

Lucas & Kanade (1981) realized this and proposed a

method for estimating warp displacement using the principles of gradients and spatial coherence.

Technique applies Taylor series approximation to any

spatially coherent area governed by the warp .

5

W(x; p)

I(p + ∆p) ≈ I(p) + ∂I(p) ∂pT ∆p

SLIDE 8

Review - LK Algorithm

Lucas & Kanade (1981) realized this and proposed a

method for estimating warp displacement using the principles of gradients and spatial coherence.

Technique applies Taylor series approximation to any

spatially coherent area governed by the warp .

5

“We consider this image to always be static....”

W(x; p)

I(p + ∆p) ≈ I(p) + ∂I(p) ∂pT ∆p

SLIDE 9

Review - LK Algorithm

Lucas & Kanade (1981) realized this and proposed a

method for estimating warp displacement using the principles of gradients and spatial coherence.

Technique applies Taylor series approximation to any

spatially coherent area governed by the warp .

6

W(x; p)

I(p + ∆p) ≈ I(p) + ∂I(p) ∂pT ∆p

SLIDE 10

Review - LK Algorithm

Lucas & Kanade (1981) realized this and proposed a

method for estimating warp displacement using the principles of gradients and spatial coherence.

Technique applies Taylor series approximation to any

spatially coherent area governed by the warp .

7

W(x; p)

I(p + ∆p) ≈ I(p) + ∂I(p) ∂pT ∆p

∂I(p) ∂pT =    

∂I(x0

1)

∂x0T

1

. . . 0T . . . ... . . . 0T . . .

∂I(x0

N)

∂x0T

N

       

∂W(x1;p) ∂pT

. . .

∂W(xN;p) ∂pT

    x0 = W(x; p)

SLIDE 11

Reminder - Template Coordinates

8

W(x1; p)

“Template” “Source Image”

W(x1; 0) I T

SLIDE 12

Reminder - Template Coordinates

8

x1

“Template” “Source Image”

I T x0

1

SLIDE 13

Review - LK Algorithm

9

rxI ryI

∂I(p) ∂pT =    

∂I(x0

1)

∂x0T

1

. . . 0T . . . ... . . . 0T . . .

∂I(x0

N)

∂x0T

N

       

∂W(x1;p) ∂pT

. . .

∂W(xN;p) ∂pT

   

SLIDE 14

Which Way?

10

Step 1: Warp Image Step 2: Estimate Gradients

∗

“Horizontal” “Vertical”

I I(p) I(p) rxI(p) ryI(p)

SLIDE 15

Which Way?

10

Step 1: Warp Image Step 2: Estimate Gradients

∗

“Horizontal” “Vertical”

I I(p) I(p) ∂I(x0) ∂x

SLIDE 16

Which Way?

10

Step 1: Warp Image Step 2: Estimate Gradients

∗

“Horizontal” “Vertical”

I I(p) I(p) ∂I(x0) ∂x

SLIDE 17

Which Way?

11

Step 1: Estimate Gradients Step 2: Warp Gradients

I rxI(p) ryI(p)

∗

“Horizontal” “Vertical”

ryI

rxI

SLIDE 18

Which Way?

11

Step 1: Estimate Gradients Step 2: Warp Gradients

I

∗

“Horizontal” “Vertical”

ryI

rxI

∂I(x0) ∂x0

SLIDE 19

Review - LK Algorithm

12

∂I(p) ∂pT =    

∂I(x0

1)

∂x0T

1

. . . 0T . . . ... . . . 0T . . .

∂I(x0

N)

∂x0T

N

       

∂W(x1;p) ∂pT

. . .

∂W(xN;p) ∂pT

   

SLIDE 20

Deriving the

For an affine warp,

13

∂W(x; p) ∂pT W(x; p) = 1 − p1 p2 p3 p4 1 − p5 p6 2 4 x y 1 3 5 ∂W(x; p) ∂pT = −x y 1 x −y 1

SLIDE 21

Deriving the

For an homography warp,

14

∂W(x; p) ∂pT ∂W(x; p) ∂pT = −x y 1 x −y 1

W(x; p) =

1 p7x + p8y + p9 1 − p1 p2 p3 p4 1 − p5 p6 2 4 x y 1 3 5

SLIDE 22

N × 1 N = number of pixels

Review - LK Algorithm

Lucas & Kanade (1981) realized this and proposed a

method for estimating warp displacement using the principles of gradients and spatial coherence.

Technique applies Taylor series approximation to any

spatially coherent area governed by the warp .

15

N × 1

I(p + ∆p) ≈ I(p) + ∂I(p)T ∂p ∆p

W(x; p) P = number of warp parameters N × P

SLIDE 23

Often just refer to,

as the “Jacobian” matrix.

Also refer to,

as the “pseudo-Hessian”.

Finally, we can refer to,

as the “template”.

Review - LK Algorithm

16

T (0) = I(p + ∆p) JI = ∂I(p) ∂pT HI = JT

I JI

SLIDE 24

Actual algorithm is just the application of the following steps,

keep applying steps until converges.

LK Algorithm

17

Step 1: Step 2:

∆p p ← p + ∆p ∆p = H−1

I JT I [T (0) − I(p)]

SLIDE 25

Examples of LK Alignment

18

SLIDE 26

Examples of LK Alignment

18

SLIDE 27

Gauss-Newton Algorithm

LK is essentially an application of the Gauss-Newton

algorithm,

19

s.t. F : RN → RM arg min

∆x ||y − F(x) − ∂F(x)

∂xT ∆x||2

2

x ← x + ∆x

Step 1: Step 2:

keep applying steps until converges.

∆x

arg min

x ||y − F(x)||2 2

“Carl Friedrich Gauss” “Isaac Newton”

SLIDE 28

Optimization Interpretation

The optimization employed with the LK algorithm can be

interpreted as Gauss-Newton optimization.

Other non-linear least-squares optimization strategies have

been investigated (Baker et al. 2003)

Levenberg-Marquadt
Newton
Steepest-Descent
Gauss-Newton in empirical evaluations has appeared to be

the most robust (Baker et al.).

20

SLIDE 29

LK Event Horizon

Initial warp estimate has to be suitably close to the ground-

truth for gradient-search methods to work.

Kind of like a black hole’s event horizon.
You got to be inside it to be sucked in!

SLIDE 30

Expanding the Event Horizon

Best strategy is to expand the neighborhood across which

gradients are estimated.

Simplest to do this in practice with a blur.
Often apply what is known as “coarse-to-fine” alignment.

I(x + ∆x) − I(x)

∂I(x) ∂x N x ∈ N

SLIDE 31

Questions

Why is LK sensitive to the initial guess?
Why can’t you perform the optimization in a single shot?

SLIDE 32

Today

Review - LK Algorithm
Inverse Composition
Robust Extensions

SLIDE 33

?

Efficient search is essential on mobile and desktop!!

SLIDE 34

Computation Concerns

Unfortunately, the LK algorithm can be computationally

expensive.

Requires the re-computation of the Jacobian matrix at each iteration.
With the additional inversion of the Hessian matrix at each iteration.

26

“Is there any way we can pre-compute any of this?”

HI = JT

I JI

JI =    

∂I(x0

1)

∂x0T

1

. . . 0T . . . ... . . . 0T . . .

∂I(x0

N)

∂x0T

N

       

∂W(x1;p) ∂pT

. . .

∂W(xN;p) ∂pT

   

SLIDE 35

Linearizing the Template?

27

I(p + ∆p) ≈ I(p) + ∂I(p)T ∂p ∆p

SLIDE 36

Linearizing the Template?

27

T (0) ≈ I(p) + ∂I(p) ∂pT ∆p

“template”

SLIDE 37

Linearizing the Template?

27

T (0) ≈ I(p) + ∂I(p) ∂pT ∆p

“template”

T (0) + ∂T (0) ∂pT ∆p ≈ I(p)

“Why is this useful if the template must be static?”

SLIDE 38

Simple Example

arg min

∆p N

X

n=1

||W(xn; p) + ∂W(xn; p) ∂pT ∆p − W(xn; 0)||2

2

SLIDE 39

Simple Example

arg min

∆p N

X

n=1

||W(xn; p) + ∂W(xn; p) ∂pT ∆p − W(xn; 0)||2

2

x2 x3 x0

3

x1 x0

1

x0

2

where: x0 = W(x; p) x = W(x; 0) p + ∆p

SLIDE 40

Simple Example

arg min

∆p∗ N

X

n=1

||W(xn; p) − W(xn; 0) − ∂W(xn; 0) ∂pT ∆p∗||2

2

SLIDE 41

Simple Example

arg min

∆p∗ N

X

n=1

||W(xn; p) − W(xn; 0) − ∂W(xn; 0) ∂pT ∆p∗||2

2

x2 x3 x0

3

x1 x0

1

x0

2

where: x0 = W(x; p) x = W(x; 0) 0 + ∆p∗

SLIDE 42

Simple Example

arg min

∆p∗ N

X

n=1

||W(xn; p) − W(xn; 0) − ∂W(xn; 0) ∂pT ∆p∗||2

2

“Static”

x2 x3 x0

3

x1 x0

1

x0

2

where: x0 = W(x; p) x = W(x; 0) 0 + ∆p∗

SLIDE 43

In general one can show,

Relating and

30

∆p

∆p∗

W(x; p) = W(x; ∆p∗)

x2 x3 x0

3

x1 x0

1

x02 where: x0 = W(x; p) x = W(x; 0) 0 + ∆p∗

SLIDE 44

In general one can show,

Relating and

30

∆p

∆p∗

W(x; p) ≈ W(x; ∆p∗)

“for non-linear warp”

x2 x3 x0

3

x1 x0

1

x02 where: x0 = W(x; p) x = W(x; 0) 0 + ∆p∗

SLIDE 45

Specifically for an affine warp one can show,

Relating and

31

∆p

∆p∗

W(x; p) = W(x; ∆p∗)

SLIDE 46

Specifically for an affine warp one can show,

Relating and

31

∆p

∆p∗

˜ M(p)˜ x = ˜ M(∆p∗)˜ x

˜ M(p) =   1 − p1 p2 p3 p4 1 − p5 p6 1  

˜ x = x 1

where,

SLIDE 47

Specifically for an affine warp one can show,

Relating and

∆p

∆p∗

˜ M(∆p∗)−1 ˜ M(p)˜ x = ˜ x

x2 x3

x0

3

x1

x0

1

x0

2

where: x0 = W(x; p) x = W(x; 0) 0 + ∆p∗

SLIDE 48

Specifically for an affine warp one can show,

Relating and

∆p

∆p∗

˜ M(∆p∗)−1 ˜ M(p)˜ x = ˜ x

x2

x3 x03 x1 x0

1

x02 where: x0 = W(x; p) x = W(x; 0) p −1 ∆p∗

SLIDE 49

Specifically for an affine warp one can show,

Relating and

∆p

∆p∗

˜ M(∆p∗)−1 ˜ M(p)˜ x = ˜ x

also,

˜ M(p + ∆p)x = x

x2

x3 x03 x1 x0

1

x02 where: x0 = W(x; p) x = W(x; 0) p −1 ∆p∗

x2 x3

x03 x1 x01 x0

2

where: x0 = W(x; p) x = W(x; 0) p + ∆p

SLIDE 50

Specifically for an affine warp one can show,

Relating and

∆p

∆p∗

˜ M(∆p∗)−1 ˜ M(p)˜ x = ˜ x

also,

˜ M(p + ∆p)x = x

therefore,

˜ M(p + ∆p) = ˜ M(∆p∗)−1 ˜ M(p) = I

x2

x3 x03 x1 x0

1

x02 where: x0 = W(x; p) x = W(x; 0) p −1 ∆p∗

x2 x3

x03 x1 x01 x0

2

where: x0 = W(x; p) x = W(x; 0) p + ∆p

SLIDE 51

Extending the affine warp to the general form,

Relating and

33

∆p

∆p∗

˜ M(∆p∗)−1 ˜ M(p)˜ x = W(x; p −1 ∆p∗)

x2

x3

x0

3

x1 x0

1

x02 where: x0 = W(x; p) x = W(x; 0) p −1 ∆p∗

SLIDE 52

Inverse Composition Algorithm

34

Actual algorithm is just the application of the following steps,

keep applying steps until converges.

Step 1: Step 2:

arg min

∆p ||T (0) + ∂T (0)

∂p ∆p − I(p)||2

2

p ! p −1 ∆p ∆p

T

SLIDE 53

Inverse Composition Algorithm

35

Actual algorithm is just the application of the following steps,

keep applying steps until converges.

Step 1: Step 2:

p ! p −1 ∆p ∆p ∆p = H−1

T JT T [I(p) − T (0)]

JT = ∂T (0) ∂pT HT = JT

T JT

SLIDE 54

Inverse Composition Algorithm

35

Actual algorithm is just the application of the following steps,

keep applying steps until converges.

Step 1: Step 2:

p ! p −1 ∆p ∆p ∆p = H−1

T JT T [I(p) − T (0)]

“Static” “Inverse Composition”

JT = ∂T (0) ∂pT HT = JT

T JT

SLIDE 55

Linearizing the Template

36

∂T (0) ∂pT =    

∂T (x1) ∂xT

1

. . . 0T . . . ... . . . 0T . . .

∂T (xN) ∂xT

N

       

∂W(x1;0) ∂pT

. . .

∂W(xN;0) ∂pT

    ∂I(p) ∂pT =    

∂I{W(x1;p)} ∂W(x1;p)T

. . . 0T . . . ... . . . 0T . . .

∂I{W(xN;p)} ∂W(xN;p)T

       

∂W(x1;p) ∂pT

. . .

∂W(xN;p) ∂pT

   

“Constantly changing” “Static”

SLIDE 56

Linearizing the Template

36

∂T (0) ∂pT =    

∂T (x1) ∂xT

1

. . . 0T . . . ... . . . 0T . . .

∂T (xN) ∂xT

N

       

∂W(x1;0) ∂pT

. . .

∂W(xN;0) ∂pT

    ∂I(p) ∂pT =    

∂I{W(x1;p)} ∂W(x1;p)T

. . . 0T . . . ... . . . 0T . . .

∂I{W(xN;p)} ∂W(xN;p)T

       

∂W(x1;p) ∂pT

. . .

∂W(xN;p) ∂pT

   

“Constantly changing” “Static”

SLIDE 57

Rules for Compositional Warps

Not just applicable to affine warps, can be applied to any set
f warps that:
have an identity warp (i.e. )
set of warps must be closed under composition,

37

W(x; 0) = x W(W(x; p1); p2) ! W(x; p2 p1)

for example a homography,

Φ3 = Φ2Φ1

where,

det(Φ) = +1

SLIDE 58

Inverse Compositional Performance

38

Template area Intensity 75 × 57 650 150 × 115 360 300 × 230 140 640 × 460 45

Table 2. Planar template tracking runtime

(fps)

Planar template tracking runtime in frames per second on single core Intel Core i7 2.8 Ghz.

H. Alismail, B. Browning, S. Lucey “Bit-Planes: Dense Subpixel Alignment of Binary Descriptors.” ACCV 2016.

SLIDE 59

Today

Review - LK Algorithm
Inverse Composition
Robust Extensions

SLIDE 60

Robust Error Functions

Least squares criterion is

not robust to outliers (e.g.

cclusion, illumination)
For example, the two
utliers here cause the fitted

line to be quite wrong.

Robust error functions can

be helpful here.

SLIDE 61

Robust Error Functions

Least squares criterion is

not robust to outliers (e.g.

cclusion, illumination)
For example, the two
utliers here cause the fitted

line to be quite wrong.

Robust error functions can

be helpful here.

SLIDE 62

Robust Error Functions

arg min

∆p η{T (0) + ∂T (0)

∂pT ∆p − I(p)}

L2 L1

Robust Error

η{x}

x

for example,

η{x} = ||x||1 → L1 error

SLIDE 63

Robust Error Functions

Computational inefficiencies creep into inverse composition

when one departs from the classical L2 distance.

Much better results for fast tracking have been achieved

recently using robust representations (e.g. Bristow et al.).

SLIDE 64

Robust Representations

1. Compute image gradients
2. Pool into local histograms
3. Concatenate histograms
4. Normalize histograms

ψ → RN : RN×K

SLIDE 65

Robust IC Algorithm

44

Actual algorithm is just the application of the following steps,

keep applying steps until converges.

Step 1: Step 2:

p ! p −1 ∆p ∆p arg min

∆p ||ψ{T (0)} + ∂ψ{T (0)}

∂pT ∆p − ψ{I(p)}||2

2

SLIDE 66

Descriptor Performance

“Raw Pixels” “BitPlanes”

Template area Intensity BitPlanes 75 × 57 650 460 150 × 115 360 170 300 × 230 140 90 640 × 460 45 35

able 2. Planar template tracking runtime in frames

Planar template tracking runtime in frames per second on single core Intel Core i7 2.8 Ghz.

SLIDE 67

More to read…

Baker & Matthews, “Lucas-Kanade 20 Years On: A

Unifying Framework”, IJCV 2004.

Bristow & Lucey, “In Defense of Gradient-Based

Alignment on Densely Sampled Sparse Features”,

Springer Book on Dense Registration Methods 2015.

Baker, Datta & Kanade “Parameterizing Homographies”,

CMU Tech Report 06-11.

In Defense of Gradient-Based Alignment

n Densely Sampled Sparse Features

Hilton Bristow and Simon Lucey Abstract In this chapter, we explore the surprising result that gradient- based continuous optimization methods perform well for the alignment of image/object models when using densely sampled sparse features (HOG, dense SIFT, etc.). Gradient-based approaches for image/object alignment have many desirable properties – inference is typically fast and exact, and diverse constraints can be imposed on the motion of points. However, the pre- sumption that gradients predicted on sparse features would be poor estima- tors of the true descent direction has meant that gradient-based optimization is often overlooked in favour of graph-based optimization. We show that this intuition is only partly true: sparse features are indeed poor predictors of the error surface, but this has no impact on the actual alignment performance. In fact, for general object categories that exhibit large geometric and ap- pearance variation, sparse features are integral to achieving any convergence

whatsoever. How the descent directions are predicted becomes an important

consideration for these descriptors. We explore a number of strategies for estimating gradients, and show that estimating gradients via regression in a manner that explicitly handles outliers improves alignment performance sub-

stantially. To illustrate the general applicability of gradient-based methods

to the alignment of challenging object categories, we perform unsupervised ensemble alignment on a series of non-rigid animal classes from ImageNet. Hilton Bristow Queensland University of Technology, Australia. e-mail: hilton.bristow@gmail.com Simon Lucey The Robotics Institute, Carnegie Mellon University, USA. e-mail: slucey@cs.cmu.edu 1 Parameterizing Homographies CMU-RI-TR-06-11 Simon Baker, Ankur Datta, and Takeo Kanade The Robotics Institute Carnegie Mellon University 5000 Forbes Avenue Pittsburgh, PA 15213 The motion of a plane can be described by a homography. We study how to parameterize homographies to maximize plane estimation performance. We compare the usual 3 × 3 matrix parameterization with a parameterization that combines 4 fixed points in one of the images with 4 variable points in the other image. We empirically show that this 4pt parameterization is far

superior. We also compare both parameterizations with a variety of direct parameterizations. In

the case of unknown relative orientation, we compare with a direct parameterization of the plane equation, and the rotation and translation of the camera(s). We show that the direct parameterization is both less accurate and far less robust than the 4-point parameterization. We explain the poor performance using a measure of independence of the Jacobian images. In the fully calibrated setting, the direct parameterization just consists of 3 parameters of the plane equation. We show that this parameterization is far more robust than the 4-point parameterization, but only approxi- mately as accurate. In the case of a moving stereo rig we find that the direct parameterization of plane equation, camera rotation and translation performs very well, both in terms of accuracy and

robustness. This is in contrast to the corresponding direct parameterization in the case of unknown

relative orientation. Finally, we illustrate the use of plane estimation in 2 automotive applications.