image motion 2 Tues. Feb. 6, 2018 1 Overview of Today Abstract - - PowerPoint PPT Presentation

1

COMP 546

Lecture 8

image motion 2

• Tues. Feb. 6, 2018

Overview of Today

• Abstract computational problem:

how to estimate the local image velocity ?

• Solution based on V1 motion detectors

2

Intensity changes in XY

(

𝜖 𝜖𝑦 𝐽 𝑦, 𝑧 , 𝜖 𝜖𝑧 𝐽 𝑦, 𝑧 )

is the 2D spatial gradient at 𝑦, 𝑧 .

𝑦 𝑧 +

• +

+ +

• +

+

3

∇ 𝐽

Intensity changes in XYT

(

𝜖 𝜖𝑦 𝐽 𝑦, 𝑧, 𝑢 , 𝜖 𝜖𝑧 𝐽 𝑦, 𝑧, 𝑢 , 𝜖 𝜖𝑢 𝐽 𝑦, 𝑧, 𝑢 )

is the 3D spatio-temporal gradient at 𝑦, 𝑧, 𝑢 .

𝑧 𝑢 𝑦

4

∇ 𝐽

Suppose that image “objects” are “moving”. (What do these terms mean?)

(𝑦, 𝑧, 𝑢) 𝑤𝑦, 𝑤𝑧

5

Intensity conservation

𝐽 𝑦, 𝑧, 𝑢 = 𝐽 𝑦 + 𝑤𝑦 ∆𝑢, 𝑧 + 𝑤𝑧∆𝑢, 𝑢 + ∆𝑢

Assume a moving point’s intensity doesn’t change over time. This is similar to the assumption made last lecture, namely that the left and right images are related by a (disparity) shift.

6

𝐽 𝑦 + 𝑤𝑦 ∆𝑢, 𝑧 + 𝑤𝑧∆𝑢, 𝑢 + ∆𝑢 = 𝐽 𝑦, 𝑧, 𝑢 +

𝜖 𝜖𝑦 𝐽 𝑦, 𝑧, 𝑢

𝑤𝑦∆𝑢 +

𝜖 𝜖𝑧 𝐽 𝑦, 𝑧, 𝑢

𝑤𝑧∆𝑢 +

𝜖 𝜖𝑢 𝐽 𝑦, 𝑧, 𝑢

∆𝑢 + H.O.T.

Taylor series

7

8

𝐽 𝑦 + ∆𝑦 ≈ 𝐽 𝑦 + 𝜖𝐽 𝜖𝑦 ∆𝑦 𝑦 𝑦 + ∆𝑦

𝜖𝐽 𝜖𝑦

We can estimate by taking

𝐽 𝑦 + 1 − 𝐽(𝑦 − 1) 2 .

𝐽 𝑦 + 𝑤𝑦 ∆𝑢, 𝑧 + 𝑤𝑧∆𝑢, 𝑢 + ∆𝑢 = 𝐽 𝑦, 𝑧, 𝑢 +

𝜖 𝜖𝑦 𝐽 𝑦, 𝑧, 𝑢

𝑤𝑦∆𝑢 +

𝜖 𝜖𝑧 𝐽 𝑦, 𝑧, 𝑢

𝑤𝑧∆𝑢 +

𝜖 𝜖𝑢 𝐽 𝑦, 𝑧, 𝑢

∆𝑢 + H.O.T.

≈ 0

9

𝐝𝐛𝐨𝐝𝐟𝐦 𝐜𝐟𝐝𝐛𝐯𝐭𝐟 𝐩𝐠 𝐣𝐨𝐮𝐟𝐨𝐭𝐣𝐮𝐳 𝐝𝐩𝐨𝐭𝐟𝐬𝐰𝐛𝐮𝐣𝐩𝐨

“Motion Constraint Equation”

𝜖 𝐽 𝜖𝑦

𝑤𝑦 +

𝜖 𝐽 𝜖𝑧 𝑤𝑧 + 𝜖 𝐽 𝜖𝑢 = 0

10

( 𝜖 𝐽

𝜖𝑦 , 𝜖 𝐽 𝜖𝑧 , 𝜖 𝐽 𝜖𝑢 ) ∙ (

𝑤𝑦, 𝑤𝑧, 1 ) = 0

Previous slides (and assumptions) imply:

𝜖 𝐽 𝜖𝑦

𝑤𝑦 + 𝜖 𝐽

𝜖𝑧 𝑤𝑧 + 𝜖 𝐽 𝜖𝑢 = 0

𝑤𝑦 𝑤𝑧

Motion constraint equation is a line in velocity space

11

𝜖 𝐽 𝜖𝑦

𝑤𝑦 +

𝜖 𝐽 𝜖𝑧 𝑤𝑧 + 𝜖 𝐽 𝜖𝑢 = 0

𝑤𝑦 𝑤𝑧 But many velocities satisfy this equation.

12

𝑧 𝑢 𝑦

The black vector in the figure below is the 3D image gradient. It is perpendicular to the gray plane. The red vectors are perpendicular to the 3D image gradient (see equation) and thus the red vectors lie in the grey plane.

13

(

𝜖 𝐽 𝜖𝑦 , 𝜖 𝐽 𝜖𝑧 , 𝜖 𝐽 𝜖𝑢 ) ∙ (

𝑤𝑦, 𝑤𝑧, 1 ) = 0

𝜖 𝐽 𝜖𝑦

𝑤𝑦 +

𝜖 𝐽 𝜖𝑧 𝑤𝑧 + 𝜖 𝐽 𝜖𝑢 = 0

“Normal velocity” is the velocity component in the direction of the XY gradient.

Normal velocity

14

𝑤𝑦 𝑤𝑧

“Aperture Problem”

The same issue arises with any 1D pattern e.g. single bar, edge, constant gradient.

15

Solution of Aperture Problem

We need more than one line or edge (or gradient

• rientation).

Could be in the same aperture or neighboring aperture.

normal velocities shown

16

“Intersection of Constraints” (IOC)

𝑤𝑦 𝑤𝑧

Suppose two nearby points have two different spatial gradients and the same image velocity.

normal velocities shown

17

𝜖 𝐽 𝜖𝑦 (𝑦1, 𝑧1, 𝑢) 𝑤𝑦 + 𝜖 𝐽 𝜖𝑧 (𝑦1, 𝑧1, 𝑢) 𝑤𝑧 + 𝜖 𝐽 𝜖𝑢 (𝑦1, 𝑧1, 𝑢) = 0

Intersection of Constraints (IOC)

𝑤𝑦 𝑤𝑧

𝜖 𝐽 𝜖𝑦 (𝑦2, 𝑧2, 𝑢) 𝑤𝑦 + 𝜖 𝐽 𝜖𝑧 (𝑦2, 𝑧2, 𝑢) 𝑤𝑧 + 𝜖 𝐽 𝜖𝑢 (𝑦2, 𝑧2, 𝑢) = 0

IOC gives a unique solution.

18

Another Example (counterintuitive)

𝑤𝑦 𝑤𝑧

19

Overview of Today

• Abstract computational problem:

how to estimate the local image velocity ?

• Solution based on V1 motion detectors

20

Recall: XYT Gabor

21

sin 2𝜌 𝑂 (𝑙0 𝑦 + 𝑙1 𝑧) + 2𝜌 𝑈 𝜕 𝑢

𝑦 𝑧 𝑢

sin

2𝜌 𝑂 (𝑙0 𝑦 + 𝑙1 𝑧) + 2𝜌 𝑈 𝜕 𝑢

𝐻(𝑦, 𝑧, 𝑢, 𝜏𝑦, 𝜏𝑧, 𝜏𝑢)

𝑦 𝑢 𝑧 𝑦 𝑢 𝑧

cos

2𝜌 𝑂 (𝑙0 𝑦 + 𝑙1 𝑧) + 2𝜌 𝑈 𝜕 𝑢

𝐻(𝑦, 𝑧, 𝑢, 𝜏𝑦, 𝜏𝑧, 𝜏𝑢)

(

𝜖 𝜖𝑦 𝐽, 𝜖 𝜖𝑧 𝐽, 𝜖 𝜖𝑢 𝐽 )

𝑧 𝑢 𝑦

22

(2𝜌 𝑂 𝑙0 , 2𝜌 𝑂 𝑙1 ,

2𝜌 𝑈 𝜕 )

We should get maximum response from a Gabor whose frequency (𝑙0 , 𝑙1 , 𝜕 ) is parallel to intensity gradient.

Each motion sensitive V1 cell is sensitive only to the 1D motion component perpendicular to its orientation.

23

24

Suppose true motion vector 𝑤𝑦, 𝑤𝑧 is toward the right and the image contains lots of oriented structure in each local neighborhood. Which V1 motion cells would have a large response ? 𝑤𝑦 𝑤𝑧

25

Suppose true motion vector 𝑤𝑦, 𝑤𝑧 is toward the right and the image contains lots of oriented structure in each local neighborhood. Which V1 motion cells would have a large response ? 𝑤𝑦 𝑤𝑧

26

Suppose true motion vector 𝑤𝑦, 𝑤𝑧 is toward the right and the image contains lots of oriented structure in each local neighborhood. Which V1 motion cells would have a large response ? 𝑤𝑦 𝑤𝑧

27

Suppose true motion vector 𝑤𝑦, 𝑤𝑧 is toward the right and the image contains lots of oriented structure in each local neighborhood. Which V1 motion cells would have a large response ? 𝑤𝑦 𝑤𝑧

Motion pathway in the brain

28

(contains V1) (temporal lobe contains motion area MT and MST : “middle temporal” & “medial superior temporal”)

MST ⟵ MT ⟵V1

(next lecture) (today)

V1 → MT

29

MT cells receive inputs from orientation/motion tuned V1 cells. Many MT cells are velocity tuned. They require multiple

• rientations in their

receptive field (to avoid the aperture problem). 𝑤𝑦 𝑤𝑧

V1 → MT

30

𝑤𝑦 𝑤𝑧 MT cells receive inputs from orientation/motion tuned V1 cells. Many MT cells are velocity tuned. They require multiple

• rientations in their

receptive field (to avoid the aperture problem).

V1 → MT

31

𝑤𝑦 𝑤𝑧 For any velocity vector, there is a family of

• rientation/motion tuned

V1 cells that would respond well, provided the moving image contains the spatial

• rientaiton that this cell is

sensitive to.

32

For full version of the MT model, see [Simoncelli and Heeger Vision Research 1998]

Gabors can have different sizes and spatial frequencies.

dot pattern and velocity

33

Random dot patterns (such as above) contain oriented structure at all frequencies. Later in the course when we discuss linear systems and Fourier transforms, you will learn a more formal (mathematical) statement of what this means.