image motion Thurs. Jan. 31, 2018 1 Time varying images (XYT) - PowerPoint PPT Presentation

COMP 546 Lecture 7 image motion Thurs. Jan. 31, 2018 1

Time varying images (XYT) 𝑧 𝑧 𝑦 𝑢 𝑢 𝑦 𝑦 2

Motion in XYT e.g. translating vertical edge 𝑧 𝑧 𝑦 𝑢 𝑢 𝑦 𝑦 3

Motion in XYT e.g. translating vertical bar 𝑧 𝑧 𝑦 𝑢 𝑢 𝑦 𝑦 4

𝑧 𝑦 𝑢 5

How do the eye and brain (retina, LGN, V1) measure image motion? 6

Photoreceptor response to a brief flash of light (recall from lecture 3) - 40 mV Response - 50 mV 0 100 time (ms) flash of light 7

Retinal Ganglion and LGN cells Our models up to now have been static only. - + + - + - ON center, - OFF center, + OFF surround ON surround + - + - - + 8

XY and XT slices through DOG(x,y,t) 𝑧 𝑢 - - - 𝑧 0 + - - - 𝑦 0 𝑦 0 𝑦 𝑦 + - - The response at t = 0 depends on the image intensities in the past (t < 0). 9

Space-time separable model 𝑕 𝑦, 𝑧, 𝑢 = 𝐸𝑃𝐻 𝑦, 𝑧 𝑔(𝑢) 𝑧 𝑢 𝑢 - - - 𝑧 0 + - - - 𝑦 0 𝑦 0 𝑦 0 𝑦 𝑦 𝑦 + - - The response at t = 0 depends on the image intensities in the past (t < 0). 10

This cell would respond better to the static image. 𝑢 𝑢 𝑢 𝑦 0 𝑦 0 𝑦 0 𝑦 𝑦 𝑦 + - - moving static 11

Space-time separable model (with temporal sensitivity) 𝑧 𝑢 𝑢 - - - 𝑧 0 + - - - 𝑦 0 𝑦 0 𝑦 0 𝑦 𝑦 𝑦 - + - + + - 12

This cell would respond better to motion than to static image. 𝑢 𝑢 𝑢 𝑢 𝑦 0 𝑦 0 𝑦 0 𝑦 0 𝑦 + - - + + - moving moving static 13

This cell would respond This cell would respond better to slow motion. better to fast motion. 𝑢 𝑢 𝑦 0 𝑦 0 𝑦 𝑦 + - - + + + - - - + + - 14

For your interest... (not on exam) There are two classes of retinal ganglion cells... Sensitive to temporal intensity variations. Large receptive fields. Insensitive to temporal intensity variations. Small receptive fields. Sensitive to color. 15

… and these two classes map to distinct layers in the LGN. LGN Layers 3-6 L R L R R L Layers 1, 2 16

V1 cells can detect oriented structure in XY. What about XT and YT ? cosGabor(x,y) sinGabor(x,y) 17

Hubel and Wiesel’s idea for a simple cell (line detector). - - + - - - - - + - - - - - 𝑧 + - - - - - + - - - 𝑦 18

Reichardt (1950’s): Hubel and Wiesel’s idea for Temporal delays a simple cell (line detector). combined with spatial shifts could produce - - motion direction + - - - - sensitivity. - + - - - - - 𝑧 + 𝑢 - - - - - + - - - 𝑦 0 𝑦 𝑦 𝑦 0 - - + - - + - - + 19

𝑢 𝑦 0 𝑦 + - - Alternatively, using time + + - varying DOGs: - + + - - - + - - + + - - - + - + + + - 20

Orientation and direction tuning in V1 + - - + - 𝑧 - 𝑢 + 𝑦 0 𝑦 𝑦 𝑦 0 Cell is selective for vertically line moving to the left. 21

XT slice through receptive field profile of V1 cell 𝑢 𝑦 0 𝑦 22

Simple and Complex Cells are Orientation Tuned (XY slice) and many are binocularly disparity tuned. 23

Many are also motion direction and speed tuned. 24

Vertical edge moving to the right 𝑧 𝑧 𝑦 𝑢 𝑢 𝑦 𝑦 25

Vertical bar moving to the right 𝑧 𝑧 𝑦 𝑢 𝑢 𝑦 𝑦 26

Vertically 2D sine moving to the right (wave) 𝑧 𝑧 𝑦 𝑢 𝑢 𝑦 𝑦 27

Recall: 2D sine sin 2𝜌 𝑂 (𝑙 𝑦 𝑦 + 𝑙 𝑧 𝑧) 𝑓. 𝑕. 𝑙 𝑦 = 8 𝑙 𝑧 = 2 𝑂 = 256 28

sine wave in XYT sin 2𝜌 𝑂 (𝑙 𝑦 𝑦 + 𝑙 𝑧 𝑧) + 2𝜌 𝑈 𝜕 𝑢 Temporal frequency (cycles per 𝑈 frames) Spatial frequency (cycles per 𝑂 pixels) Exercise: what is the speed of the wave? 29

3D sine in XYT sin 2𝜌 𝑂 (𝑙 𝑦 𝑦 + 𝑙 𝑧 𝑧) + 2𝜌 𝑈 𝜕 𝑢 2𝜌 2𝜌 𝑂 (𝑙 𝑦 𝑦 + 𝑙 𝑧 𝑧) + 𝑈 𝜕 𝑢 = c is the equation of a plane in XYT. 2𝜌 2𝜌 2𝜌 ( 𝑂 𝑙 𝑦 , 𝑂 𝑙 𝑧 , 𝑈 𝜕) ∙ ( 𝑦, 𝑧, 𝑢) = c 3D vector normal to the plane 30

3D sine Gabor 𝑧 𝑦 𝑢 2𝜌 2𝜌 sin 𝑂 (𝑙 𝑦 𝑦 + 𝑙 𝑧 𝑧) + 𝑈 𝜕 𝑢 𝐻(𝑦, 𝑧, 𝑢, 𝜏 𝑦 , 𝜏 𝑧 , 𝜏 𝑢 ) 31

Normal Velocity V1 cells only respond to the motion component that is normal (perpendicular) to their preferred orientation. 32

V1 : “Aperture Problem” http://www.opticalillusion.net/optical-illusions/the-barber-pole-illusion/ The same issue arises with single bar, edge, or constant gradient. 33

To estimating image velocity (𝑤 𝑦 , 𝑤 𝑧 ) at 𝑦, 𝑧 , the visual system needs to combine the responses of many V1 (normal velocity) cells . ( 𝑦, 𝑧) (𝑤 𝑦 , 𝑤 𝑧 ) 34