egomotion: translation & rotation, eye movements Thurs. Feb. 8, - PowerPoint PPT Presentation

COMP 546 Lecture 9 egomotion: translation & rotation, eye movements Thurs. Feb. 8, 2018 1

What is the image motion seen by a moving observer? (“ egomotion ”) Translation Rotation 2

Motion field seen by moving observer For each image location (𝑦, 𝑧) , there is a velocity (𝑤 𝑦 , 𝑤 𝑧 ) . The Yosemite sequence (Last lecture, I discussed how to estimate image velocities.) 3

Motion field seen by a translating observer 𝑌 0 , 𝑍 0 , 𝑎 0 (𝑈 𝑌 , 𝑈 𝑍 , 𝑈 𝑎 ) 𝑎 𝑌 𝑍 eye 4

𝑌 0 , 𝑍 0 , 𝑎 0 (𝑈 𝑌 , 𝑈 𝑍 , 𝑈 𝑎 ) 𝑎 𝑌 𝑍 eye The path of the scene point in the eye’s coordinate system is: 𝑌 𝑢 , 𝑍 𝑢 , 𝑎(𝑢) = 𝑌 0 − 𝑈 𝑦 𝑢, 𝑍 0 − 𝑈 𝑧 𝑢, 𝑎 0 − 𝑈 𝑨 𝑢 The relative 3D velocity of the scene point (−𝑈 𝑌 , −𝑈 𝑍 , −𝑈 𝑎 ) 5

What is the image path of the scene point? 𝑦 𝑢 = 𝑌(𝑢) (𝑌 0 , 𝑎 0 ) 𝑔 𝑎(𝑢) 𝑎 (𝑈 𝑌 , 𝑈 𝑎 ) = 𝑌 𝑝 − 𝑈 𝑌 𝑢 𝑎 𝑝 − 𝑈 𝑎 𝑢 𝑎 = 𝑔 𝑌 Notation: here 𝑦 𝑢 is a position in the plane 𝑎 = 𝑔. 6

What is the image velocity of the scene point? 𝑒 𝑦 𝑢 (𝑌 0 , 𝑎 0 ) 𝑤 𝑦 = | 𝑢=0 𝑒𝑢 𝑔 𝑎 − 𝑈 𝑌 𝑎 0 + 𝑈 𝑎 𝑌 0 (𝑈 𝑌 , 𝑈 𝑎 ) = 2 𝑎 𝑝 𝑤 𝑦 𝑎 = 𝑔 𝑌 Notation: here 𝑤 𝑦 𝑢 is an angular velocity (radians/sec, assuming small angle 7 approximation) rather than a velocity in the plane 𝑎 = 𝑔.

− 𝑈 𝑌 𝑎 0 + 𝑈 𝑎 𝑌 0 , − 𝑈 𝑍 𝑎 0 + 𝑈 𝑎 𝑍 0 (𝑤 𝑦 , 𝑤 𝑧 ) = 2 2 𝑎 𝑝 𝑎 𝑝 = 1 𝑈 𝑎 𝑌 𝑝 , 𝑍 0 (−𝑈 𝑌 , −𝑈 𝑍 ) + 𝑎 𝑝 𝑎 𝑝 𝑎 𝑝 𝑎 𝑝 𝑦 𝑔 , 𝑧 𝑔 Lateral component Forward component 8

Lateral Component ( 𝑈 𝑎 = 0 ) 𝑤 𝑦 , 𝑤 𝑧 = 1 (−𝑈 𝑌 , −𝑈 𝑍 ) 𝑎 𝑝 Example: (𝑈 𝑌 , 𝑈 𝑍 ) wall ( 𝑎 = 10) , square ( 𝑎 = 3) 9 eccentricity (deg.)

Lateral Component ( 𝑈 𝑎 = 0 ) 𝑤 𝑦 , 𝑤 𝑧 = 1 (−𝑈 𝑌 , −𝑈 𝑍 ) 𝑎 𝑝 Example: ground plane (𝑈 𝑌 , 𝑈 𝑍 ) 𝑎 = − 𝑔ℎ 𝑧 10 eccentricity (deg.)

Lateral Motion and Balance Holding this pose is more difficult when looking up than when looking down. Why? 11

Dizziness (‘height vertigo’) Not to be confused with a more general ‘acrophobia’ (fear of heights) 12

Forward Component ( 𝑈 𝑌 = 𝑈 𝑍 = 0 ) 𝑈 𝑎 𝑦 𝑔 , 𝑧 𝑤 𝑦 , 𝑤 𝑧 = 𝑎 𝑝 𝑔 What does image flow depend on ? 13

Forward Component ( 𝑈 𝑌 = 𝑈 𝑍 = 0 ) 𝑈 𝑎 𝑦 𝑔 , 𝑧 𝑤 𝑦 , 𝑤 𝑧 = 𝑎 𝑝 𝑔 wall 14 𝑎 𝑝 = constant.

Forward Component ( 𝑈 𝑌 = 𝑈 𝑍 = 0 ) 𝑈 𝑎 𝑦 𝑔 , 𝑧 𝑤 𝑦 , 𝑤 𝑧 = 𝑎 𝑝 𝑔 ground plane wall (see Exercises) 𝑎 𝑝 = constant 15

What does a pilot see when approaching the runway? (from JJ Gibson 1950) 16

General Translation (when 𝑈 𝑨 ≠ 0 ) 1 𝑈 𝑎 𝑦 𝑔 , 𝑧 𝑤 𝑦 , 𝑤 𝑧 = + (−𝑈 𝑌 , −𝑈 𝑍 ) 𝑎 𝑝 𝑔 𝑎 𝑝 forward lateral 𝑔 − 𝑈 𝑈 𝑎 𝑔 − 𝑈 𝑦 𝑧 𝑧 𝑦 = , 𝑎 𝑝 𝑈 𝑈 𝑨 𝑨 17

General Translation (when 𝑈 𝑨 ≠ 0 ) 1 𝑈 𝑎 𝑦 𝑔 , 𝑧 𝑤 𝑦 , 𝑤 𝑧 = + (−𝑈 𝑌 , −𝑈 𝑍 ) 𝑎 𝑝 𝑔 𝑎 𝑝 forward lateral 𝑔 − 𝑈 𝑈 𝑎 𝑔 − 𝑈 𝑦 𝑧 𝑧 𝑦 = , 𝑎 𝑝 𝑈 𝑈 𝑨 𝑨 , 𝑈 𝑈 𝑧 𝑦 is called the “direction of heading”. 𝑈 𝑈 𝑨 𝑨 18

Example: 𝑈 𝑌 , 𝑈 𝑍 , 𝑈 𝑎 = (.3, 0, 1) Eccentricity (deg) 19

How can a translating observer estimate heading? 1) Estimate motion field 𝑤 𝑦 , 𝑤 𝑧 . 𝑈 𝑈 𝑧 𝑦 2) Estimate the direction of heading 𝑨 , to be the image 𝑈 𝑈 𝑨 point where all motion vectors point away from that point. 20

How/where does the brain solve this problem? V1: measure normal velocity components with Gabors  MT (middle temporal lobe): estimate velocities 𝑤 𝑦 , 𝑤 𝑧  MST (medial superior temporal lobe): estimate global motion field V1 21

MST cell ‘templates’ (computation model) Huge receptive fields Each vector represents a normal component of velocity (V1 cell). Each disk represents an MT cell (last lecture). [Perrone & Stone, 1998] 22

COMP 546 Lecture 9 egomotion: translation & rotation (eye movements) Thurs. Feb. 8, 2018 23

Motion field seen by a rotating observer ? 24

25

26

27

28

Thalamus (where LGN is) Oculomotor nerve … and its various branches Mid-brain All eye movement motor (output) signals come from mid-brain e.g. oculomotor nerves. These nerves also control accommodation, blinks, pupil contraction. 29

Types of eye movements • vestibulo-ocular reflex (VOR) • smooth pursuit • vergence (next lecture) • saccades (later in course) • OKN (optokinetic nystagmus) OMIT 30

VOR (eye rotations due to head movement) 31

Vestibular System (in the inner ear) 32

Vestibular system: the brain’s IMU ( inertial measurement unit – a term used in robotics ) It measures: • linear acceleration of head 𝑒 𝑈 𝑌 , 𝑈 𝑍 , 𝑈 𝑨 𝑒𝑢 • angular acceleration of head 𝑒 𝑒𝑢 (pan, tilt, roll) 33

otoliths Translation Rotation (linear acceleration) (angular acceleration) 34

Vestibular System (in the inner ear) Translation Rotation Hearing 35

Smooth Pursuit Eye Movements Tracking a static object as the observer moves (or smoothly tracking a moving object) Reduces retinal motion of object to 0. 36

Combined translation and rotation Zero image velocity Q: where is direction of (tracking) heading? translation only rotation only translation + rotation 37

Combined translation and rotation Direction of heading Zero image velocity (tracking) translation only rotation only translation + rotation 38

Discussion/Summary • Motion field seen by moving observer is the sum of translation & rotation fields • Cells in MST are sensitive to global motion fields (huge receptive fields) and are believed to be involved in estimating egomotion. [It is not entirely clear what exactly the computational role of these cells is. But there is a lot of evidence that these cells exist, and that they do play a role in estimating heading direction.] • VOR adds eye rotation to cancel the image motion that is due to head motion. Smooth pursuit adds eye rotation that reduces the retinal velocity of certain “interest points” to zero, allowing a detailed “still” image analysis. In both cases, these rotations are known (controlled by the brain) so their effects can be accounted for, i.e. the translation and rotation components of the motion field can be disentangled. 39