NYU/CNS Center for Neural Science The Electric Monk was a labor - PowerPoint PPT Presentation

The elements of early vision or, what vision ( and this course ) is all about NYU/CNS Center for Neural Science

The Electric Monk was a labor saving device, like a dishwasher or a video recorder. Dishwashers washed tedious dishes for you, thus saving you the bother of washing them yourself, video recorders watched tedious television for you, thus saving you the bother of looking at it yourself; Electric Monks believed things for you, thus saving you what was becoming an increasingly onerous task, that of believing all the things the world expected you to believe.

Theory What’s in the image? The task of early vision How do neurons encode visual information? How are neuronal representations decoded? y P g s o y l o c i h s o y p h p h y o s r u i c e s N

The challenge of scale after Churchland and Sejnowski, 1988

The challenge of scale 1000000 100000 0.0001 10000 0.001 1000 0.01 100 0.1 10 1 1000 PET imaging 1000 Brain EEG and MEG Lobe 100 100 fMRI Map TMS VSD 10 10 imaging imaging Brain Nucleus 2-DG lesions Microstimulation Size (mm) 1 1 imaging Optogenetics Layer 0.1 0.1 Light Field potentials microscopy Neuron Single units 0.01 0.01 Dendrite Patch clamp 0.001 0.001 Calcium imaging Synapse Electron microscopy 0.0001 0.0001 0.0001 0.001 0.01 0.1 1 10 100 1000 10000 100000 1000000 Time (s) Millisecond Second Minute Hour Day Month Sejnowski, Churchland & Movshon ( 2014 ) after Churchland & Sejnowski ( 1988 )

The challenge of level David Courtnay Marr ( 1946 - 1980 ) Marr, 1982

Theory What’s in the image? The task of early vision How do neurons encode visual information? How are neuronal representations decoded? y P g s o y l o c i h s o y p h p h y o s r u i c e s N

The Plenoptic Function and the Elements of Early Vision Edward H. Adelson and James R. Bergen In M. Landy and J. A. Movshon (eds), Computational Models of Visual Processing (pp. 3-20). Cambridge, MA: MIT Press (1991). Every body in light and shade fills the surrounding air with infinite images of itself; and these, by infinite pyramids diffused in the air, represent this body throughout space and on every side. Each pyramid that is composed of a long assemblage of rays includes within itself an infinite number of pyramids and each has the same power as all, and all as each. – The Notebooks of Leonardo da Vinci

The plenoptic function describes all the information available to an observer anywhere in space and time What is there to see? Black and white photo: x, y Black and white movie: x, y, t Color movie: x, y, t, λ Holographic movie: x, y, t, λ , V x , V y , V z

Practical applications IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 14, NO. 2, FEBRUARY 1992 99 Single Lens Stereo with a Plenoptic Camera Edward H. Adelson and John Y.A. Wang (a) (b) (c) (d) Fig. 2. (a) Pinhole camera forms an image from a single viewpoint; (b) in a stereo system, two images are formed from different viewpoints; (c) in a motion parallax system, a sequence of images are captured from many adjacent viewpoints; (d) a lens gathers light from a continuum of viewpoints; in an ordinary camera these images are averaged at the sensor plane.

Practical applications

What is there to see? Black and white photo: x, y Black and white movie: x, y, t Color movie: x, y, t, λ Holographic movie: x, y, t, λ , V x , V y , V z Hence, the plenoptic function: P(x, y, t, λ , V x , V y , V z ) Proposition 1 . The task of early vision is to deliver a small set of useful measurements about each observable location in the plenoptic function. Proposition 2 . The elemental operations of early vision measure local change along various directions within the plenoptic function.

� 1 1 700 picture moving grey background y t plane red bar -1 -1 400 -1 0 1 -1 0 1 -1 0 1 x x x 1 1 1 V V Vz y x -1 -1 -1 -1 0 1 -1 0 1 -1 0 1 x x x A hypothetical scene that produces a The plenoptic structures found along various planes. variety of simple plenoptic structures. Each panel represents a slice through the plenoptic function.

� Some edge-like structures found in particular planes within the plenoptic function Vertical edge Horizontal edge Static edge Brightness step y y t t x x x x Edge with horizontal Tilted edge Moving edge Color sweep binocular parallax V x y t x x x x

Local derivatives will turn out to be handy

� � Local derivatives will turn out to be handy f(s)*g(s) f(s) local average *g(s) derivative filter derivative *g'(s) � s � s f(s)*g'(s) f'(s) local average *g(s)

� � Derivatives along single dimensions yield some basic visual measurements response binocular vertical horizontal flicker blue-yel. anticorrel. "edge" "edge" (brightening) opponency ("luster") V x y t dimension x of interest response vertical horizontal flicker green-red "bar" "bar" (pulse) opponency V dimension x y t x of interest Low-order derivatives lead to a few two-dimensional operators (receptive fields)

� � � The same receptive field structures produce different measurements when placed along different planes in plenoptic space Horizontal Vertical Diagonal Uniform Achromatic Spatiochromatic edgelike structure edgelike structure edgelike structure bluishness edgelike structure structure y y y x x x x x x Uniform intensity Uniform Static Moving No horizontal Horizontal change with brightening edgelike structure edgelike structure parallax parallax eye position V x V V x x t t t x x x x x x

� � What can you measure with a tilted second derivative? x diag. "bar" static y achromatic no dispar vert. "bar" hor. "bar" leftward downward t achromatic achromatic no dispar no dispar vertical horiz. full-field static static sequential hue-sweep hue-sweep hue-sweep no dispar no dispar no dispar vert. "bar" hor. "bar" full-field full-field static static sequential static V x achromatic achromatic achromatic hue-shift hor. dispar vert. dispar eye-order luster x y t V x

Orientation selectivity in V1 Hubel & Wiesel ( 1962, 1968 )

Orientation in space can be detected and measured by oriented filters Hawken & Parker, 1991

Orientation in space can be detected and measured by oriented filters 20 c/deg 15 10 5 0 5 10 15 20 - + - + DeValois, Albrecht and Thorell, 1982

Motion is orientation in space-time

Motion is orientation in space-time and spatiotemporally oriented filters can be used to detect and measure it Adelson & Bergen (1985)

���� ���� ����� ����� ��������฀��������฀ ��������฀���� ���� ������ �� �� � ��� � ������ ���� ���������� Greg DeAngelis

��฀��฀��฀ �����฀��฀ ��� � � � � ฀ � � � � ฀ � � � � � ������฀�฀�����฀ � � � � � ��฀���฀��฀ �����฀��฀ DeAngelis, Ohzawa & Freeman, 1995

DIsparity is orientation in space-eye position

DIsparity is orientation in space-eye position Binocular correlation: Binocular anticorrelation: Uncrossed disparity: Crossed disparity: “tuned excitatory” “tuned inhibitory” “far” “near” R.E. R.E. R.E. R.E. V x V V x V x x L.E. L.E. L.E. L.E. x x x x L.E. R.E. L.E. R.E. R.E. L.E. L.E. R.E. Four examples of binocular receptive fields. Humans only take two samples from the V x axis, as shown by the two lines labeled R.E. and L.E. for right eye and left eye. The curves beneath each receptive field indicate the individual weighting functions for each eye alone.

DIsparity is orientation in space-eye position Poggio (1981)

� � � � � � � � � Many psychophysical tasks look like Vernier acuity in different planes hor. vernier vert. bipart. vert. edge static pulse order static x achromatic achromatic diff. no dispar. no dispar. no dispar. vert. vernier hor. bipart. hor. edge y t y y static pulse order static y achromatic achromatic diff. no dispar. no dispar. no dispar. x x t vert. line horiz. line full-field jump left jump down sequential t achromatic achromatic change no dispar. no dispar. no dispar. R.E. R.E. V V 2 vert. lines 2 hor. lines full-field x x static static pulse order L.E. L.E. change change blue-yel. no dispar. no dispar. no dispar. x t horiz. line full-field full-field vert. line static static pulse order static V x achromatic achromatic achromatic diff. h disp. v disp. anticorr. anticorr. x y t V x