Marta Favali Thesis director: Alessandro Sarti Thesis co-director: - - PowerPoint PPT Presentation

École Doctorale Cerveau-Cognition Comportement

Doctorate in Theoretical Neuroscience CAMS (CNRS-EHESS) Doctorate in Mathematics Department of Mathematics (University of Bologna)

Marta Favali

Thesis director: Alessandro Sarti Thesis co-director: Giovanna Citti

Title of the project: Formal models of visual perception based on cortical architectures.

1

Objectives

• The neurogeometry of the visual cortex
• Models of cortical connectivity, with different stochastic kernels
• Spectral analysis of connectivity matrix
• Simulations (Kanizsa figures and retinal images).

Methods and development of work:

• Mathematical models of the primary visual cortex
• Mathematical models of visual perception

2

Individuation of perceptual units: the association fields

Field et al, 1993

3

Mathematical models of the functional architecture of V1

— J.J. Koenderink, A.J van Doorn, Representation of local geometry in the visual system.,

• Biol. Cybernet. 55,367-375, 1987.

— J. Petitot, The neurogeometry of pinwheels as a sub-Riemannian contact structure, in

Journal Physiol, Pages 97(2-3):265-309, 2003.

— G. Citti, A.Sarti, A cortical based model of perceptual completion in the roto-translation

space, Journal of Mathematical Imaging and Vision, 24(3):307-326, 2006

— S.W. Zucker, Differential geometry from the Frenet point of view: boundary detection,

stereo, texture and color., In: Paragios, N., Chen, Y., Faugeras, O. (eds.) Handbook of Mathematical Models in Computer Vision, pp. 357-373. Springer, US, 2006.

— A.Sarti, G. Citti, J. Petitot, The symplectic structure of the primary visual cortex, Biol.

• Cybern. 98, 33-48, 2008.

— R. Duits, E.M. Franken, Left invariant parabolic evolution equations on SE(2) and

contour enhance- ment via invertible orientation scores, part I: Linear left-invariant diffusion equations on SE(2), Q. Appl. Math. 68, 255-292, 2010.

4

The neurogeometry of V1

Hypercolumnar structure Receptive profile of a simple cell and its representation as a even-symmetric and odd- symmetric Gabor filters.

Hubel-Wiesel, 1965

Daugman, 1985

ϕ(x, y,θ) = 1 2πσ 2 e

[−( ! x2+! y2 ) σ 2 +i ! y σ ]

! x = xcos(θ)+ ysin(θ) ! y = −xsin(θ)+ ycos(θ)

5

6

• Simple cells are modeled with Gabor filters and represent a group:

! X1 = (cosθ,sinθ,0)

! X2 = (0,0,1)

! X3 = (−sinθ,cosθ,0)

Sarti Citti, 2006

Output of simple cells: Lifting: nonmaximal suppression

h(x, y,θ) = ϕx,y,θ(x', y')I(x', y')

∫

dx' dy' maxθ(h(x, y,θ)) = h(x, y,θ)

7

! X1 = (cosθ,sinθ,0)

! X2 = (0,0,1)

X1 = cos(θ)∂x +sin(θ)∂y X 2 = ∂θ

! X1, ! X2, ! X3

generator of the tangent space.

Citti-Sarti, 2006

X3 =[X2, X1]= −sin(θ)∂x +cos(θ)∂y

8

Differential model of Citti-Sarti

Citti-Sarti, 2006

X1 = cos(θ)∂x +sin(θ)∂y X 2 = ∂θ γ '(t) = ! X1(γ)+ k ! X2(γ) γ(0) = (x0, y0,θ0)

9

The Fokker Planck operator has a nonnegative fundamental solution that satisfies:

X1Γ1((x, y,θ),(x', y',θ '))+σ 2X22Γ1((x, y,θ),(x', y',θ ')) =δ(x, y,θ)

Γ1

Sanguinetti Citti Sarti, 2008

25 y 15 5 50 30 x 10

3

:

2: 3 : 3

ω1

The Sub-Riemannian Laplacian operator has a nonnegative fundamental solution that satisfies:

Γ2

σ 2

1X11Γ2((x, y,θ),(x', y',θ '))+σ 2 2X22Γ2((x, y,θ),(x', y',θ ')) =δ(x, y,θ)

ω2

10

Bosking et al, 1997 The connectivity map measured by Bosking in tree shrew: Maximum values along dimension of the connectivity kernels associated to the fundamental solution of a FP (left) and SRL equations (right).

θ

11

Affinity Matrix

ω((xi, yi,θi),(x j, yj,θ j))h(x j, yj,θ j)

j=1 N

∑

Ai, j =ω((xi, yi,θi),(x j, yj,θ j))

Propagation of to close cells:

h(xi, yi,θi)

12

Individuation of perceptual units: Kanizsa figure

20 40 60 80 20 40 60 80

• 1. Define the affinity matrix from the approximated connectivity

kernel.

• 2. Solve the eigenvalue problem , where the order of i is such

that is decreasing.

• 3. Find and represent on the segments the eigenvector associated to its

largest eigenvalue.

Ai, j Ai, jui = λiui

λi

u1

Numerical algorithm

13

First eigenvector of the affinity matrix, using the fundamental solutions of FP and SRL equations. The affinity matrix is updated removing the detected perceptual unit; the first eigenvector of the new matrix is visualized.

14

(a) (b) (c) (d) In red the first eigenvectors

• f the affinity matrix using

both connectivity kernel.

15

F., Citti, Sarti: “Local and global gestalt laws: A neurally based spectral approach”, submitted to Neural Computation, 2015.

16

Individuation of perceptual units: retinal images

Analyzed problems:

bifurcation crossing disconnected vessels

In collaboration with TU/e

17

— In presence of an input stimuli, the visual cortex codifies the features

• f position and orientation.

— The proposed method models the connectivity as the fundamental solution

• f the Fokker-Planck equation.

Image patch: crossing Oriented segments

1 y 11 21 1 11 x

3

21

2: 3 : 3

:

Lifted image

— In order to measure the distances between intensities we introduce the

kernel :

— The final connectivity kernel can be written as the product of the two

components:

— Starting from that connectivity kernel it is possible to extract perceptual

units from images by means of spectral analysis of suitable affinity matrix:

ω3 ω3( fi, f j) = e

(−1 2( fi− f j σ 2 ))2

ω ((xi, yi,θi, fi),(x j, yj,θ j, f j)) = ω1((xi, yi,θi),(x j, yj,θ j))ω3( fi, f j)

18

Aij = ω ((xi, yi,θi, fi),(x j, yj,θ j, f j))

20 40 60 80 20 40 60 80

Normalized Spectral Clustering

19

• 1. After defining the affinity matrix from the connectivity kernel
• 2. We evaluate the normalized affinity matrix where is the diagonal

degree matrix having elements:

• 3. Solve the eigenvalue problem:
• 4. Define the thresholds and evaluate the largest integer K such that

for

A

P = D

−1

A

Pum = λmum ε,τ λτ

m >1−ε

Shi Malik, 2000 Meila Shi, 2001

5 10 15 0.2 0.4 0.6 0.8 1

Eigenvalues

m =1,..., K D di = ai, j

j=1 n

∑

Normalized Spectral Clustering

20

• 5. Define the clusters from the eigenvector
• 6. Find and remove the clusters that contain less than a minimum cluster size

elements.

uK

Perceptual units Image patch

21

F., Abbasi, Romeny, Sarti: “Analysis of Vessel Connectivities in Retinal Images by Cortically Inspired Spectral Clustering”, submitted to JMIV, 2015.

Conclusion

— We have presented a neurally based model for figure-ground segmentation

using spectral methods.

• Different connectivity kernels are compatible with the functional

architecture of V1, we have compared their properties and modelled them as fundamental solution of Fokker Planck, Sub-Riemannian Laplacian equations.

• With this model we have identified perceptual units of different Kanizsa

figures and retinal images.

• We have shown how this can be considered a good quantitative model for

the constitution of perceptual units.

22

23

• Our method represents some limitations at blood vessels with high
• curvature. These structures will be analyzed in an higher dimensional

group (Engel group) adding other features.

• Other images containing tree structures will be analyzed.
• We will compare the results obtained with this model with functional

fMRI data, that represent measurements of cortical neural activity.

Future work

Thanks for your attention

24