Scale-space theory and PDE-based image filtering Nicolas Rougon - - PowerPoint PPT Presentation

▶
scale space theory and pde based image filtering
SMART_READER_LITE
LIVE PREVIEW

Scale-space theory and PDE-based image filtering Nicolas Rougon - - PowerPoint PPT Presentation

Dec 10, 2023 421 likes •1.46k views

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Scale-space theory and PDE-based image filtering Nicolas Rougon Institut Mines-Tlcom / Tlcom SudParis ARTEMIS Department; CNRS UMR 8145

slide-1
SLIDE 1

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography

Scale-space theory and PDE-based image filtering

Nicolas Rougon

Institut Mines-Télécom / Télécom SudParis ARTEMIS Department; CNRS UMR 8145 nicolas.rougon@telecom-sudparis.eu

May 17, 2020

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-2
SLIDE 2

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Outline

1

Foundations

2

Linear scale-space

3

Geometric scale-spaces

4

Conservative scale-spaces

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-3
SLIDE 3

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Scale

◮ An intuitive polysemous concept Definition (Collins | Webster)

1 a distinctive relative size, extent or degree 2 a proportion between 2 sets of dimensions 3 a sequence of marks at regular intervals or representing

equal steps, used as a reference in making measurements

4 a measuring instrument having such a graduation 5 an established measure or standard 6 a graduated series or scheme of rank or order Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-4
SLIDE 4

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Scale in images

◮ Formalizing the concept of scale in images is a complex issue linked to basic modeling questions, since this requires to understand the way objects are embedded in image space

◮ what is an image?

to understand their appearance and relations to their context

◮ what is the structure of images?

to define measuring instruments (operators)

◮ what are optimal measurements? ◮ how accurate measurements can be? ◮ what are the best apertures?

to define measurement units

◮ how to sample the scale range?

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-5
SLIDE 5

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Scale in basic sciences

◮ A concept ignored in Mathematics k-dimensional subspaces of Rn (k < n) have measure zero infinitesimal neighborhoods → local operators no units ◮ An ubiquitous notion in Physics

  • bjects belong to distinctive scale intervals
  • bservable scales (≈ 50 decades) are upper / lower-bounded

sensors / measuring instruments have finite aperture → nonlocal operators dimensional units are essential → dimensional analysis

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-6
SLIDE 6

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Scales of the physical world

1025 galatic super clusters 1024 galactic clusters 1023 cosmic region 1022 galactic clouds 1021 1020 galactic disc 1019 Zodiac 1018 star spectra 1017 stars 1016 the Sun as a star 1015 stellar geometry 1014 solar system 1013 external planets 1012 asteroids & 1011 internal planets 1010 Earth orbit 109 Moon & Earth 108 the Earth 107 continents 106 continental regions 105 countries 104 towns 103 nature & human 102 constructions 101 living creatures & 100

  • bjects

10−1 hand & tool 10−2 finger 10−3 skin 10−4 blood vessel 10−5 lymphocite 10−6 cell nucleus 10−7 chromosome 10−8 DNA 10−9 nucleotids 10−10 atomic surface 10−11 electronic cloud 10−12 nuclear volume 10−13 atomic nucleus 10−14 protons & neutrons 10−15 quarks 10−16 ?

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-7
SLIDE 7

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Scale in vision sciences

◮ Biology & Neurosciences the human retina consists of receptive fields with varying spatial extension, defining a large range of sampling apertures the human visual system is designed to extract / process multiscale information ◮ Digital imaging digital images are sampled over discrete grids with finite resolution / size which bound the range of observable scales computer vision systems perform scene decomposition into

  • bjects with distinctive extension

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-8
SLIDE 8

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Human visual pathways

Optjc tract Lateral Geniculate Nucleus Optjc nerve Optjc chiasm LEFT RIGHT Optjc radiatjon Visual cortex Visual fields

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-9
SLIDE 9

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Human visual system

Optjc tract LGN Optjc chiasm Optjc radiatjon Motor cortex Somato- sensory cortex

19 18 18 18 17 17 19 19

Visual cortex Corpus callosum

4 4 2 1 1 2 3 3

LEFT MID TOP Optjc nerve Olfactory bulb Central sulcus Central sulcus

◮ Visual areas V1 (17) striate cortex V2 (18) V3 (19) V3a, V4, V7

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-10
SLIDE 10

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Retinal cells

Optjc nerve fibers Glanglion cells Cones - Rods nuclei Horizontal - Bipolar Amacrine cells Retjna

> 10⁹ INTEGRATION 10⁶

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-11
SLIDE 11

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Retinal cells

Light Optjc nerve fibers Y-ganglion cell Cone - Rod nuclei Horizontal cell Bipolar cell Amacrine cell Rod Cone X-ganglion cell

◮ Cellular connections induce neighborhood relationships over the retina ◮ This topology enables nonlocal processing

  • f light information

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-12
SLIDE 12

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Visual fields

The vertical meridian splits each eye’s visual field into L/R hemifields ◮ Lateralization The R(L) half of the brain receives sensory information from and sends motor commands to the L(R) half of body The L(R) retinal hemifields are jointly mapped to the R(L) LGN via the optic chiasm

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-13
SLIDE 13

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Lateral Geniculate Nucleus

◮ Binocular retinal mapping to a layered architecture Monocular nervous terminations are segregated into 6 layers

Parvocellular (P) layers (3-6) P cells (small | slow response) get inputs from X-ganglion cells linked to red / green cones > detailed shape, color Magnocellular (M) layers (1,2) M cells (large | rapid response) get inputs from Y-ganglion cells linked to rods > motion, depth, small luminance differences

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-14
SLIDE 14

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Lateral Geniculate Nucleus

◮ Binocular retinal mapping to a layered architecture The magno / parvocellular layers are interleaved between 6 contact layers

Koniocellular (K) layers (K1-K6) K cells (very small) receive inputs from X-ganglion cells linked to blue cones > unclear role

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-15
SLIDE 15

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Lateral Geniculate Nucleus

◮ Binocular retinal mapping to a layered architecture The magno / parvocellular layers alternate inputs from each eye

Layers 2,3,5 get input from the eye

  • n the same side ≡ ipsilateral eye

Layers 1,4,6 get input from the eye

  • n the opposite side ≡ contralateral

eye

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-16
SLIDE 16

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Lateral Geniculate Nucleus

◮ Binocular retinal mapping to a layered architecture

LEFT eye RIGHT eye LEFT LGN

Axons from neighboring retinal areas connect to neighboring LGN cells in each LGN layer Each LGN layer receives an entire map of a visual hemifield Maps in adjacent layers are registered → binocular information processing by the cortex is simplified This architecture is referred to as the LNG retinotopic map

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-17
SLIDE 17

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Primary visual cortex

V7 V3a V3 V2 V1 V4

◮ The 6 LGN layers are mapped via the optic radiation to the primary visual cortex (V1) ◮ V1 maintains a retinotopic map The central 10° of the visual field occupies roughly half of V1 This distorsion, called cortical magnification, echoes the increased acuity of the fovea

RIGHT eye V1 Scene 5 6 7 89

1 2 3 5 6 7 8 9 4 1 2 3 5 6 7 8 9 4

The L(R) retinal hemifield is mapped onto the R(L) hemisphere V1 (lateralization)

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-18
SLIDE 18

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

V1 cells

◮ V1 neurons organize into a layered architecture

I II III IVa IVb IVc V VI

I IVa,b IVc V VI II + III Pyramidal cell Axons Stellate cell

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-19
SLIDE 19

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

V1 physiology

◮ V1 physiology and functional organization have been elucidated by David Hubel and Torsten Wiesel (Nobel Prize in Medecine, 1981) ◮ V1 neurons divide into simple / complex / hypercomplex cells. These 3 types differ by the way they respond to visual stimuli Neuron activity is monitored via electrophysiology Most V1 neurons are orientation selective i.e. respond strongly to lines/bars/edges with a specific orientation Some V1 neurons are direction selective i.e. respond strongly to oriented lines/bars/edges moving in a specific direction

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-20
SLIDE 20

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Receptive fields

◮ The LGN / V1 retinopic mapping enables correlating cell / neuron activity with retinal stimuli Receptive field Region of the retina influencing the activity of a cell / neuron assembly when exposed to a light stimulus Receptive fields of V1 simple cells

best stimulus A elongated light bar B elongated dark bar C elongated dark bar D edge

B

X inbibitory zone excitatory zone

C A D

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-21
SLIDE 21

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Receptive fields

cell type geometry best stimuli selectivity binocular

  • rientation direction

disparity

X-Y ganglion light no no no no M-P-K blob no no no no simple elongated bar / edge yes some some some complex elongated bar / edge yes yes some yes hypercomplex short edge corner yes yes some yes

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-22
SLIDE 22

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

V1 functional architecture

◮ V1 cells organize functionally into a columnar architecture

LGN

IVb IVc III Ipsilateral eye Contraletral eye Orientatjon columns Ocular dominance columns

1 2 3 4 5 6

IVa Complex cells Simple cells

Neurons with activity mainly influenced by one eye organize into ocular dominance columns Neurons with a given orientation selectivity organize into orientation columns Hypercolumns gather neurons having the same receptive field location, but all different

  • rientation/direction selectivities

and both eye dominances

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-23
SLIDE 23

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

V1 functional architecture

  • cular dominance columns

◮ V1 columns can be investigated in vitro via histology in vivo via intrinsic optical imaging

  • r high-field fMRI
  • rientations

columns

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-24
SLIDE 24

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Biological visual information processing

◮ Governing principles Retinopic mapping Retinal maps are preserved / registered along visual pathways Functional simplicity Visual cells / neurons are divided into a few specialized types with preferred response to a given class of stimuli Architectural efficiency Visual cells / neurons with the same dominance / selectivity

  • rganize into layers

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-25
SLIDE 25

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Biological visual information processing

The human visual system ≡ a geometric engine ◮ Along the pathways to the cortex, visual information is processed in a massively parallel way locally at multiple integration scales hierarchically with no feedback in its early steps selectively with an increasing degree of nonlinearity ◮ These biological features are reflected by scale-space theory which can be seen as a mathematical theory of early vision

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-26
SLIDE 26

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Scale-space theory

◮ A deterministic framework for deriving hierarchical image / shape representations according to level of detail (LOD) based on continuous data modeling

– images as smooth functions over Ω ⊂ Rn – shapes as smooth submanifolds of Rn

applicable to nD, scalar/vector, still/animated data delivering scene decomposition into LOD

– scene description at given LOD → filtering / restoration – relations between scene components at varying LODs –

  • bject assignment to distinctive scale range

enabling multiscale image analysis

– feature extraction – segmentation / grouping – motion estimation / tracking – scene / shape reconstruction

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-27
SLIDE 27

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Scale-space theory

◮ Though developed in image science, scale-space theory is deeply connected to calculus ◮ differential geometry | PDE theory | variational calculus theoretical physics ◮ quantum field theory behavioral / cognitive neurosciences ◮ neuropsychology | psychobiology | psychophysics

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-28
SLIDE 28

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Scale operators

T T T

image space t scale t1 t2 t1 t2 t1,t2 t

◮ Scale-spaces are built / explored using local image operators parameterized by a scale variable t ∈ [ t, t ] ⊂ R+ t inner scale (pixel size) t outer scale (image size) scale operators Tt produce images at scale t from original ones scale transition operators Tt,t′ generate images at scale t′ from images at lower scale t the image family (Tt f )t is the scale-space representation

  • f the image f

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-29
SLIDE 29

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Structural axioms

◮ Causality axioms ensure that scale operators do not create details Semigroup Associativity Tt1 Tt2 = Tt1+t2 Identity T0 = Id Scale-spaces have a hierarchical structure ≡ image pyramids Local comparison ∀y ∈ N(x) f (y) > g(y) ⇒ (Tt f ) (x) > (Tt g) (x) for any scale t less than an extinction scale te Scale operators preserve ordering between image level sets during their lifetime te in scale-space Scale operators do not enhance any image structures

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-30
SLIDE 30

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Structural axioms

◮ The smoothness axiom ensures that image simplification by scale

  • perators is a C1-continuous process

Smoothness Scale operators have a continuous derivative ∂T at t = 0 ∂T f = lim

t→0

Tt f − f t ∂T is known as the infinitesimal generator of the scale

  • perator semigroup (or, in short, as the scale-space generator)

Scale operators preserve image smoothness

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-31
SLIDE 31

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Structural axioms

◮ Smoothness axiom Let Qf ,x be the quadratic approximation of the (locally smooth) image f over the neighborhood N(x) of pixel x ∈ Rd Qf ,x(y) = (x − y)TA (x − y) + pT(x − y) + c (A symmetric (d × d) matrix, p ∈ Rd, = c ∈ R The smoothness axiom states that ∂T Qf ,x = lim

t→0

Tt Qf ,x − Qf ,x t is a function F(A, p, c, x, t) continuous w.r.t. the highest frequency component A

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-32
SLIDE 32

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Multiscale vs multiresolution representations

◮ Whereas scale-space methods operate on the original pixel grid, multiresolution techniques combine image simplication with grid decimation to yield image pyramids

n-2 n-1 n level

subsampling (usually dyadic) u(k−1) = ↓2 u(k) smoothing u(k−1)(x) =

N

  • n=−N

cn u(k)(2x−n)

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-33
SLIDE 33

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Multiscale vs multiresolution representations

◮ Low-pass pyramid positivity cn ≥ 0 unimodularity c|n| ≥ c|n|+1 symmetry cn = c−n normalization

cn = 1

equidistribution

c2n = c2n+1

◮ Example: Gaussian pyramids N = 1 (binomial filter) 1 4

  • 1

2 1

  • N = 2 (a ≈ 0.4)

1 4

  • 1 − 2a

1 4a 1 1 − 2a

  • ◮ Differences of low-pass pyramids yield band-pass pyramids

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-34
SLIDE 34

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Multiscale vs multiresolution representations

◮ Gaussian scale-space

t 1 2 3 4

◮ Gaussian pyramid (rescaled to full resolution)

k 1 2 3 4 Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-35
SLIDE 35

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Generating PDE

Generating PDE The scale-space representation of the image f is the solution u(x, t) = (Tt f )(x) of the PDE ut = ∂T u with u(·, 0) = f and Neumann (mirroring) boundary conditions ◮ Photometric interpretation: The generating PDE describes how luminance varies during a scale transition

t 10 25 75 150 300 600 Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-36
SLIDE 36

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Generating PDE

◮ Geometric interpretation: The generating PDE determines how image level sets evolve during a scale transition Level set evolution is described by a speed law xt = V (x) n along its external normal n = − ∇u

|∇u|

V (x) is obtained by substitution from differenciating the level set equation u(x, t) = c ut + ∇u · xt = 0 Level set scale-space flow V (x) = ∂T u |∇u|

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-37
SLIDE 37

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Scale-space generators

Representation theorem (Alvarez-Morel-Lions) Scale-space generators are differential operators of order n ≤ 2 ∂T u = F(D2u, Du, u) ◮ The generating PDE combines 3 processes diffusion responsible for smoothing is driven by order-2 terms related to image curvature/convexity ◮ parabolic reaction responsible for transport is driven by order-1 terms related to local contrast/orientation ◮ hyperbolic addition responsible for luminance transformation is driven by

  • rder-0 terms

◮ The dominant process is dictated by the highest-order term

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-38
SLIDE 38

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Morphological axioms

◮ Morphological axioms state how scale operators depend / act on image content (involved features / targeted structures) ◮ Added to structural axioms, they allow for completely elucidating scale-space generators under closed-form ◮ They divide into 2 groups Linearity Tt (α u + β v) = α Ttu + β Ttv Invariance ≡ commutation between scale operators and a specific group G of transforms ∀g ∈ G g Tt′ = Tt g

– strong t′ = t – weak t′ = ϕ(t) (resynchronization)

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-39
SLIDE 39

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Morphological axioms

◮ Linearity Universality: scale-space is built w/o any prior assumption Blind processing: scale-space operators act similarly on image content whatever its (local) characteristics ◮ Invariance Introducing priors implies nonlinearity Globally: G models some variability on sensor The scale-space representation of an image does not depend

  • n the related sensor calibration

Locally: G models some variability on image content Scale operators act similarly on image structures whatever their related appearance

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-40
SLIDE 40

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Morphological axioms

◮ Photometric invariance G consists of transforms over the luminance interval [0, 2b − 1] group transforms invariance general invertible strong (Tt f )t does not depend on sensor photometric calibration Tt acts similarly on image content whatever its contrast

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-41
SLIDE 41

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Morphological axioms

◮ Spatial invariance G consists of transforms over the image domain Rd group transforms invariance Euclidean isometry strong affine affine weak t′ =

  • |det(g)| t

projective perspective weak (Tt f )t does not depend on sensor geometric calibration Rotation-invariance ≡ Tt has no preferred orientation Image processing is isotropic Zoom-invariance ≡ Tt has no preferred extension Image structures are processed similarly whatever their size

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-42
SLIDE 42

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Motivation Biological foundations Mathematical foundations

Scale-space families

◮ Scale-spaces can be classified into 3 families depending on the set of selected morphological axioms linearity invariance Geometric Conservative Gaussian linear spatial contrast non linear Structural axioms Morphological axioms

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-43
SLIDE 43

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE & Scale operators Properties Discrete linear scale-space Limitations

Outline

1

Foundations

2

Linear scale-space

3

Geometric scale-spaces

4

Conservative scale-spaces

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-44
SLIDE 44

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE & Scale operators Properties Discrete linear scale-space Limitations

Linear scale-space

◮ Assuming linearity leads to a single scale-space model Generating PDE (Marr-Koenderink-Witkin) There is a unique linear, isometry-invariant and zoom-invariant scale-space. Its generating PDE is the isotropic heat equation ut = ∆u This model is referred to as the linear scale-space Its generator is the Laplacian operator ∂T = ∆ Image simplification is performed via luminance diffusion

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-45
SLIDE 45

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE & Scale operators Properties Discrete linear scale-space Limitations

Linear scale-space

◮ Solutions of linear PDEs are obtained by convolving their initial condition against a particular solution (kernel) The heat equation kernel is the isotropic Gaussian Gσ(x) = 1 (2πσ2)

d 2

exp

  • −|x|2

2σ2

  • Scale operators

The scale-operators for the linear scale-space are Gaussian convolutions with variance proportional to scale Tt = G√

2t ⋆

Equivalent terminology: Gaussian scale-space

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-46
SLIDE 46

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE & Scale operators Properties Discrete linear scale-space Limitations

Extensions of linear scale-space

◮ Spatial frequency tuning: Gabor filters Gσ, k(x) = 1 (2πσ2)

d 2

exp

  • −|x|2

2σ2

  • exp (−i k · x)

The wave vector k defines spatial orientation/frequency

  • rientatjon

frequency

Sampling the (t, k)-space yields the Gabor space Widely used for texture modeling and discrimination in computer vision / pattern recognition (e.g. character, fingerprint, iris) Relevant model for simple cells in mammals visual cortex

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-47
SLIDE 47

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE & Scale operators Properties Discrete linear scale-space Limitations

Extensions of linear scale-space

◮ Spatial tuning Gσ, ξ(x) = 1 (2πσ2)

d 2

exp

  • −|x − ξ|2

2σ2

  • Modeling receptive field assemblies in bio-inspired vision

◮ Spatio-temporal linear scale-space Gσs, στ (x) = 1 (2πσ2

s )

d 2

1 (2πσ2

τ)

1 2

exp

  • −|x|2

2σ2

s

− τ 2 2σ2

τ

  • σ2

s = 2t (spatial scale) | σ2 τ = 2τ (temporal scale)

Multiscale modeling / analysis of video sequences

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-48
SLIDE 48

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE & Scale operators Properties Discrete linear scale-space Limitations

Properties of linear scale-space

◮ Causality: the generating PDE verifies a maximum principle max

Ω×R+ u(x, t) = max Ω

u(x, 0) d = 1: non-creation of local extrema

(Tt f )t

te

(∆Tt f = 0)t

d > 1: non-enhancement of local extrema survival time te is a measure of feature saliency

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-49
SLIDE 49

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE & Scale operators Properties Discrete linear scale-space Limitations

Properties of linear scale-space

◮ Smoothness: the linear scale-space yields C∞ image families ◮ Convolution theorem: image derivatives belong to scale-space ∂xiyj u = G√

2t ⋆ ∂xiyj f

Image local geometry is extensively available in the linear scale-space ≡ universal front-end for image understanding Gaussian derivatives ≡ precomputable geometric kernels ∂xiyj u = ∂xiyj G√

2t ⋆ f

Natural Gaussian derivatives w.r.t. normalized coordinates

  • x = x/σ are multiscale extensions of standard derivatives

lim

t→0 ∂ xi yj G√ 2t = ∂xiyj

Images are processed as distributions

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-50
SLIDE 50

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE & Scale operators Properties Discrete linear scale-space Limitations

Properties of linear scale-space

◮ Gaussian differential kernels (d = 2)

Gσ (Gσ)x (Gσ)y (Gσ)xx (Gσ)xy (Gσ)yy (Gσ)xxx (Gσ)xxy (Gσ)xyy (Gσ)yyy

D2nGσ are wavelets

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-51
SLIDE 51

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE & Scale operators Properties Discrete linear scale-space Limitations

Properties of linear scale-space

◮ Multiscale feature extraction u |∇u| ∆u

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-52
SLIDE 52

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE & Scale operators Properties Discrete linear scale-space Limitations

Properties of linear scale-space

◮ Isotropy: simplication is performed by diffusing out image blobs ◮ Separability: 1D computations In the spatial domain using order-4 IRR filters [Deriche]

  • r Hermite polynomials Hn

∂n

x Gσ = (−1)n

σn Hn

x

σ

  • Gσ

∂n

x e−x2 = (−1)n Hn(x) e−x2

In the spectral domain using FFT Gσ ⋆ f = FFT−1 FFT(Gσ) · FFT(f )

  • Nicolas Rougon

IMA4509 | Scale-space & PDE filtering

slide-53
SLIDE 53

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE & Scale operators Properties Discrete linear scale-space Limitations

Discrete linear scale-space

◮ Transposing the linear scale-space framework from Rd to digital grids Ω ⊂ Zd is a hard problem Directly discretizing the Gaussian scale operators results in violating causality axioms ◮ An intrinsically discrete derivation of scale operators, ensuring that scale-space structural axioms hold over Zd, is mandatory The key point lies in satisfying the non-enhancement of local extrema property in a discrete setting

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-54
SLIDE 54

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE & Scale operators Properties Discrete linear scale-space Limitations

Discrete linear scale-space

1D discrete linear scale-space (Lindeberg) The scale operators for the 1D discrete linear scale-space are convolutions against a discrete kernel Tt Tt = Tt ⋆ which is related to Bessel functions Jn Tt(n) = e−αt In(αt) I±n(t) = (−i)nJn(it) Tt is known as the discrete analog of the Gaussian kernel Gσ Sampling the continuous Gaussian scale operators does not lead to discrete linear scale operators

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-55
SLIDE 55

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE & Scale operators Properties Discrete linear scale-space Limitations

Discrete linear scale-space

Generating PDE (d = 1) (Lindeberg) The 1D discrete linear scale-space is generated by the semi-discrete heat equation ut(x, t) = 1 2

  • u(x + 1, t) − 2u(x, t) + u(x − 1, t)
  • Its generator is the discrete Laplacian kernel

∆3 =

1

−2 1

  • This PDE holds exactly

Recursion properties of Bessel functions The discrete linear scale-space is properly derived by discretizing the continuous linear scale-space generator

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-56
SLIDE 56

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE & Scale operators Properties Discrete linear scale-space Limitations

Discrete linear scale-space

Generating PDE (d = 2) (Lindeberg) A 2D discrete linear scale-space is generated by a semi-discrete heat equation (α ∈ [0, 1]) ut = α ∆4u + (1 − α) ∆8u ∆4, ∆8 are discrete Laplacian kernels ∆4 =

  

1 1 −4 1 1

  

∆8 = 1 2

  

1 1 −4 1 1

  

For d > 1, the discrete linear scale-space is not unique Each generator relates to a local topology on the image grid e.g. separability: α = 1 | isotropy: α = 2

3

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-57
SLIDE 57

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE & Scale operators Properties Discrete linear scale-space Limitations

Discrete linear scale-space

Generating PDE (d = 3) (Lindeberg) A 3D discrete linear scale-space is generated by a semi-discrete heat equation (α, β ∈ [0, 1]2) ut = α ∆6u + β ∆18u + (1 − α − β) ∆26u ∆6, ∆18, ∆26 are discrete Laplacian kernels ∆6uijk =

  • N ∗

6 (ijk)

ulmn − 6uijk N ∗

6

∆18uijk = 1 4

  • N ∗

18\6(ijk)

ulmn − 12uijk

  • N ∗

18\6

∆26uijk = 1 4

  • N ∗

26\18(ijk)

ulmn − 8uijk

  • N ∗

26\18

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-58
SLIDE 58

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE & Scale operators Properties Discrete linear scale-space Limitations

Discrete linear scale-space

◮ Implementation Let (∂Tu)ij be a discrete linear scale-space generator at pixel ij Finite difference discretization of ut (ut)ij = uij(t + δt) − uij(t) δt Explicit scheme uij(t + δt) = uij(t) + δt

∂Tu(t)

  • ij

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-59
SLIDE 59

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE & Scale operators Properties Discrete linear scale-space Limitations

Limitations of linear scale-space

◮ Oversmoothing artifacts due to strong regularization properties

  • f Gaussian scale operators Tt

contrast loss of salient image structures ◮ Non-preservation of image geometry due to linearity and isotropy

  • f Laplacian scale-space generators ∂T

delocation of image structures

  • rientation smoothing

◮ complex multiscale image analysis schemes

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-60
SLIDE 60

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE & Scale operators Properties Discrete linear scale-space Limitations

Limitations of linear scale-space

t 4 16 64 256

◮ Overcoming these limitations requires nonlinear and anisotropic scale-space models

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-61
SLIDE 61

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE Morphological scale-spaces Intrinsic scale-spaces Limitations

Outline

1

Foundations

2

Linear scale-space

3

Geometric scale-spaces

4

Conservative scale-spaces

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-62
SLIDE 62

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE Morphological scale-spaces Intrinsic scale-spaces Limitations

Euclidean geometric scale-spaces

Generating PDE (Alvarez-Morel-Lions) Contrast- and isometry-invariant scale-spaces are generated by reaction-diffusion PDEs of the form ut = |∇u| F

curv(u)

  • where F is an increasing function of image level set mean curvature

curv(u) = ∇ ·

  • ∇u

|∇u|

  • ◮ Geometric interpretation: Image simplification is performed by

evolving level sets according to a purely geometric speed law V (x) = F

curv u(x)

  • These models are known as Euclidean geometric scale-spaces

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-63
SLIDE 63

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE Morphological scale-spaces Intrinsic scale-spaces Limitations

Euclidean geometric scale-spaces

◮ Classical models F(x) PDE type model c reaction (hyperbolic) differential mathematical morphology x diffusion (parabolic) Euclidean intrinsic scale-space c + αx reaction-diffusion (parabolic) entropic scale-space

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-64
SLIDE 64

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE Morphological scale-spaces Intrinsic scale-spaces Limitations

Morphological scale-spaces

◮ Multiscale structuring elements Sets: tB = {tb : b ∈ B} 0B = {0} Functions: gt(x) = tg

x

t

  • g0(0) = 0

Multiscale morphological operators Multiscale erosion Eg

t f = f ⊖ ˇ

g t with Eg

0 = Id

Multiscale dilation Dg

t f = f ⊕ ˇ

g t with Dg

0 = Id

Eg

t

and Dg

t are dual operators

Multiscale erosions/dilations w.r.t. structuring sets are derived by using flat structuring functions g : B → {0}

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-65
SLIDE 65

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE Morphological scale-spaces Intrinsic scale-spaces Limitations

Morphological scale-spaces

Scale-space structure (Brockett-Maragos) Multiscale dilations Dg

t (erosions Eg t ) are scale operators

◮ Semigroup: Dg

t1 Dg t2 = Dg t1+t2

and Eg

t1 Eg t2 = Eg t1+t2

for nonnegative concave functions g (i.e. with convex subgraph) ◮ Local comparison: Dg

t (Eg t ) is an increasing operator

◮ Smoothness: Modeling images as Lipschitz functions, the semi- group

Dg

t

  • t (

Eg

t

  • t ) has an infinitesimal generator ∂Dg (∂Eg)

Duality: ∂Eg = −∂Dg

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-66
SLIDE 66

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE Morphological scale-spaces Intrinsic scale-spaces Limitations

Morphological scale-spaces

Generating PDE (Brockett-Maragos-Boomgaard) Morphological scale-spaces are generated by eikonal equations i.e. (hyperbolic) Hamilton-Jacobi PDEs g dilation PDE V (x) unit ball ut = |∇u| 1 unit diamond ut = max |uxi|

1 |∇u| max |uxi|

unit cube ut =

  • |uxi|

1 |∇u|

  • |uxi|

unit disc ut =

  • |∇u|2 + g2(0)
  • 1 + g2(0)

|∇u|2

parabola ut = |∇u|2 |∇u|

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-67
SLIDE 67

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE Morphological scale-spaces Intrinsic scale-spaces Limitations

Morphological scale-spaces

◮ Implementation In order to handle shocks along image level sets concavities, the generators ∂Tu are discretized using entropic schemes based on lateral finite difference approximations of ∇u D+

x uij

= ui+1,j − uij D−

x uij

= uij − ui−1,j D+

y uij

= ui,j+1 − uij D−

y uij

= uij − ui,j−1 Explicit scheme uij(t + δt) = uij(t) + δt

∂Tu(t)

  • ij

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-68
SLIDE 68

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE Morphological scale-spaces Intrinsic scale-spaces Limitations

Morphological scale-spaces

◮ Implementation Spherical dilation ∂Tu = |∇u| |∇uij| =

ξ=x,y

  • max

D+

ξ uij, 0

2 + min D−

ξ uij, 0

2

Spherical erosion ∂Tu = −|∇u| |∇uij| =

ξ=x,y

  • min

D+

ξ uij, 0

2 + max D−

ξ uij, 0

2

Note: the same schemes are used for discretizing pressure forces in level set-based active contour models

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-69
SLIDE 69

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE Morphological scale-spaces Intrinsic scale-spaces Limitations

Euclidean intrinsic scale-space

Generating PDE (Morel-Osher-Sethian-Gage-Hamilton) The Euclidean intrinsic scale-space is generated by the (parabolic) Euclidean intrinsic heat equation ut = |∇u| ∇ ·

∇u

|∇u|

  • ◮ Geometric interpretation: Image simplification is performed by

evolving level sets according to the mean curvature motion xt = curv(u) n This is the speed law of a membrane

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-70
SLIDE 70

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE Morphological scale-spaces Intrinsic scale-spaces Limitations

Euclidean intrinsic scale-space

◮ Mean curvature motion decreases | curv(u) |

1 First, level sets are convexified

Discontinuities are instantly smoothed out

2 Once convex, level sets are

then contracted to a point which finally vanishes

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-71
SLIDE 71

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE Morphological scale-spaces Intrinsic scale-spaces Limitations

Gauge coordinates

◮ In global Cartesian coordinates, generating PDEs can be complex and does not highlight how image geometry is simplified ◮ Hence the idea of finding simpler expressions in a local coordinate system related to image geometry Relevant local frames should share the contrast- and isometry- invariance properties of scale operators These properties are verified by image level lines

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-72
SLIDE 72

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE Morphological scale-spaces Intrinsic scale-spaces Limitations

Gauge coordinates

level lines stream lines

x O y

t n u = cte

n = − ∇u |∇u| t = ∇⊥u |∇u| ◮ The local frame (t, n) induces a local coordinate system (ξ, η)

  • n the image grid

ξ (η ) is an arclength along level (stream) lines ∂ξ , ∂η are Lie derivatives

∂ξ , ∂η = t · ∇, n · ∇

  • Nicolas Rougon

IMA4509 | Scale-space & PDE filtering

slide-73
SLIDE 73

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE Morphological scale-spaces Intrinsic scale-spaces Limitations

Gauge coordinates

◮ 1st-order local image derivatives uξ = 0 uη = −|∇u| Gauge property: Choosing a reference for u yields a simplified representation of ∇u ◮ 2nd-order local image derivatives uξξ = |∇u|−2 uxxu2

y − 2 uxyuxuy + uyyu2 x

  • uξη = |∇u|−2 uxxuxuy − uyyuxuy + uxyu2

y − uxyu2 x

  • uηη = |∇u|−2 uxxu2

x + 2 uxyuxuy + uyyu2 y

  • Level line curvature

curv(u) = −uξξ uη Stream line curvature curv⊥(u) = −uξη uη

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-74
SLIDE 74

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE Morphological scale-spaces Intrinsic scale-spaces Limitations

Euclidean intrinsic vs linear scale-space

◮ Expressing generating PDEs in gauge coordinates highlights how scale-space acts on image local geometry scale-space Cartesian coordinates gauge coordinates linear ut = ∆u ut = uξξ + uηη intrinsic Euclidean ut = |∇u| ∇ ·

∇u

|∇u|

  • ut = uξξ

The isotropic heat equation has a diffusion component uηη across image level sets, inducing inter-region smoothing (biais) The Euclidean intrinsic heat equation performs anisotropic diffusion solely along image level sets

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-75
SLIDE 75

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE Morphological scale-spaces Intrinsic scale-spaces Limitations

Affine intrinsic scale-space

Generating PDE (Alvarez-Morel-Lions-Sapiro-Tannenbaum) There is a unique contrast- and affine-invariant scale-space, known as the affine intrinsic scale-space. Its generating PDE is the (parabolic) affine intrinsic heat equation ut = |∇u|

  • ∇ ·

∇u

|∇u|

1

3

◮ Geometric interpretation: Image simplification is performed by evolving level sets according to the affine mean curvature motion xt =

curv(u) 1

3 n Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-76
SLIDE 76

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE Morphological scale-spaces Intrinsic scale-spaces Limitations

Euclidean/affine intrinsic scale-spaces

◮ Implementation The scale-space generator ∂Tu is discretized by substituting in its expression finite difference approximations of (Du, D2u) (ux)ij = 1

2 (ui+1,j − ui−1,j)

(uxx)ij = ui+1,j − 2uij + ui−1,j (uxy)ij = ui+1,j+1 − ui−1,j+1 − ui+1,j−1 + ui−1,j−1 To avoid singularities, mean curvature motion is replaced by (discrete) linear diffusion (∂Tu = ∆u) when |∇u| ≪ 1 Explicit scheme uij(t + δt) = uij(t) + δt

∂Tu(t)

  • ij

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-77
SLIDE 77

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE Morphological scale-spaces Intrinsic scale-spaces Limitations

Limitations of geometric scale-spaces

◮ Though better suited to content-adapted simplification than the linear models, geometric scale-spaces suffer from artifacts Level set curvature-driven PDEs delocate / convexify image structures and smooth out corners / junctions Image level sets (and more generally, all image structures) are processed similarly whatever their contrast ◮ Scale operators capable of processing image content selectively, by preserving well-contrasted level sets while filtering the others, would allow for improved representations

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-78
SLIDE 78

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE Morphological scale-spaces Intrinsic scale-spaces Limitations

Limitations of geometric scale-spaces

Linear

separable Gaussian

Geometric

Euclidean intrinsic

Conservative

Perona-Malik

t 1 4 7 10 Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-79
SLIDE 79

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE Morphological scale-spaces Intrinsic scale-spaces Limitations

Limitations of geometric scale-spaces

Linear

separable Gaussian

Geometric

Euclidean intrinsic

Conservative

Perona-Malik

t 2 5 8 11 Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-80
SLIDE 80

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Generating PDE Morphological scale-spaces Intrinsic scale-spaces Limitations

Limitations of geometric scale-spaces

Linear

separable Gaussian

Geometric

Euclidean intrinsic

Conservative

Perona-Malik

t 2 5 8 11 Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-81
SLIDE 81

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Anisotropic diffusion Biased anisotropic diffusion Shock filters

Outline

1

Foundations

2

Linear scale-space

3

Geometric scale-spaces

4

Conservative scale-spaces

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-82
SLIDE 82

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Anisotropic diffusion Biased anisotropic diffusion Shock filters

Anisotropic diffusion

Generating PDE (Perona-Malik) The (parabolic) variable-conduction heat equation ut = ∇ ·

g (|∇u|) ∇u

  • generates an isometry-invariant scale-space for any given

C1-continuous, positive, decreasing function g such that

1

g(0) = 1

2

lim

x→+∞ g(x) = 0

This PDE is known as anisotropic diffusion (it is in fact an isotropic inhomogeneous diffusion equation) Diffusion is governed by the conduction function g xg(x) is known as the flux function

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-83
SLIDE 83

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Anisotropic diffusion Biased anisotropic diffusion Shock filters

Anisotropic diffusion

◮ Generating PDE in gauge coordinates ut = g(uη)uξξ +

uη g(uη) ′uηη

Weak diffusion across image level sets occurs for high-contrast values, even though lim

x→+∞[ xg(x) ]′ = 0

Edge blurring is avoided if diffusion along level sets dominates. This is ensured if the conduction function g verifies lim

x→+∞

g(x) [ xg(x) ]′ = 0

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-84
SLIDE 84

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Anisotropic diffusion Biased anisotropic diffusion Shock filters

Anisotropic diffusion

◮ Convex flux result in smoothing x xg(x) name g(x) Green

tanh x x

L1 - L2

1 √ 1+x2

Fair

1 1+x

Total variation

1 x

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-85
SLIDE 85

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Anisotropic diffusion Biased anisotropic diffusion Shock filters

Anisotropic diffusion

◮ Nonconvex flux result in joint smoothing / contrast enhancement K xg(x) x name g(x) Perona-Malik e−( x

K ) 2

Lorentzian

1 1+( x

K ) 2

Geman-McClure

1

  • 1+( x

K ) 22

Tuckey

x ≤ K x > K 1 − x

K

22

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-86
SLIDE 86

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Anisotropic diffusion Biased anisotropic diffusion Shock filters

Anisotropic diffusion

◮ Nonconvex flux result in joint smoothing / contrast enhancement K xg(x) x The hyperparameter K acts as a contrast threshold |∇u| ≤ K Image is viewed as low-texture and smoothed |∇u| > K Image is viewed as salient edge and enhanced (the PDE behaves locally as an inverse heat equation)

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-87
SLIDE 87

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Anisotropic diffusion Biased anisotropic diffusion Shock filters

Anisotropic diffusion

◮ Influence of hyperparameter K K = 100 K = 400

t 2 5 8 11 Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-88
SLIDE 88

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Anisotropic diffusion Biased anisotropic diffusion Shock filters

Anisotropic diffusion

◮ Influence of hyperparameter K K = 100 K = 400

t 2 5 8 11 Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-89
SLIDE 89

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Anisotropic diffusion Biased anisotropic diffusion Shock filters

Properties of anisotropic diffusion

◮ Simplification: Anisotropic diffusion decreases image Lp-norms, centered statistical moments and Shannon entropy ◮ Well-posedness: The variable-conduction heat equation is well- posed iff. the flux function is convex For nonconvex flux functions, inverse diffusion occurs when |∇u| > K, generating local instability Joint image smoothing / contrast enhancement is ill-posed

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-90
SLIDE 90

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Anisotropic diffusion Biased anisotropic diffusion Shock filters

Well-posed anisotropic diffusion

Generated PDE (Catté-Dibos-Whitaker-Pizer) The (parabolic) variable-conduction heat equation ut = ∇ ·

g (|∇Gσ ⋆ u|) ∇u

  • generates an isometry-invariant scale-space for any given σ > 0

and C1-continuous, positive, decreasing function g such that

1

g(0) = 1

2

lim

x→+∞ g(x) = 0

This PDE is well-posed Unconditional well-posedness is obtained by computing conduction from smooth contrast estimates This model is known as image selective smoothing

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-91
SLIDE 91

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Anisotropic diffusion Biased anisotropic diffusion Shock filters

Properties of anisotropic diffusion

◮ Conservation of mean luminance 1 |Ω|

  • Ω

u(x, t) dx = 1 |Ω|

  • Ω

u(x, 0) dx Variable-conduction diffusion scale-spaces are also known as conservative scale-spaces ◮ Trivial asymptotics: Anisotropic diffusion yields uniform images at the large scale limit lim

t→+∞ u(x, t) =

1 |Ω|

  • Ω

u(x, 0) dx Diffusion must be stopped before excessive loss of detail No optimal stopping criterion is currently available

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-92
SLIDE 92

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Anisotropic diffusion Biased anisotropic diffusion Shock filters

Biased anisotropic diffusion

Generating PDE (Nordström) The (parabolic) variable-conduction heat equation ut = ∇ ·

g (|∇u|) ∇u + λ(u0 − u)

generates an isometry-invariant scale-space for any given λ > 0 and C1-continuous, positive, decreasing function g such that

1

g(0) = 1

2

lim

x→+∞ g(x) = 0

λ(u0 − u) is a data link term, constraining filtered images u to remain similar to the original image u0

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-93
SLIDE 93

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Anisotropic diffusion Biased anisotropic diffusion Shock filters

Biased anisotropic diffusion

Variational formulation (Barlaud-Aubert) Biased anisotropic diffusion corresponds to solving a regularized denoising problem by iteratively minimizing the energy E(u) =

  • Ω

ϕ (|∇u|) dx + λ 2

  • Ω

(u0 − u)2 dx where ϕ is a 1st-order discontinuity-preserving stabilizer Variational derivative ∂uE(u) = −∇ ·

ϕ′(|∇u|)

|∇u| ∇u

  • − λ(u0 − u)

Biased anisotropic diffusion arises as a gradient descent ut = −∂uE(u) with conduction defined as g(x) = ϕ′(x)

x

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-94
SLIDE 94

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Anisotropic diffusion Biased anisotropic diffusion Shock filters

Biased anisotropic diffusion

◮ Implementation Finite difference discretization of the diffusion term |∇u| is approximated by absolute lateral differences ∇ ·

g (|∇u|) ∇u

  • ij ≈
  • ξ = x,y

D−

ξ

  • g

|D+

ξ uij|

D+

ξ uij

  • Denoting

∆n

ij u = ui−1,j − uij

∆w

ij u = ui,j−1 − uij

∆s

ij u = ui+1,j − uij

∆e

ij u = ui,j+1 − uij

this rewrites as ∇ ·

g (|∇u|) ∇u

  • ij ≈
  • ξ = n,s,w,e

g(|∆ξ

ij u|) ∆ξ ij u

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-95
SLIDE 95

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Anisotropic diffusion Biased anisotropic diffusion Shock filters

Biased anisotropic diffusion

◮ Implementation (cont’d) Finite difference discretization of the diffusion term This corresponds to a convolution ∇ ·

g (|∇u|) ∇u

  • ij ≈ K(uij) ⋆ uij

with an image-based diffusion kernel K(u) K(uij) =

   

g

|∆n

iju|

  • g

|∆w

ij u|

  • − g(|∆ξ

iju|)

g

|∆e

iju|

  • g

|∆s

iju|

  

Note: In the linear diffusion limit (ϕ(x) = x2 i.e. g(x) = 1), K(u) reduces to the 4-connected Laplacian kernel ∆4

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-96
SLIDE 96

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Anisotropic diffusion Biased anisotropic diffusion Shock filters

Biased anisotropic diffusion

◮ Implementation (cont’d) Finite difference discretization of the biaised diffusion term

∂Tu

  • ij ≈ K(uij) ⋆ uij + λ(u0 − u)ij

Finite difference discretization of ut (ut)ij = uij(t + δt) − uij(t) δt Explicit scheme uij(t + δt) = uij(t) + δt

∂Tu(t)

  • ij

◮ Under the CFL condition δt ≤

1 4, this scheme is stable and

verifies a maximum principle

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-97
SLIDE 97

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Anisotropic diffusion Biased anisotropic diffusion Shock filters

Shock filters

Generating PDE (Rudin-Osher) For any 2nd-order diffusive (elliptic) operator L, the (hyperbolic) PDE ut = −|∇u| sign

L(u)

  • is well-posed and yields piecewise C0 images at the large scale limit

L behaves as a 2nd-order edge detector. Classical choices are L(u) = Gσ ⋆ ∆u L(u) = (∇u)T D2u ∇u = uηη Image level lines are pushed towards edges with unit speed V (x) = sign

L(u)(x)

  • Discontinuities are created as shocks along edges

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-98
SLIDE 98

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Anisotropic diffusion Biased anisotropic diffusion Shock filters

Shock filters

◮ Generalization A broad family of shock filters is derived by replacing the sign function by an arbitrary Lipschitz function F s.t. xF(x) > 0 ut = −|∇u| F

L(u)

  • Given 0 < α ≤ 1, the well-posed (hyperbolic) PDE

ut = −

  • α
  • 1 + |∇u|2 + (1 − α) |∇u|
  • F

L(u)

  • yields piecewise C1 images at the large scale limit

In both cases, the backward PDEs are also well-posed

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-99
SLIDE 99

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography Anisotropic diffusion Biased anisotropic diffusion Shock filters

Conclusion

◮ Scale-space theory provides a well-established framework for describing images/shapes w.r.t. level of detail The linear scale-space is a universal visual front-end Nonlinear scale-spaces provide geometry-preserving models tailored to specific invariance constraints ◮ Scale-spaces enable hierarchical implementations of a variety of image/shape understanding problems Restoration/enhancement Feature extraction Segmentation Grouping/decomposition into parts Motion analysis Matching Shape from X . . .

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-100
SLIDE 100

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography

Bibliography

  • T. Lindeberg

Scale space theory in computer vision Kluwer Academic Publishers, 1994 L.M.J. Florack Image structure Kluwer Academic Publishers, 1997 B.M. ter Haar Romeny Geometry-driven diffusion in computer vision Kluwer Academic Publishers, 1994

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-101
SLIDE 101

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography

Bibliography

B.M. ter Haar Romeny Front-end vision and multiscale image analysis - Introduction to scale-space theory Kluwer Academic Publishers, 1997

  • J. Sporring, M. Nielsen, L.M.J. Florack, P. Johansen

Gaussian scale-space theory Kluwer Academic Publishers, 1997

Nicolas Rougon IMA4509 | Scale-space & PDE filtering

slide-102
SLIDE 102

Foundations Linear scale-space Geometric scale-spaces Conservative scale-spaces Bibliography

Scale-space theory and PDE-based image filtering

Nicolas Rougon

Institut Mines-Télécom / Télécom SudParis ARTEMIS Department; CNRS UMR 8145 nicolas.rougon@telecom-sudparis.eu

May 17, 2020

Nicolas Rougon IMA4509 | Scale-space & PDE filtering