Roto-Translation Equivariant Convolution Networks for Medical Image - - PowerPoint PPT Presentation

▶
roto translation equivariant convolution networks for
SMART_READER_LITE
LIVE PREVIEW

Roto-Translation Equivariant Convolution Networks for Medical Image - - PowerPoint PPT Presentation

Aug 13, 2022 454 likes •942 views

Roto-Translation Equivariant Convolution Networks for Medical Image Analysis Erik J Bekkers, Remco Duits Dep. Math. & Computer Science CASA TU/e Lie Analysis Based on work by Bekkers 1 , Lafarge 2 , Veta 2 , Eppenhof 2 , Pluim 2 , Duits 1

slide-1
SLIDE 1

Roto-Translation Equivariant Convolution Networks for Medical Image Analysis

Erik J Bekkers, Remco Duits

  • Dep. Math. & Computer Science CASA TU/e

Based on work by Bekkers1, Lafarge2, Veta2, Eppenhof2, Pluim2, Duits1 (MICCAI 2018)

_______2019/05/04________Imaging & Machine Learning_______Institut Henri Poincaré, Paris, France________

1Department of Mathematics and Computer Science, and 2Department of Biomedical Engineering,

Eindhoven University of Technology Lie Analysis

slide-2
SLIDE 2

2

Presentation outline

slide-3
SLIDE 3

Presentation outline

  • Motivation 1: Geometric image analysis via orientation scores needs automation.
  • Motivation 2: Machine learning needs group equivariance.
  • Overview of related work.
  • Theoretical background:
  • Neural networks (NNs)
  • Convolutional neural networks (CNNs)
  • Group convolutional neural networks (G-CNNs)

General Theorem on Equivariant linear operators Architecture G-CNNs.

  • Results on 3 different medical imaging applications.
  • Conclusion.

3

slide-4
SLIDE 4

4

Equivariant image analysis

via orientation scores

slide-5
SLIDE 5

Invertible Orientation scores

PhD Thesis R. Duits 2005 PhD Thesis E.M. Franken 2009 PhD Thesis E.J. Bekkers 2017 PhD Thesis J.M. Portegies 2018 Image Orientation Score (real part)

Curved & Torqued Geometry of roto-translation group SE(2) visible in Score:

Image OS Cake wavelets Gabor wavelets Moving frame

PDEs For tracking and regularization

Image OS

5

slide-6
SLIDE 6

SE(d) equivariant processing via orientation scores

6

JMIV 2018 SIIMS 2016 JDCS 2016 JMIV 2018

slide-7
SLIDE 7

Diffusion & Brownian Motions in Roto-Translation Group

2nd Workshop IHP: “PDEs on the Homogeneous Space of Positions and Orientations.” R. Duits Ne Next: : Machine Learning to train such kernels and their offsets. 2014-onwards incl. IEEE-PAMI E.J. Bekkers & R. Duits et al.

ISKR BM3D

7

Entropy 2019 DGA 2018 QAM-AMS 2008, 2010

slide-8
SLIDE 8

Template matching via Group convolutions

ICIAR 2014 & PAMI 2018 Bekkers-Loog-tHRomeny-Duits

  • riented

structures (vessels) isotropic structures (optic disk)

! " #

image score

State-of-the-Art Results (99,8 % success rate) on 3 Detection Tasks. E.g. Optic Disc Detection via Template Matching using group convolutions in SE(2)

Template

Special Case 2 layer Group CNN !

8

slide-9
SLIDE 9
  • Minimize Functional:
  • With Sub-Riemannian Brownian prior
  • Expansion in B-splines

Accurate: Comparison to exact sol’s Duits

Convex-problem. Train parameters via Generalized Cross-Validation:

training label patch

2 layer G-CNN: ICIAR 2014 & PAMI 2018 Bekkers-Loog-Duits

9

Output: 1 delta patch

slide-10
SLIDE 10

10

Motivation 1

Extend from 2 Layer G-CNN for template matching in OS to

Deep Learning in OS

slide-11
SLIDE 11

Related work Group equivariant networks

Group convolution networks (domain extension) Steerable filter networks (co-domain extension) LeCun et al 1990 ℤ" translation networks Mallat et al. 2013, 2015 SE(2) Scattering transform & SVM Bekkers et al. 2014-2018 SE(2) via B-splines, 2 layer G-CNN Cohen-Welling 2016 p4m via 90o rotations + flips + theory! Dieleman et al. 2016 p4m via 90o rotations + flips Weiler et al. 2017 SE(2) via circular harmonics Zhou et al. 2017 SE(2) via bilinear interpolation Bekkers et al. 2018 SE(2) via bilinear interpolation Hoogeboom et al. 2018 S(2,6) hexagonal grids Winkels-Cohen 2018 SE(3,N) + m 90o rotations + flips Worrall-Brostow 2018 SE(3,N) 90o rotations Cohen et al. 2018 SO(3) via spherical harmonics Worrall et al. 2017 SE(2) irrep Marcos et al. 2017 SE(2) vector field networks Kondor 2018 SE(3) irrep, N-body nets Thomas et al. 2018 SE(3) irrep, point clouds Weiler et al. 2018 SE(3) irrep Esteves SO(3)/SO(2) irrep Kondor-Trivedi 2018 SO(d) irrep (on compact quotient sp.)

Continuous Discrete

11

slide-12
SLIDE 12

12

Motivation 2

Roto-Translation Covariant Convolution Neural Networks for Medical Image Analysis

slide-13
SLIDE 13

Many computer vision and MedIA problems require invariance/equivariance properties (e.g. w.r.t. rotation)

Rotation-invariant detection of pathological cells (mitotic figure) in histopathology

13

slide-14
SLIDE 14

Transformation invariance

“healthy cell”

Classification via CNN:

Label (e.g. 1: healthy cell, 0: pathological cell) Input image CNN

Transformation:

Geometric transformation by (e.g. rotation)

14

slide-15
SLIDE 15

Transformation invariance by data augmentation

Example: vessel segmentation Training set (naïve example of only vertically aligned vessels)

Blood vessels (vertically aligned) Background Conv layer “vessel” “background”

A simple network will do

“vessel”

??

15

slide-16
SLIDE 16

Transformation invariance by data augmentation

Example: vessel segmentation Expand training set with rotated copies -> data augmentation

Blood vessels Background Conv layer “vessel” “background”

A more complex network is required

Now you learn rotated versions of the same feature..

16

slide-17
SLIDE 17

Redundancy in feature representations

Learned convolution filters in the first layer ImageNet Challenge

17

slide-18
SLIDE 18

Symmetries in Medical Imaging

Problems with classical CNNs:

  • No guarantee of equivariance (other then translation).
  • Redundancy in feature representation.
  • Artificially create extra data samples by data augmentation.

Solution: Group convolutional neural networks: G-CNNs Moreover, G-CNNs:

  • Exploit symmetries in data.
  • Weight sharing.
  • Perform better by not having to spend valuable

network capacity on learning geometric properties.

18

slide-19
SLIDE 19

19

Theory: From NNs to CNNs to G-CNNs

NNs CNNs G-CNNs

Add geometric structure (local spatial coherence, weight sharing, …) Adding more geometric structure (beyond translation equivariance)

slide-20
SLIDE 20

Problem description

Given a training set consisting of pairs of inputs and desired outputs , find a function , parameterized by , that best maps each input to a desired output. Here “best” is quantified by (local) minima of a loss: computed by stochastic gradient descent (backward propagation via chain-law)

, 0: normal , 0: normal , 1: mitotic … 0.14

Training set Testing

“probably a healthy cell”

20

slide-21
SLIDE 21

Neural Network:

  • A Neural Network is a composition of operators:
  • In which each operator (“layer”) has the following form:

21

slide-22
SLIDE 22

A classical neural network

0,2 9 7 5 3 1

Class probability

Probability

2D function/image 2D array 1D vector With “soft-max” Boltzmann distr.

22

slide-23
SLIDE 23

A classical neural network

23

slide-24
SLIDE 24

A fully connected neural layer

  • Too many degrees of freedom
  • Does not exploit structure in data

24

slide-25
SLIDE 25

A convolution layer

+ Localized transformations + Shift equivariance + Sparsification of the linear operator + Weight sharing

25

slide-26
SLIDE 26

26

Theory: From NNs to CNNs to G-CNNs

NNs CNNs G-CNNs

slide-27
SLIDE 27

Input and output spaces

Input vector Output vector Convolution layer

27

slide-28
SLIDE 28

Convolution layer (cross-correlation layer)

Translation by x

28

slide-29
SLIDE 29

Convolution layer (cross-correlation layer)

Translation by x template matching

29

slide-30
SLIDE 30

A linear mapping parameterized by a collection

  • f convolution filters

Convolution layers on ℝ"

30

slide-31
SLIDE 31

A typical CNN architecture

31

LeCun et al. 1989-1998

slide-32
SLIDE 32

32

Theory: From NNs to CNNs to G-CNNs

NNs CNNs G-CNNs

slide-33
SLIDE 33

The translation group

33

slide-34
SLIDE 34

The roto-translation group SE(2)

34

slide-35
SLIDE 35

Group Representations of SE(2)

Shift-twist

The representation of SE(2) on The representation of SE(2) on

35

slide-36
SLIDE 36

Group correlation layers

Lifting layer: G-correlation layer:

36

slide-37
SLIDE 37

Group correlation layers

Lifting layer: G-correlation layer: SE(2) equivariance

37

slide-38
SLIDE 38

38

Theorem on Equivariant Linear Operators motivating the G-CNN Design

slide-39
SLIDE 39

39

slide-40
SLIDE 40

40

slide-41
SLIDE 41

G-CNN design for rotation invariant patch classification

“normal” (0) vs “mitotic” (1) Class probability L i f t i n g c

  • n

v G

  • c
  • n

v G

  • c
  • n

v G

  • c
  • n

v P r

  • j

e c t i

  • n

l a y e r F u l l y c

  • n

n e c t e d

  • u

t p u t l a y e r Input image

  • Thm. equivariant linear operator design !

(in practice interleafed ReLu: not affecting equiv.) Max-pooling over rotations only

  • Equivariance. Better choices…

Bekkers et al. MICCAI 2018 ICIAR 2014 (2 layers)

41

slide-42
SLIDE 42

42

Steerable Implementations possible via Fourier Transform on the homogeneous space of positions and orientations:

slide-43
SLIDE 43

43

Results

slide-44
SLIDE 44

Example (rotation equivariance and invariance)

44

Invariance !

slide-45
SLIDE 45

Results

45

histopathology Optical image of eye Electron microscopy Same capacity But no waist as N increases.

slide-46
SLIDE 46

46

Conclusion

slide-47
SLIDE 47

Conclusion

47