A Probabilistic U-Net for Segmentation of Ambiguous Images Simon A. - - PowerPoint PPT Presentation

▶

a probabilistic u net for segmentation of ambiguous images

A Probabilistic U-Net for Segmentation of Ambiguous Images Simon A. - - PowerPoint PPT Presentation

Jun 16, 2023 342 likes •670 views

A Probabilistic U-Net for Segmentation of Ambiguous Images Simon A. A. Kohl 1*,2 , Bernardino Romera-Paredes 1 , Clemens Meyer 1 , Jeffrey De Fauw 1 , Joseph R. Ledsam 1 , Klaus H. Maier-Hein 2 , S. M. Ali Eslami 1 , Danilo Jimenez Rezende 1 , Olaf

slide-1

SLIDE 1

A Probabilistic U-Net for Segmentation of Ambiguous Images

Simon A. A. Kohl1*,2, Bernardino Romera-Paredes1, Clemens Meyer1, Jeffrey De Fauw1, Joseph R. Ledsam1, Klaus H. Maier-Hein2,

S. M. Ali Eslami1, Danilo Jimenez Rezende1, Olaf Ronneberger1

1DeepMind 2German Cancer Research Center

*work done during an internship at DeepMind

slide-2

SLIDE 2

A Probabilistic U-Net for Segmentation of Ambiguous Images

Simon A. A. Kohl1*,2, Bernardino Romera-Paredes1, Clemens Meyer1, Jeffrey De Fauw1, Joseph R. Ledsam1, Klaus H. Maier-Hein2,

S. M. Ali Eslami1, Danilo Jimenez Rezende1, Olaf Ronneberger1

1DeepMind 2German Cancer Research Center

Poster #127 Medical Imaging Workshop Talk: Sat, Dec 8, 9:45 am

*work done during an internship at DeepMind

slide-3

SLIDE 3

3

Images are often Ambiguous

slide-4

SLIDE 4

4

Images are often Ambiguous

Potential Cancer Expert Graders

slide-5

SLIDE 5

5

Images are often Ambiguous

Potential Cancer Expert Graders Segmentations from our model (U-Net + conditional VAE)

slide-6

SLIDE 6

6

U-Net Image

Deterministic U-Net

Inference

slide-7

SLIDE 7

7

Probabilistic U-Net

U-Net Latent Space Prior Net Image 𝛎,𝞃prior

Sampling

slide-8

SLIDE 8

8

Probabilistic U-Net

U-Net Latent Space Prior Net Image 𝛎,𝞃prior

Sampling

1

Sample

*

1

slide-9

SLIDE 9

9

Probabilistic U-Net

U-Net Latent Space Prior Net Image 𝛎,𝞃prior

Sampling

1

*

1 2 2

Sample

slide-10

SLIDE 10

10

Probabilistic U-Net

U-Net Latent Space Prior Net Image 𝛎,𝞃prior

1 3 2

*

1

Sampling

2 3

Sample

slide-11

SLIDE 11

11

Probabilistic U-Net

U-Net Image Sample 𝐴 Sample Groundtruth Cross- Entropy Prior Net 𝛎,𝞃prior

Training

slide-12

SLIDE 12

12

Probabilistic U-Net

U-Net Image Sample 𝐴 Sample Groundtruth Cross- Entropy Prior Net 𝛎,𝞃prior

Training

Position in Latent Space for this GT example?

slide-13

SLIDE 13

13

Probabilistic U-Net

U-Net Image Posterior Net Sample 𝐴 Sample Groundtruth KL Cross- Entropy Prior Net Latent Space 𝛎,𝞃prior 𝛎,𝞃post

Training

slide-14

SLIDE 14

14

Image

Latent Space Analysis

Probabilistic U-Net

slide-15

SLIDE 15

15

Image

Latent Space Analysis

Probabilistic U-Net

slide-16

SLIDE 16

16

Graders Image

Latent Space Analysis

Probabilistic U-Net 1 3 2

slide-17

SLIDE 17

17

Lung Abnormalities Segmentation: Quantitative Results

Energy distance (lower is better)

slide-18

SLIDE 18

18

Lung Abnormalities Segmentation: Quantitative Results

Energy distance (lower is better)

1

slide-19

SLIDE 19

19

Lung Abnormalities Segmentation: Quantitative Results

Energy distance (lower is better)

1 4

slide-20

SLIDE 20

20

Lung Abnormalities Segmentation: Quantitative Results

Energy distance (lower is better)

1 4 8

slide-21

SLIDE 21

21

Lung Abnormalities Segmentation: Quantitative Results

Energy distance (lower is better)

1 4 8 16

slide-22

SLIDE 22

22

Lung Abnormalities Segmentation: Quantitative Results

Energy distance (lower is better)

slide-23

SLIDE 23

23

Lung Abnormalities Segmentation: Quantitative Results

Energy distance (lower is better)

slide-24

SLIDE 24

24

Lung Abnormalities Segmentation: Quantitative Results

Energy distance (lower is better)

slide-25

SLIDE 25

25

Cityscapes segmentation: Qualitative Results

sidewalk person car veget.

Input Image Ground-truth Grader Styles Samples (Probabilistic U-Net)

person 2 car 2

veget. 2

sidewalk 2 47 % 41 % 35 % 29 %

stochastic flips:

road road 2 24 %

slide-26

SLIDE 26

26

Cityscapes segmentation: Qualitative Results

sidewalk person car veget.

Input Image Ground-truth Grader Styles Samples (Probabilistic U-Net)

person 2 car 2

veget. 2

sidewalk 2 47 % 41 % 35 % 29 %

stochastic flips:

road road 2 24 %

slide-27

SLIDE 27

27

Cityscapes segmentation: Quantitative Results

slide-28

SLIDE 28

28

Conclusions

Learn conditional probability over segmentation maps
Each sample is a valid & consistent segmentation
The likelihoods are well calibrated
Works on large-scale, real-world data
Can also be trained with a uni-modal GT
Can be used to asses annotations under the model

code: github.com/SimonKohl/probabilistic_unet

slide-29

SLIDE 29

A Probabilistic U-Net for Segmentation of Ambiguous Images

Simon A. A. Kohl1*,2, Bernardino Romera-Paredes1, Clemens Meyer1, Jeffrey De Fauw1, Joseph R. Ledsam1, Klaus H. Maier-Hein2,

S. M. Ali Eslami1, Danilo Jimenez Rezende1, Olaf Ronneberger1

1DeepMind 2German Cancer Research Center

Poster #127 Medical Imaging Workshop Talk: Sat, Dec 8, 9:45 am

*work done during an internship at DeepMind

slide-30

SLIDE 30

30

Probabilistic Segmentation: Clinical Use-Cases

Best-fit could be picked by clinician and adjusted if necessary.
Hypotheses could be propagated into next diagnostic pipeline steps.
Hypotheses could inform actions to resolve ambiguities.

slide-31

SLIDE 31

31

Evaluation Metric for Quantitative Comparison

We use the Energy Distance1 statistic (aka MMD): where d(x,y) = 1 - IoU(x,y) and

Pout Pgt

1 Székely, G.J., Rizzo, M.L.: Energy statistics: A class of statistics based on distances.

Journal of statistical planning and inference 143(8) (2013) 1249–1272

slide-32

SLIDE 32

32

Baselines

1 2 m

U-Net Ensemble

1 2 m

M-Heads Dropout U-Net

1,2,3,... U-Net Normal Prior Sample 𝐴1,𝐴2,𝐴3,... 1 3 2

Image2Image VAE