4 Deep Generative Models BVM 2018 Tutorial: Advanced Deep Learning - - PowerPoint PPT Presentation

▶
4 deep generative models
SMART_READER_LITE
LIVE PREVIEW

4 Deep Generative Models BVM 2018 Tutorial: Advanced Deep Learning - - PowerPoint PPT Presentation

Nov 19, 2023 813 likes •1.48k views

4 Deep Generative Models BVM 2018 Tutorial: Advanced Deep Learning Methods Jens Petersen Dept. of Neuroradiology, Heidelberg University Hospital Div. of Medical Image Computing, DKFZ Heidelberg Faculty of Physics & Astronomy, Heidelberg

slide-1
SLIDE 1

4 Deep Generative Models

Jens Petersen

  • Dept. of Neuroradiology, Heidelberg University Hospital
  • Div. of Medical Image Computing, DKFZ Heidelberg

Faculty of Physics & Astronomy, Heidelberg University

BVM 2018 Tutorial: Advanced Deep Learning Methods

slide-2
SLIDE 2

2

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Data Shortage Transfer learning Noisy labels and data

Challenges in MIC

slide-3
SLIDE 3

3

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Basic Principle of Generative Models

Assumption Observations X generated from latent variables Z via mapping f

Z X f(x|z)

slide-4
SLIDE 4

4

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Basic Principle of Generative Models

Assumption Observations X generated from latent variables Z via mapping f Goal

  • Be able to generate more samples that

follow distribution of X

  • Z interpretable in some way

Z X f(x|z)

slide-5
SLIDE 5

5

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Basic Principle of Deep Generative Models Z X f(x|z)

[pexels.com, pixabay.com, pngimg.com]

slide-6
SLIDE 6

6

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Basic Principle of Deep Generative Models Z X f(x|z)

Realism Panda-nessTM

[pexels.com, pixabay.com, pngimg.com]

slide-7
SLIDE 7

Generative Adversarial Networks

slide-8
SLIDE 8

8

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

slide-9
SLIDE 9

9

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

[https://twitter.com/goodfellow_ian]

slide-10
SLIDE 10

10

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Basic GAN Layout

[https://deeplearning4j.org/generative-adversarial-network] [1] Generative Adversarial Networks, Goodfellow et al., 2014, NIPS

slide-11
SLIDE 11

11

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

GAN Learning Objective

slide-12
SLIDE 12

12

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

D(real) → 1 D(fake) → 0

GAN Learning Objective

slide-13
SLIDE 13

13

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

D(real) → 1 D(fake) → 0 D(fake) → 1

GAN Learning Objective

slide-14
SLIDE 14

14

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Trying

  • to find saddle point

→ Very hard to optimize Lot

  • f work on different objectives and „tricks“ for training

D(real) → 1 D(fake) → 0 D(fake) → 1

[2] Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks Radford et al., 2015, arXiv:1511.06434 [3] Are GANs Created Equal? A Large Scale Study, Lucic et al., 2017, arXiv:1711.10337

GAN Learning Objective

slide-15
SLIDE 15

15

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Original Examples

slide-16
SLIDE 16

16

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Important Concepts Conditional GAN

General case Generative models make no default assumptions for p(z) → Could be random noise and/or real data

slide-17
SLIDE 17

17

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Important Concepts Conditional GAN

[4] Adversarial Networks for the Detection of Aggressive Prostate Cancer, Kohl et al., 2017, NIPS Workshop

slide-18
SLIDE 18

18

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Important Concepts Conditional GAN

[4] Adversarial Networks for the Detection of Aggressive Prostate Cancer, Kohl et al., 2017, NIPS Workshop

slide-19
SLIDE 19

19

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Important Concepts CycleGAN

Assumption Have two unpaired sets A,B of images with some set- specific characteristic (e.g. photos & paintings) Goal Be able to transform image so it looks like images in different set

slide-20
SLIDE 20

20

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Important Concepts CycleGAN

Assumption Have two unpaired sets A,B of images with some set- specific characteristic (e.g. photos & paintings) Goal Be able to transform image so it looks like images in different set Naive Approach GANs that take images from A(B) and create images that similar to others from B(A)

slide-21
SLIDE 21

21

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Important Concepts CycleGAN

Assumption Have two unpaired sets A,B of images with some set- specific characteristic (e.g. photos & paintings) Goal Be able to transform image so it looks like images in different set Naive Approach GANs that take images from A(B) and create images that similar to others from B(A) → no guarantee that output looks similar to input

slide-22
SLIDE 22

22

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Important Concepts CycleGAN

[5] Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, Zhu et al., 2017, arXiv:1703.10593

slide-23
SLIDE 23

23

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Important Concepts CycleGAN

L1-Norm Cycle consistency loss

[5] Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, Zhu et al., 2017, arXiv:1703.10593

slide-24
SLIDE 24

24

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Important Concepts CycleGAN

[5] Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, Zhu et al., 2017, arXiv:1703.10593

slide-25
SLIDE 25

25

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Examples Progressive Growing

slide-26
SLIDE 26

26

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Examples Progressive Growing

slide-27
SLIDE 27

27

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Examples Progressive Growing

slide-28
SLIDE 28

28

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Examples Progressive Growing

Samples Nearest Neighbours

slide-29
SLIDE 29

29

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

  • Pixel similarity
  • mean squared error (= L2 norm)
  • other norms

Image Similarity

slide-30
SLIDE 30

30

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

  • Pixel similarity
  • mean squared error (= L2 norm)
  • other norms
  • Semantic similarity
  • Inception score (score for entire model)
  • Combined distance of multiple feature layers in discriminator
  • Human evaluation (e.g. Mechanical Turk)

Image Similarity

slide-31
SLIDE 31

31

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Examples MRI to CT Image Synthesis

slide-32
SLIDE 32

32

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Examples MRI to CT Image Synthesis

FCN architecture

slide-33
SLIDE 33

33

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Examples MRI to CT Image Synthesis

Combined adversarial & MSE loss FCN architecture

slide-34
SLIDE 34

34

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Examples MRI to CT Image Synthesis

slide-35
SLIDE 35

35

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Examples Domain Transfer for Lesion Segmentation

slide-36
SLIDE 36

36

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Examples Domain Transfer for Lesion Segmentation

Assumption (X, Y) in source domain, (X*) in target domain

slide-37
SLIDE 37

37

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Examples Domain Transfer for Lesion Segmentation

Assumption (X, Y) in source domain, (X*) in target domain ... + GE + Lesion Segmentation in source ... + SWI in target Goal Segmentation in target domain

slide-38
SLIDE 38

38

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Examples Domain Transfer for Lesion Segmentation

slide-39
SLIDE 39

39

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Examples Domain Transfer for Lesion Segmentation

DeepMedic architecture

slide-40
SLIDE 40

40

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Examples Domain Transfer for Lesion Segmentation

DeepMedic architecture Auxiliary adversarial loss ensures domain invariant feature maps

slide-41
SLIDE 41

41

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Examples Domain Transfer for Lesion Segmentation

Higher is better

slide-42
SLIDE 42

42

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Summary GANs

✔ High-quality, high-resolution outputs possible ✔ Adversarial training extremely versatile ✖ Difficult to train ✖ No inference (latent representation from data)

slide-43
SLIDE 43

Variational Autoencoders

slide-44
SLIDE 44

44

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Probabilistic Perspective Z X pΦ(x|z) qθ(z|x) p(z)

[6] Auto-encoding variational Bayes, Kingma & Welling, 2014, ICLR [7] Stochastic backpropagation and approximate inference in deep generative models, Rezende et al., 2014, ICML

slide-45
SLIDE 45

45

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Probabilistic Perspective Z X pΦ(x|z) qθ(z|x) p(z)

[6] Auto-encoding variational Bayes, Kingma & Welling, 2014, ICLR [7] Stochastic backpropagation and approximate inference in deep generative models, Rezende et al., 2014, ICML

map to distribution parameters

slide-46
SLIDE 46

46

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

It looks like an autoencoder

[http://kvfrans.com/variational-autoencoders-explained/]

slide-47
SLIDE 47

47

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Reparametrization Trick !~ℱ !; % & = ((!) +& +% = +( +! +! +%

slide-48
SLIDE 48

48

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Reparametrization Trick !~ℱ !; % & = ((!) +& +% = +( +! +! +% ! = ,(%; -)

  • ~ℱ∗ -; %∗

+& +% = +( +, +, +%

slide-49
SLIDE 49

49

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Reparametrization Trick !~ℱ !; % & = ((!) +& +% = +( +! +! +% ! = ,(%; -)

  • ~ℱ∗ -; %∗

+& +% = +( +, +, +% !~Ν !; 0, 2 ! = 0 + 2 ∗ -

  • ~Ν -; 0,1
slide-50
SLIDE 50

50

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

VAE Learning Objective

slide-51
SLIDE 51

51

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

VAE Learning Objective

Maximize reconstruction fidelity (e.g. MSE) Make encodings conform to prior

slide-52
SLIDE 52

52

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Original Examples

slide-53
SLIDE 53

53

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Example Corrupted Data

slide-54
SLIDE 54

54

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Example Corrupted Data

slide-55
SLIDE 55

55

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Example Combining GANs & VAEs

slide-56
SLIDE 56

56

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Example Combining GANs & VAEs

slide-57
SLIDE 57

57

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Example Combining GANs & VAEs

slide-58
SLIDE 58

58

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Example Combining GANs & VAEs

l-th layer discriminator

slide-59
SLIDE 59

59

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Example Combining GANs & VAEs

l-th layer discriminator

slide-60
SLIDE 60

60

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Example Combining GANs & VAEs

slide-61
SLIDE 61

61

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Example Combining GANs & VAEs

slide-62
SLIDE 62

62

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

GANs

  • designed to generate new data

VAEs

  • designed to find interpretable latent representation

Notes on VAEs

slide-63
SLIDE 63

63

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

  • GANs designed to generate new data
  • VAEs designed to find interpretable latent representation
  • can go from data to latent representation
  • good for uncertainty estimation
  • latent representation tends to focus on most important features

Notes on VAEs

slide-64
SLIDE 64

64

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

GANs

  • designed to generate new data

VAEs

  • designed to find interpretable latent representation

can

  • go from data to latent representation

good

  • for uncertainty estimation

latent

  • representation tends to focus on most important features

Hard

  • to produce high quality outputs

Need

  • better image similarity measure than MSE

Combination

  • with GANs promising

Notes on VAEs

slide-65
SLIDE 65

65

11.3.2018 l Deep Generative Models l Jens Petersen, Div. of Medical Image Computing

Literature

  • verview GANs

https://github.com/nightrome/really-awesome-gan Literature

  • verview GANs for MIC

https://github.com/xinario/awesome-gan-for-medical-imaging VAE Tutorial (

  • Doersch)

https://arxiv.org/abs/1606.05908 PyTorch

  • DCGAN

https://github.com/pytorch/examples/tree/master/dcgan PyTorch

  • VAE

https://github.com/pytorch/examples/tree/master/vae Improving

  • VAE outputs

(Autoregressive flow) https://arxiv.org/abs/1606.04934 (Normalizing flows) https://arxiv.org/abs/1505.05770 Combining

  • GANs and VAEs

(Adversarial Autoencoder) https://arxiv.org/abs/1511.05644 (Variational GAN) https://arxiv.org/abs/1706.04987 Related

  • generative models

(NICE) https://arxiv.org/abs/1410.8516 (Real NVP) https://arxiv.org/abs/1605.08803

Further Reading