5 Semi-Supervised Learning BVM Tutorial: Advanced Deep Learning - - PowerPoint PPT Presentation

5 Semi-Supervised Learning

BVM Tutorial: Advanced Deep Learning Methods David Zimmerer, Division of Medical Image Computing

Page2 02.11.16 | Author Division

Semi-Supervised Learning

2 | David Zimmerer, Division of Medical Image Computing

Page3 02.11.16 | Author Division

Semi-Supervised Learning

3 | David Zimmerer, Division of Medical Image Computing

Page4 02.11.16 | Author Division

Semi-Supervised Learning

4 | David Zimmerer, Division of Medical Image Computing

Page5 02.11.16 | Author Division

Semi-Supervised Learning

5 | David Zimmerer, Division of Medical Image Computing

Page6 02.11.16 | Author Division

Semi-Supervised Learning

6 | David Zimmerer, Division of Medical Image Computing

Page7 02.11.16 | Author Division

Semi-Supervised Learning

7 | David Zimmerer, Division of Medical Image Computing

Page8 02.11.16 | Author Division

Semi-Supervised Learning

8 | David Zimmerer, Division of Medical Image Computing

Page9 02.11.16 | Author Division

Semi-Supervised Learning

9 | David Zimmerer, Division of Medical Image Computing

Page10 02.11.16 | Author Division

Semi-Supervised Learning

10 | David Zimmerer, Division of Medical Image Computing

Page11 02.11.16 | Author Division

Semi-Supervised Learning

11 | David Zimmerer, Division of Medical Image Computing

Page12 02.11.16 | Author Division

Semi-Supervised Learning

12 | David Zimmerer, Division of Medical Image Computing

Page13 02.11.16 | Author Division

Semi-Supervised Learning

13 | David Zimmerer, Division of Medical Image Computing

Page14 02.11.16 | Author Division

Semi-Supervised Learning

14 | David Zimmerer, Division of Medical Image Computing

Page15 02.11.16 | Author Division

Semi-Supervised Learning

15 | David Zimmerer, Division of Medical Image Computing

Page16 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

16 | David Zimmerer, Division of Medical Image Computing

Page17 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

17 | David Zimmerer, Division of Medical Image Computing

x z

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page18 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

18 | David Zimmerer, Division of Medical Image Computing

x z

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page19 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

19 | David Zimmerer, Division of Medical Image Computing

x z

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page20 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

20 | David Zimmerer, Division of Medical Image Computing

x z x μ σ

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page21 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

21 | David Zimmerer, Division of Medical Image Computing

x z x μ σ z

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page22 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

22 | David Zimmerer, Division of Medical Image Computing

x z x μ σ x’ z

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page23 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

23 | David Zimmerer, Division of Medical Image Computing

x z x μ σ x’ z MSE

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page24 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

24 | David Zimmerer, Division of Medical Image Computing

x z

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page25 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

25 | David Zimmerer, Division of Medical Image Computing

x z

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page26 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

26 | David Zimmerer, Division of Medical Image Computing

x z y National Park

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page27 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

27 | David Zimmerer, Division of Medical Image Computing

x z y

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page28 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

28 | David Zimmerer, Division of Medical Image Computing

x z y

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page29 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

29 | David Zimmerer, Division of Medical Image Computing

x z y

→ Unlabeled Data

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page30 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

30 | David Zimmerer, Division of Medical Image Computing

x z y x y’

→ Unlabeled Data

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page31 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

31 | David Zimmerer, Division of Medical Image Computing

x z y x μ σ y’

→ Unlabeled Data

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page32 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

32 | David Zimmerer, Division of Medical Image Computing

x z y x μ σ z y’

→ Unlabeled Data

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page33 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

33 | David Zimmerer, Division of Medical Image Computing

x z y x μ σ x’ z y’

→ Unlabeled Data

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page34 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

34 | David Zimmerer, Division of Medical Image Computing

x z y x μ σ x’ z MSE y’

→ Unlabeled Data

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page35 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

35 | David Zimmerer, Division of Medical Image Computing

x z y

→ Labeled Data

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page36 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

36 | David Zimmerer, Division of Medical Image Computing

x z y x y

→ Labeled Data

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page37 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

37 | David Zimmerer, Division of Medical Image Computing

x z y x y y’

→ Labeled Data

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page38 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

38 | David Zimmerer, Division of Medical Image Computing

x z y x μ σ y y’

→ Labeled Data

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page39 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

39 | David Zimmerer, Division of Medical Image Computing

x z y x μ σ z y y’

→ Labeled Data

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page40 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

40 | David Zimmerer, Division of Medical Image Computing

x z y x μ σ x’ z y y’

→ Labeled Data

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page41 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

41 | David Zimmerer, Division of Medical Image Computing

x z y x μ σ x’ z y y’ MSE CE

→ Labeled Data

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page42 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

42 | David Zimmerer, Division of Medical Image Computing

x z y

→ Labeled Data

x μ σ x’ z y y’ x μ σ x’ z y ’

→ Unlabeled Data

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page43 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

43 | David Zimmerer, Division of Medical Image Computing

y z KNN VAE + KNN Semi-Sup. VAE 22.07 34.37 63.98 Classification Accuracy on the SVHN dataset with 1000 labels

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page44 02.11.16 | Author Division

Semi-Supervised Variational Autoencoder

44 | David Zimmerer, Division of Medical Image Computing

y z KNN VAE + KNN Semi-Sup. VAE 22.07 34.37 63.98 Classification Accuracy on the SVHN dataset with 1000 labels

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014.

Page45 02.11.16 | Author Division

Self-Supervised Learning

45 | David Zimmerer, Division of Medical Image Computing

Zhang, Richard, et al.. "Colorful image colorization." European Conference on Computer Vision. Springer, Cham, 2016. Ross, Tobias, et al. "Exploiting the potential of unlabeled endoscopic video data with self-supervised learning." arXiv preprint arXiv:1711.09726 (2017).

Page46 02.11.16 | Author Division

Self-Supervised Learning

46 | David Zimmerer, Division of Medical Image Computing

Zhang, Richard, et al.. "Colorful image colorization." European Conference on Computer Vision. Springer, Cham, 2016. Ross, Tobias, et al. "Exploiting the potential of unlabeled endoscopic video data with self-supervised learning." arXiv preprint arXiv:1711.09726 (2017).

Page47 02.11.16 | Author Division

Self-Supervised Learning

47 | David Zimmerer, Division of Medical Image Computing

Labels ???

Zhang, Richard, et al.. "Colorful image colorization." European Conference on Computer Vision. Springer, Cham, 2016. Ross, Tobias, et al. "Exploiting the potential of unlabeled endoscopic video data with self-supervised learning." arXiv preprint arXiv:1711.09726 (2017). Lin, Tsung-Yi, et al. "Microsoft coco: Common objects in context." European conference on computer vision. Springer, Cham, 2014.

Page48 02.11.16 | Author Division

Self-Supervised Learning → Use an auxiliary task for unlabeled data

48 | David Zimmerer, Division of Medical Image Computing

Page49 02.11.16 | Author Division

Self-Supervised Learning → Use an auxiliary task for unlabeled data Semi-Supervised Variational Autoencoder ? Problem: Reconstruction is a “weak” task

49 | David Zimmerer, Division of Medical Image Computing

Page50 02.11.16 | Author Division

Self-Supervised Learning → Use an auxiliary task for unlabeled data Semi-Supervised Variational Autoencoder ? Problem: Reconstruction is a “weak” task → What is a good auxiliary task ?

50 | David Zimmerer, Division of Medical Image Computing

Page51 02.11.16 | Author Division

Self-Supervised Learning - Context Encoders → Use inpainting as auxiliary task

51 | David Zimmerer, Division of Medical Image Computing

Pathak, Deepak, et al. "Context encoders: Feature learning by inpainting." Proceedings of the IEEE CVPR. 2016.

Page52 02.11.16 | Author Division

Self-Supervised Learning - Context Encoders → Use inpainting as auxiliary task

52 | David Zimmerer, Division of Medical Image Computing

Pathak, Deepak, et al. "Context encoders: Feature learning by inpainting." Proceedings of the IEEE CVPR. 2016.

Page53 02.11.16 | Author Division

Self-Supervised Learning - Context Encoders → Use inpainting as auxiliary task

53 | David Zimmerer, Division of Medical Image Computing

Pathak, Deepak, et al. "Context encoders: Feature learning by inpainting." Proceedings of the IEEE CVPR. 2016.

Page54 02.11.16 | Author Division

Self-Supervised Learning - Context Encoders

54 | David Zimmerer, Division of Medical Image Computing

Pathak, Deepak, et al. "Context encoders: Feature learning by inpainting." Proceedings of the IEEE CVPR. 2016.

Page55 02.11.16 | Author Division

Self-Supervised Learning - Context Encoders

55 | David Zimmerer, Division of Medical Image Computing

ImageNet Pretrained Random Initialization Autoencoder Context Encoders 78.2 53.3 53.8 56.5 Classification Accuracy on the Pascal VOC dataset with different pretraining methods

Pathak, Deepak, et al. "Context encoders: Feature learning by inpainting." Proceedings of the IEEE CVPR. 2016.

Page56 02.11.16 | Author Division

Self-Supervised Learning - Image Recolorization → Use image recoloriation as auxiliary task

56 | David Zimmerer, Division of Medical Image Computing

Zhang, Richard, et al.. "Colorful image colorization." European Conference on Computer Vision. Springer, Cham, 2016.

Page57 02.11.16 | Author Division

Self-Supervised Learning - Image Recolorization → Use image recoloriation as auxiliary task

57 | David Zimmerer, Division of Medical Image Computing

Zhang, Richard, et al.. "Colorful image colorization." European Conference on Computer Vision. Springer, Cham, 2016.

Page58 02.11.16 | Author Division

Self-Supervised Learning - Image Recolorization

58 | David Zimmerer, Division of Medical Image Computing

Zhang, Richard, et al.. "Colorful image colorization." European Conference on Computer Vision. Springer, Cham, 2016.

Page59 02.11.16 | Author Division

Self-Supervised Learning - Image Recolorization

59 | David Zimmerer, Division of Medical Image Computing

ImageNet Pretrained Random Initialization Autoencoder Recolorization 79.9 53.3 53.8 67.1 Classification Accuracy on the Pascal VOC dataset with different pretraining methods

Zhang, Richard, et al.. "Colorful image colorization." European Conference on Computer Vision. Springer, Cham, 2016. Zhang, Richard, et al.. "Split-brain autoencoders: Unsupervised learning by cross-channel prediction." CVPR. Vol. 1. No. 2. 2017.

Page60 02.11.16 | Author Division

Self-Supervised Learning - Image Recolorization → Application to medical

60 | David Zimmerer, Division of Medical Image Computing

Adversarial Discriminator Encoder

Ross, Tobias, et al. "Exploiting the potential of unlabeled endoscopic video data with self-supervised learning." arXiv preprint arXiv:1711.09726 (2017).

Page61 02.11.16 | Author Division

Self-Supervised Learning - Image Recolorization → Application to medical

61 | David Zimmerer, Division of Medical Image Computing

Adversarial Discriminator Encoder

Ross, Tobias, et al. "Exploiting the potential of unlabeled endoscopic video data with self-supervised learning." arXiv preprint arXiv:1711.09726 (2017).

Page62 02.11.16 | Author Division

Self-Supervised Learning - Image Recolorization → Application to medical

62 | David Zimmerer, Division of Medical Image Computing

Adversarial Discriminator Encoder

Ross, Tobias, et al. "Exploiting the potential of unlabeled endoscopic video data with self-supervised learning." arXiv preprint arXiv:1711.09726 (2017).

Page63 02.11.16 | Author Division

Self-Supervised Learning - Image Recolorization

63 | David Zimmerer, Division of Medical Image Computing

Ross, Tobias, et al. "Exploiting the potential of unlabeled endoscopic video data with self-supervised learning." arXiv preprint arXiv:1711.09726 (2017).

Page64 02.11.16 | Author Division

Self-Supervised Learning - Image Recolorization

64 | David Zimmerer, Division of Medical Image Computing

Dice Score Fraction of data better

Ross, Tobias, et al. "Exploiting the potential of unlabeled endoscopic video data with self-supervised learning." arXiv preprint arXiv:1711.09726 (2017).

Page65 02.11.16 | Author Division

Self-Supervised Learning - Medical Applications → Medical Images

65 | David Zimmerer, Division of Medical Image Computing

Cai, Yunliang, et al."Multi-modality vertebra recognition in arbitrary views using 3D deformable hierarchical model." IEEE transactions on medical imaging 2015 Zhang, Pengyue, et al. "Self supervised deep representation learning for fine-grained body part recognition." Biomedical Imaging (ISBI 2017), 2017 IEEE 14th International Symposium on. IEEE, 2017.

Page66 02.11.16 | Author Division

Self-Supervised Learning - Medical Applications → Sorting body part (from top to toe)

66 | David Zimmerer, Division of Medical Image Computing

Zhang, Pengyue, et al. "Self supervised deep representation learning for fine-grained body part recognition." Biomedical Imaging (ISBI 2017), 2017 IEEE 14th International Symposium on. IEEE, 2017.

Page67 02.11.16 | Author Division

Self-Supervised Learning - Medical Applications → Sorting body part (from top to toe)

67 | David Zimmerer, Division of Medical Image Computing

Zhang, Pengyue, et al. "Self supervised deep representation learning for fine-grained body part recognition." Biomedical Imaging (ISBI 2017), 2017 IEEE 14th International Symposium on. IEEE, 2017.

Page68 02.11.16 | Author Division

Self-Supervised Learning - Medical Applications

68 | David Zimmerer, Division of Medical Image Computing

Recognition error (in mm) of body part recognition on CT and MR data Image 1 Image 2 proposed ordering

Zhang, Pengyue, et al. "Self supervised deep representation learning for fine-grained body part recognition." Biomedical Imaging (ISBI 2017), 2017 IEEE 14th International Symposium on. IEEE, 2017.

Page69 02.11.16 | Author Division

Self-Supervised Learning - Medical Applications

69 | David Zimmerer, Division of Medical Image Computing

Recognition error (in mm) of body part recognition on CT and MR data (lower is better)

Zhang, Pengyue, et al. "Self supervised deep representation learning for fine-grained body part recognition." Biomedical Imaging (ISBI 2017), 2017 IEEE 14th International Symposium on. IEEE, 2017.

Page70 02.11.16 | Author Division

Self-Supervised Learning - Medical Applications

70 | David Zimmerer, Division of Medical Image Computing

Recognition error (in mm) of body part recognition on CT and MR data (lower is better)

Zhang, Pengyue, et al. "Self supervised deep representation learning for fine-grained body part recognition." Biomedical Imaging (ISBI 2017), 2017 IEEE 14th International Symposium on. IEEE, 2017.

Page71 02.11.16 | Author Division

Self-Supervised Learning - Medical Applications

71 | David Zimmerer, Division of Medical Image Computing

Recognition error (in mm) of body part recognition on CT and MR data (lower is better)

Zhang, Pengyue, et al. "Self supervised deep representation learning for fine-grained body part recognition." Biomedical Imaging (ISBI 2017), 2017 IEEE 14th International Symposium on. IEEE, 2017.

Page72 02.11.16 | Author Division

Self-Supervised Learning - Medical Applications → Differentiate between subjects

72 | David Zimmerer, Division of Medical Image Computing

Jamaludin, Amir, et al.. "Self-Supervised Learning for Spinal MRIs." Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision

• Support. Springer, Cham, 2017. 294-302.

Page73 02.11.16 | Author Division

Self-Supervised Learning - Medical Applications → Differentiate between subjects

73 | David Zimmerer, Division of Medical Image Computing

Jamaludin, Amir, et al.. "Self-Supervised Learning for Spinal MRIs." Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision

• Support. Springer, Cham, 2017. 294-302.

Page74 02.11.16 | Author Division

Self-Supervised Learning - Medical Applications → Differentiate between subjects

74 | David Zimmerer, Division of Medical Image Computing

Jamaludin, Amir, et al.. "Self-Supervised Learning for Spinal MRIs." Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision

• Support. Springer, Cham, 2017. 294-302.

Page75 02.11.16 | Author Division

Self-Supervised Learning - Medical Applications

75 | David Zimmerer, Division of Medical Image Computing

Classification Accuracy Number of Scans better

Jamaludin, Amir, et al.. "Self-Supervised Learning for Spinal MRIs." Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision

• Support. Springer, Cham, 2017. 294-302.

Page76 02.11.16 | Author Division

Semi-Supervised Learning - “Mean teachers are better role models”

76 | David Zimmerer, Division of Medical Image Computing

Tarvainen, Antti, and Harri Valpola. "Weight-averaged consistency targets improve semi-supervised deep learning results."arXiv:1703.01780 (2017).

Page77 02.11.16 | Author Division

Semi-Supervised Learning - “Mean teachers are better role models”

77 | David Zimmerer, Division of Medical Image Computing

Student Teacher

Tarvainen, Antti, and Harri Valpola. "Weight-averaged consistency targets improve semi-supervised deep learning results."arXiv:1703.01780 (2017).

Page78 02.11.16 | Author Division

Semi-Supervised Learning - “Mean teachers are better role models”

78 | David Zimmerer, Division of Medical Image Computing

Student Teacher Input

Tarvainen, Antti, and Harri Valpola. "Weight-averaged consistency targets improve semi-supervised deep learning results."arXiv:1703.01780 (2017).

Page79 02.11.16 | Author Division

Semi-Supervised Learning - “Mean teachers are better role models”

79 | David Zimmerer, Division of Medical Image Computing

Student Teacher Input cat dog

Tarvainen, Antti, and Harri Valpola. "Weight-averaged consistency targets improve semi-supervised deep learning results."arXiv:1703.01780 (2017).

Page80 02.11.16 | Author Division

Semi-Supervised Learning - “Mean teachers are better role models”

80 | David Zimmerer, Division of Medical Image Computing

Student Teacher Input cat dog MSE

Tarvainen, Antti, and Harri Valpola. "Weight-averaged consistency targets improve semi-supervised deep learning results."arXiv:1703.01780 (2017).

Page81 02.11.16 | Author Division

Semi-Supervised Learning - “Mean teachers are better role models”

81 | David Zimmerer, Division of Medical Image Computing

Student Teacher Input cat dog dog Label MSE

Tarvainen, Antti, and Harri Valpola. "Weight-averaged consistency targets improve semi-supervised deep learning results."arXiv:1703.01780 (2017).

Page82 02.11.16 | Author Division

Semi-Supervised Learning - “Mean teachers are better role models”

82 | David Zimmerer, Division of Medical Image Computing

Student Teacher Input cat dog dog Label CE MSE

Tarvainen, Antti, and Harri Valpola. "Weight-averaged consistency targets improve semi-supervised deep learning results."arXiv:1703.01780 (2017).

Page83 02.11.16 | Author Division

Semi-Supervised Learning - “Mean teachers are better role models”

83 | David Zimmerer, Division of Medical Image Computing

Old Student New Student Error Gradient

+ =

Tarvainen, Antti, and Harri Valpola. "Weight-averaged consistency targets improve semi-supervised deep learning results."arXiv:1703.01780 (2017).

Page84 02.11.16 | Author Division

Semi-Supervised Learning - “Mean teachers are better role models”

84 | David Zimmerer, Division of Medical Image Computing

Teacher Old Students

=

Weighted Sum

Tarvainen, Antti, and Harri Valpola. "Weight-averaged consistency targets improve semi-supervised deep learning results."arXiv:1703.01780 (2017).

Page85 02.11.16 | Author Division 85 | David Zimmerer, Division of Medical Image Computing

Semi-Supervised Learning - “Mean teachers are better role models”

Top-5 validation error on Imagenet with 10% of the labels (lower is better)

Tarvainen, Antti, and Harri Valpola. "Weight-averaged consistency targets improve semi-supervised deep learning results."arXiv:1703.01780 (2017).

Page86 02.11.16 | Author Division

Thanks !

86 | David Zimmerer, Division of Medical Image Computing

Page87 02.11.16 | Author Division

References

Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing

• Systems. 2014.

Pathak, Deepak, et al. "Context encoders: Feature learning by inpainting." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016. Zhang, Richard, et al.. "Colorful image colorization." European Conference on Computer Vision. Springer, Cham, 2016. Zhang, Richard, et al.. "Split-brain autoencoders: Unsupervised learning by cross-channel prediction." CVPR. Vol. 1. No. 2. 2017. Ross, Tobias, et al. "Exploiting the potential of unlabeled endoscopic video data with self-supervised learning." arXiv preprint arXiv:1711.09726 (2017). Jamaludin, Amir, et al.. "Self-Supervised Learning for Spinal MRIs." Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. Springer, Cham, 2017. 294-302. Zhang, Pengyue, et al. "Self supervised deep representation learning for fine-grained body part recognition." Biomedical Imaging (ISBI 2017), 2017 IEEE 14th International Symposium on. IEEE, 2017. Tarvainen, Antti, and Harri Valpola. "Weight-averaged consistency targets improve semi-supervised deep learning results." arXiv preprint arXiv:1703.01780 (2017). Cai, Yunliang, et al. "Multi-modality vertebra recognition in arbitrary views using 3D deformable hierarchical model." IEEE transactions on medical imaging 34.8 (2015): 1676-1693. Lin, Tsung-Yi, et al. "Microsoft coco: Common objects in context." European conference on computer vision. Springer, Cham, 2014.

87 | David Zimmerer, Division of Medical Image Computing