Capsule Networks - An Overview Luca Dombetzki July 13, 2018 - PowerPoint PPT Presentation

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Capsule Networks - An Overview Luca Dombetzki July 13, 2018 Advisor: Marton Kajo Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Overview Introduction Convolutional Neural Networks Capsule Networks Discussion Conclusion Bibliography Appendix L. Dombetzki — Capsule Networks 2

Introduction Introduction Convolutional Neural Networks Capsule Networks Discussion Conclusion Bibliography Appendix L. Dombetzki — Capsule Networks 3

Introduction Motivation Figure 1: figure from [12] Both images are seen as "face" by a typical Convolutional Neural Network ⇒ Capsule Networks L. Dombetzki — Capsule Networks 4

Introduction Where does AI come from? Figure 2: A neuron as part of a Multi Layer Neural Network [21] BUT not like human brain anymore Designed after human brain • Advancement in modeling with math • Blackbox system • Performance gains with GPUs • Requires huge amounts of data • Deep Learning - leverage both • Very probabilistic L. Dombetzki — Capsule Networks 5

Introduction Who is Geoffrey E. Hinton? “The pooling operation used in convolutional neural networks is a big mistake and the fact that it works so well is a disaster.” - Geoffrey E. Hinton (2014) [7] • Professor at Toronto University • Working at Google Brain • Major advancements in AI [13] • Research on Capsule Networks: • Based on biological research • Understanding Human vision (1981) [9] • Talks explaining his motivation [8] • Dynamic Routing Between Capsules (2017) [19] • Matrix Capsules with EM-Routing (2018) [6] Figure 3: Geoffrey E. Hinton [24] L. Dombetzki — Capsule Networks 6

Convolutional Neural Networks Introduction Convolutional Neural Networks Capsule Networks Discussion Conclusion Bibliography Appendix L. Dombetzki — Capsule Networks 7

Convolutional Neural Networks What are CNNs? Figure 4: Typcial architecture of a CNN [16] L. Dombetzki — Capsule Networks 8

Convolutional Neural Networks Convolution and kernels Figure 5: Convolution operation [11] L. Dombetzki — Capsule Networks 9

Convolutional Neural Networks Activation functions Figure 6: Sigmoid and Rectified Linear Unit (ReLU) [20] +1 w 0 w 1 � � x 1 n σ w 0 + � w i x i w 2 i =1 x 2 Σ σ w 3 x 3 w n x n Figure 7: A single neuron [21] L. Dombetzki — Capsule Networks 10

Convolutional Neural Networks Pooling as a form of routing Routing • find important nodes (inputs) • group together • give to next layer Pooling • reduces input data • next layer can “see” more than the previous • enables detecting full objects through locational invariance Figure 8: Max pooling example [2] • static routing L. Dombetzki — Capsule Networks 11

Convolutional Neural Networks How CNNs see the world Figure 9: Feature detections of a CNN [15] L. Dombetzki — Capsule Networks 12

Convolutional Neural Networks Problems of pooling Figure 10: Distorted face from [12] Geoffrey E. Hinton’s arguments against pooling [8] • Unnatural • No use of the linear structure of vision • Static instead of dynamic routing • Invariance instead of Equivariance L. Dombetzki — Capsule Networks 13

Convolutional Neural Networks What does a neuron represent? Figure 11: Face detection with a CNN, from [10] L. Dombetzki — Capsule Networks 14

Capsule Networks Introduction Convolutional Neural Networks Capsule Networks Discussion Conclusion Bibliography Appendix L. Dombetzki — Capsule Networks 15

Capsule Networks Hinton’s idea Figure 12: Hierarchical modeling in Computer Graphics [5] Build a network to perform inverse graphics • propagate probability and pose of features • dynamic routing based on pose information • introduce concept of an entity into the network’s architecture ⇒ The capsule L. Dombetzki — Capsule Networks 16

Capsule Networks An abstract view on capsules Figure 13: Capsule face detection, from [10] L. Dombetzki — Capsule Networks 17

Capsule Networks The capsule - a group of neurons Before After layer of neurons layer of neuron groups input = n values, output = value input = n vectors, output = vector • A capsule learns parameters (skew, scale, rotation, etc) • n-dimensional capsule = n-dimensional vector out ⇒ n parameters ˆ = pose • probability = || vector out || L. Dombetzki — Capsule Networks 18

Capsule Networks Architecture - The CapsNet Figure 14: Capsule Network Architecture as described in [19] Layer Function Conv1 Convolutional layer PrimaryCaps Convolutional squashing capsules DigitCaps Normal (digit) capsules Class predictions Length of each DigitCapsule L. Dombetzki — Capsule Networks 19

Capsule Networks Routing-by-agreement - the idea Figure 15: capsule agreement [4] L. Dombetzki — Capsule Networks 20

Capsule Networks Routing by agreement Phenomenon “ coincidence filtering ” • high dimensional pose-parameter-space • similar poses by chance very unlikely (curse of dimensionality) Clustering the inputs based on their pose: repeat n times: 1. find the mean vector of the cluster 2. weighs all inputs based on their distance to this mean 3. normalize the weights Figure 16: weighted clustering [4] L. Dombetzki — Capsule Networks 21

Capsule Networks How to train the network Margin Loss Reconstruction (Decoder) network Figure 17: Capsule Network architectures [19] Goal Lossfunction Learning Parameter learning Reconstruction loss Unsupervised Classification Margin loss Supervised Margin loss Reconstruction loss • detection: || v || ≥ 0.9 • reconstruct digit by masking the active capsule • no detection: || v || ≤ 0.1 L. Dombetzki — Capsule Networks 22

Capsule Networks How does it perform? - Parameter Effects Scale and thickness Localized part Stroke thick- ness Localized skew Width and translation Localized part Figure 18: Effects of capsule parameters on reconstruction [19] L. Dombetzki — Capsule Networks 23

Capsule Networks How does it perform? - MultiMNIST R: (6, 0) R: (6, 8) R: (7, 1) R: (8, 7) R: (9, 4) R: (9, 5) R: (8, 4) L: (6, 0) L: (6, 8) L: (7, 1) L: (8, 7) L: (9, 4) L: (9, 5) L: (8, 4) Routing Rec.Loss MNIST (%) MultiMNIST (%) CNN - - 0.39 8.1 CapsNet 1 no 0.34 ± 0.032 - CapsNet 1 yes 0.29 ± 0.011 7.5 CapsNet 3 no 0.35 ± 0.036 - CapsNet 3 yes 0.25 ± 0.005 5.2 Figure 19: Cpasule Network results on MultiMNIST [19] L. Dombetzki — Capsule Networks 24

Capsule Networks How does it perform? - MultiMNIST Network was forced to reconstruct false predictions *R: (5, 7) *R: (2, 3) *R: (0, 8) *R: (1, 6) L: (5, 0) L: (4, 3) L: (1, 8) L: (7, 6) Figure 20: [19] L. Dombetzki — Capsule Networks 25

Capsule Networks Further research Authors Contribution Hinton et. al Pose capsules and EM-routing [6] Xi et. al Hyperparamter tuning for complex data [25] Phaye et. al Skip connections [17] Rawlinson et. al Unsupervised training [18] Bahadori et. al New routing (Eigen-decomposition) [3] Wang et. al Optimized routing (KL regularization) [22] L. Dombetzki — Capsule Networks 26

Discussion Introduction Convolutional Neural Networks Capsule Networks Discussion Conclusion Bibliography Appendix L. Dombetzki — Capsule Networks 27

Discussion Superior to CNNs? Advantages Challenges Viewpoint invariance Scalability Less training data needed “Explain everything” Fewer parameters Entity based structure Better generalization Loss functions White-box attacks Crowding Validatability Unoptimized implementation L. Dombetzki — Capsule Networks 28

Discussion CapsNets for real world problems Figure 21: Results from Afshar et. al [1] Authors Application Benefit Afshar et. al [1] Brain tumor classification Less training data Wang et. al [23] Sentiment analysis with RNNs State-of-the-art performance LaLonde et. al [14] medical image classification Parameter reduction by 95.4% L. Dombetzki — Capsule Networks 29

Conclusion Introduction Convolutional Neural Networks Capsule Networks Discussion Conclusion Bibliography Appendix L. Dombetzki — Capsule Networks 30

Conclusion Conclusion Big step towards human vision • Novel network architecture • Inverse graphics through pose vector capsules • Dynamic routing via routing-by-agreement • Multiple significant advantages • Early development phase But not comparable to CNNs in “mainstream areas” L. Dombetzki — Capsule Networks 31

Questions? Figure 22: [20] L. Dombetzki — Capsule Networks 32

Bibliography Introduction Convolutional Neural Networks Capsule Networks Discussion Conclusion Bibliography Appendix L. Dombetzki — Capsule Networks 33