Exploring Deep Anomaly Detection Methods Based on Capsule Net Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Electrical Engineering & Computer Science University of Ottawa xli343@uottawa.ca May 5, 2020 Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 1 / 22
Outline Introduction 1 Anomaly Detection Insight of Existing Works 2 Boundary-based & Distribution-based CapsNet Three Normality Score Functions 3 Prediction Probability Based Normality Score Reconstruction Error Based Normality Score Combined Normality Score Experiments and Results 4 Conclusion 5 Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 2 / 22
Outline Introduction 1 Anomaly Detection Insight of Existing Works 2 Boundary-based & Distribution-based CapsNet Three Normality Score Functions 3 Prediction Probability Based Normality Score Reconstruction Error Based Normality Score Combined Normality Score Experiments and Results 4 Conclusion 5 Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 3 / 22
Anomaly Detection Anomaly detection (also known as outlier detection ) aims at uncovering abnormal data points which may stand for novel or alarming events. (a) Cyber-Network Intrusion (b) Illegal Traffic detection detection [1] [2] (c) Detecting Retinal (d) IoT sensor data [4] Damage [3] Figure 1: Applications of anomaly detection technique. Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 4 / 22
Contributions CapsNet is explored and tested for deep anomaly detection task. Based on unique characteristics of capsNet [5, 6, 7], three effective normality score functions are proposed. The proposed methods are compared with principled benchmark methods and three advanced deep generative techniques and assessed their capacities for deep anomaly detection. Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 5 / 22
Outline Introduction 1 Anomaly Detection Insight of Existing Works 2 Boundary-based & Distribution-based CapsNet Three Normality Score Functions 3 Prediction Probability Based Normality Score Reconstruction Error Based Normality Score Combined Normality Score Experiments and Results 4 Conclusion 5 Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 6 / 22
Boundary-based & Distribution-based Boundary-Based Methods Kernel based support vector domain description (SVDD) [8] and one-class support vector machine (OCSVM) [8, 9] are successful ones in pre-deep-learning era. Deep hybrid methods: VAE+OCSVM [10] and DBN+OCSVM [11], and one-class deep SVDD [12]. Distribution-Based Methods Deep generative models (DGMs) based methods such as deep belief net (DBN) [13, 14], variational autoencoder (VAE) [15] and generative adversarial net (GAN) [3] are applied as unsupervised feature learning techniques. Likelihood p ( x ) serves as an anomality describer. Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 7 / 22
Capsule Vs. Traditional Neuron CNNs [16] are not able to capture the properties of entities (such as position, orientation, size, and part-whole relationship). CapsNets [5, 6, 7] show advantages in learning such information. Figure 2: Main differences between capsules and traditional neurons. Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 8 / 22
Outline Introduction 1 Anomaly Detection Insight of Existing Works 2 Boundary-based & Distribution-based CapsNet Three Normality Score Functions 3 Prediction Probability Based Normality Score Reconstruction Error Based Normality Score Combined Normality Score Experiments and Results 4 Conclusion 5 Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 9 / 22
Architecture of CapsNet Kernel: 9x9 -> 1 256 feature maps Kernel: 9x9x256 -> 8 32 feature maps/capsule types 256 kernels 32 kernels 256 16 9x9 28 8 20 A capsule of 8D 10 9x9 32 6 28 20 6 Input image 256 feature maps 6x6x32 primary capsules of 8D 10 digit capsules of 16D (a) Encoder as classifier. Target Image 4 10 digit capsules of 16D 512 units 1024 units 784 units (b) Decoder as regularizer. Figure 3: Architecture of the CapsNet designed in [6] for MNIST data. The main structure displayed in (a) is a classifier but can also be viewed as an encoder. The reconstruction regularizer can be viewed as a decoder as displayed in (b). Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 10 / 22
Normality Score Functions Prediction probability based normality score ( N PP ). N PP ( x ) � c =1 , ··· , C ( � h c � 2 ) , max (1) where x is an input image, h c represents the c -th digit capsule, � h c � 2 denotes the probability of x belonging to the c -th class. Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 11 / 22
Normality Score Functions Reconstruction error based normality score ( N RE ). N RE ( x ) � − NSE( x , x ′ ) = −� x − x ′ � 2 2 (2) , � x � 2 where x represents an actual image and x ′ represents the reconstructed image. Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 12 / 22
Normality Score Functions Combined Normality Score ( N PP + RE ). N PP + RE ( x ) � α N RE ( x ) + (1 − α ) N PP ( x ) , (3) where α ∈ [0 , 1] is the combination hyperparameter such that the two terms can effectively complement each other. Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 13 / 22
Outline Introduction 1 Anomaly Detection Insight of Existing Works 2 Boundary-based & Distribution-based CapsNet Three Normality Score Functions 3 Prediction Probability Based Normality Score Reconstruction Error Based Normality Score Combined Normality Score Experiments and Results 4 Conclusion 5 Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 14 / 22
Experiments and Results Comparison on MNIST Dataset. Figure 4: Performance of PP, RE, and benchmark methods on MNIST. Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 15 / 22
Experiments and Results Comparison Fashion-MNIST and Small-Norb Datasets. (a) On Fashion-MNIST. (b) On Small-Norb. Figure 5: Performance of PP, RE, and benchmark methods on Fashion-MNIST and Small-Norb Datasets. Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 16 / 22
Experiments and Results PP-based normality score VS. RE-based normality score (a) Digit 2 (as anomaly). (b) Digit 9 (as anomaly). (c) Digit 2 (as anomaly). (d) Digit 9 (as anomaly). Figure 6: ROC curves and images (Original digits (upper half) and reconstructed digits (lower half)) when detecting anomalous digits: “2” and “9”. Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 17 / 22
Experiments and Results Performance of PP+RE-based Normality Score. Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 18 / 22 Figure 7
Experiments and Results Performance of capsNet-based method in comparison with other advanced generative methods. The results for AnoDM and β -VAE were obtained from [17]. The results for AnoGAN and ADGAN were obtained from [18]. Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 19 / 22 Figure 8
Outline Introduction 1 Anomaly Detection Insight of Existing Works 2 Boundary-based & Distribution-based CapsNet Three Normality Score Functions 3 Prediction Probability Based Normality Score Reconstruction Error Based Normality Score Combined Normality Score Experiments and Results 4 Conclusion 5 Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 20 / 22
Recommend
More recommend
