Exploring Deep Anomaly Detection Methods Based on Capsule Net - - PowerPoint PPT Presentation

Exploring Deep Anomaly Detection Methods Based on Capsule Net

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Electrical Engineering & Computer Science University of Ottawa xli343@uottawa.ca May 5, 2020

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 1 / 22

Outline

1

Introduction Anomaly Detection

2

Insight of Existing Works Boundary-based & Distribution-based CapsNet

3

Three Normality Score Functions Prediction Probability Based Normality Score Reconstruction Error Based Normality Score Combined Normality Score

4

Experiments and Results

5

Conclusion

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 2 / 22

Outline

1

Introduction Anomaly Detection

2

Insight of Existing Works Boundary-based & Distribution-based CapsNet

3

Three Normality Score Functions Prediction Probability Based Normality Score Reconstruction Error Based Normality Score Combined Normality Score

4

Experiments and Results

5

Conclusion

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 3 / 22

Anomaly Detection

Anomaly detection (also known as outlier detection) aims at uncovering abnormal data points which may stand for novel or alarming events.

(a) Cyber-Network Intrusion detection [1] (b) Illegal Traffic detection [2] (c) Detecting Retinal Damage [3] (d) IoT sensor data [4]

Figure 1: Applications of anomaly detection technique.

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 4 / 22

Contributions

CapsNet is explored and tested for deep anomaly detection task. Based on unique characteristics of capsNet [5, 6, 7], three effective normality score functions are proposed. The proposed methods are compared with principled benchmark methods and three advanced deep generative techniques and assessed their capacities for deep anomaly detection.

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 5 / 22

Outline

1

Introduction Anomaly Detection

2

Insight of Existing Works Boundary-based & Distribution-based CapsNet

3

Three Normality Score Functions Prediction Probability Based Normality Score Reconstruction Error Based Normality Score Combined Normality Score

4

Experiments and Results

5

Conclusion

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 6 / 22

Boundary-based & Distribution-based

Boundary-Based Methods

Kernel based support vector domain description (SVDD) [8] and

• ne-class support vector machine (OCSVM) [8, 9] are successful ones

in pre-deep-learning era. Deep hybrid methods: VAE+OCSVM [10] and DBN+OCSVM [11], and one-class deep SVDD [12].

Distribution-Based Methods

Deep generative models (DGMs) based methods such as deep belief net (DBN) [13, 14], variational autoencoder (VAE) [15] and generative adversarial net (GAN) [3] are applied as unsupervised feature learning techniques. Likelihood p(x) serves as an anomality describer.

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 7 / 22

Capsule Vs. Traditional Neuron

CNNs [16] are not able to capture the properties of entities (such as position, orientation, size, and part-whole relationship). CapsNets [5, 6, 7] show advantages in learning such information.

Figure 2: Main differences between capsules and traditional neurons.

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 8 / 22

Outline

1

Introduction Anomaly Detection

2

Insight of Existing Works Boundary-based & Distribution-based CapsNet

3

Three Normality Score Functions Prediction Probability Based Normality Score Reconstruction Error Based Normality Score Combined Normality Score

4

Experiments and Results

5

Conclusion

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 9 / 22

Architecture of CapsNet

Kernel: 9x9 -> 1 256 kernels 256 feature maps Kernel: 9x9x256 -> 8 32 kernels 32 feature maps/capsule types 6x6x32 primary capsules of 8D 6 6 8 20 20 256 9x9 9x9

A capsule of 8D

32 28 28 10 digit capsules of 16D Input image 256 feature maps 10 16

(a) Encoder as classifier.

10 digit capsules of 16D 4 512 units 1024 units 784 units Target Image

(b) Decoder as regularizer.

Figure 3: Architecture of the CapsNet designed in [6] for MNIST data. The main structure displayed in (a) is a classifier but can also be viewed as an encoder. The reconstruction regularizer can be viewed as a decoder as displayed in (b).

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 10 / 22

Normality Score Functions

Prediction probability based normality score (NPP). NPP(x) max

c=1,··· ,C(hc2),

(1) where x is an input image, hc represents the c-th digit capsule, hc2 denotes the probability of x belonging to the c-th class.

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 11 / 22

Normality Score Functions

Reconstruction error based normality score (NRE). NRE(x) −NSE(x, x′) = −x − x′2

2

x2 , (2) where x represents an actual image and x′ represents the reconstructed image.

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 12 / 22

Normality Score Functions

Combined Normality Score (NPP+RE). NPP+RE(x) αNRE(x) + (1 − α)NPP(x), (3) where α ∈ [0, 1] is the combination hyperparameter such that the two terms can effectively complement each other.

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 13 / 22

Outline

1

Introduction Anomaly Detection

2

Insight of Existing Works Boundary-based & Distribution-based CapsNet

3

Three Normality Score Functions Prediction Probability Based Normality Score Reconstruction Error Based Normality Score Combined Normality Score

4

Experiments and Results

5

Conclusion

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 14 / 22

Experiments and Results

Comparison on MNIST Dataset.

Figure 4: Performance of PP, RE, and benchmark methods on MNIST.

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 15 / 22

Experiments and Results

Comparison Fashion-MNIST and Small-Norb Datasets.

(a) On Fashion-MNIST. (b) On Small-Norb.

Figure 5: Performance of PP, RE, and benchmark methods on Fashion-MNIST and Small-Norb Datasets.

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 16 / 22

Experiments and Results

PP-based normality score VS. RE-based normality score

(a) Digit 2 (as anomaly). (b) Digit 9 (as anomaly). (c) Digit 2 (as anomaly). (d) Digit 9 (as anomaly).

Figure 6: ROC curves and images (Original digits (upper half) and reconstructed

digits (lower half)) when detecting anomalous digits: “2” and “9”.

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 17 / 22

Experiments and Results

Performance of PP+RE-based Normality Score.

Figure 7

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 18 / 22

Experiments and Results

Performance of capsNet-based method in comparison with other advanced generative

• methods. The results for AnoDM and β-VAE were obtained from [17]. The results for

AnoGAN and ADGAN were obtained from [18].

Figure 8

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 19 / 22

Outline

1

Introduction Anomaly Detection

2

Insight of Existing Works Boundary-based & Distribution-based CapsNet

3

Three Normality Score Functions Prediction Probability Based Normality Score Reconstruction Error Based Normality Score Combined Normality Score

4

Experiments and Results

5

Conclusion

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 20 / 22

Conclusion

Explore novel solutions of anomaly detection in consideration of CapsNet’s distinct characteristics. Devise three normality score functions based on CapsNet’s activation probability and reconstruction error respectively. Experiments on four image datasets show that these CapsNet-based methods outperform existing solutions in many setups. AnoCapsNet framework using combined (PP+RE) normality score technique accomplishes quite impressive results on some complicated datasets (such as CIFAR-10 and Small-Norb).

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 21 / 22

Thank you!

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 22 / 22

• A. Javaid, Q. Niyaz, W. Sun, and M. Alam, “A deep learning

approach for network intrusion detection system,” in Proceedings of the 9th EAI International Conference on Bio-inspired Information and Communications Technologies (formerly BIONETICS). ICST (Institute for Computer Sciences, SocialInformatics and Telecommunications Engineering), 2016, pp. 21–26.

• X. Xie, C. Wang, S. Chen, G. Shi, and Z. Zhao, “Real-time illegal

parking detection system based on deep learning,” in Proceedings of the 2017 International Conference on Deep Learning Technologies. ACM, 2017, pp. 23–27.

• T. Schlegl, P. Seebock, S. M. Waldstein, U. S. Erfurth, and G. Langs,

“Unsupervised anomaly detection with generative adversarial networks to guide marker discovery,” in International Conference on Information Processing in Medical Imaging. Springer, 2017, pp. 146–157.

• M. Mohammadi, A. Al-Fuqaha, S. Sorour, and M. Guizani, “Deep

learning for iot big data and streaming analytics: A survey,” ArXiv, p. arXiv:1712.04301, 2017.

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 22 / 22

• G. E. Hinton, A. Krizhevsky, and S. D. Wang, “Transforming

auto-encoder,” in International Conference on Artificial Neural Networks, 2011, pp. 44–51.

• S. Sabour, N. Frosst, and G. E. Hinton, “Dynamic routing between

capsules,” in Neural Information Processing Systems, 2017, pp. 3856–3866.

• G. E. Hinton, S. Sabour, and N. Frosst, “Matrix capsules with EM

routing,” in International Conferences on Learning Representations, 2018.

• D. M. J. Tax and R. P. W. Duin, “Support vector domain description,”

Pattern Recognition Letters, vol. 20, pp. 1191–1199, 1999.

• B. Sch¨
• lkopf, J. C. Platt, J. C. Shawe-Taylor, A. J. Smola, and R. C.

Williamson, “Estimating the support of a high-dimensional distribution,” Neural Comput, vol. 13, no. 7, pp. 1443–1471, Jul 2001.

• J. T. A. Andrews, E. J. Morton, and L. D. Griffin, “Detecting

anomalous data using auto-encoders,” International Journal of Machine Learning and Computing, vol. 6, no. 1, pp. 21–26, 2016.

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 22 / 22

• S. M. Erfani, S. Rajasegarar, S. Karunasekera, and C. Leckie,

“High-dimensional and large-scale anomaly detection using a linear

• ne-class SVM with deep learning,” Pattern Recognition, vol. 58, pp.

121–134, 2016.

• L. Ruff, R. Vandermeulen, N. Goernitz, L. Deecke, S. A. Siddiqui,
• A. Binder, E. M¨

uller, and M. Kloft, “Deep one-class classification,” in Proceedings of the 35th International Conference on Machine Learning, vol. 80. PMLR, 2018, pp. 4393–4402.

• G. E. Hinton, S. Osindero, and Y. Teh, “A fast learning algorithm for

deep belief nets,” Neural Computation, vol. 18, pp. 1527–1554, 2006.

• Y. Li and X. Zhu, “Exploring Helmholtz machine and deep belief net

in the exponential family perspective,” in International Conference on Machine Learning 2018 Workshop on Theoretical Foundations and Applications of Deep Generative Models, July 2018.

• D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,” in

International Conference on Learning Representations, 2014.

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 22 / 22

• Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based

learning applied to document recognition,” Proceedings of the IEEE,

• vol. 86, no. 11, pp. 2278–2324, 1998.
• X. Li, I. Kiringa, T. Yeap, X. Zhu, and Y. Li, “Anomaly detection

based on unsupervised disentangled representation learning in combination with manifold learning,” IEEE WCCI 2020, 2020.

• L. Deecke, R. Vandermeulen, L. Ruff, S. Mandt, and M. Kloft, “Image

anomaly detection with generative adversarial networks,” in Machine Learning and Knowledge Discovery in Databases. Cham: Springer International Publishing, 2019, pp. 3–17.

Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, Yifeng Li Exploring Deep Anomaly Detection Methods Based on Capsule Net May 5, 2020 22 / 22