Haoyi Fan 1 , Fengbin Zhang 1 , Ruidong Wang 1 , Liang Xi 1 , Zuoyong - - PowerPoint PPT Presentation

• 1-

Correlation-aware Deep Generative Model for Unsupervised Anomaly Detection Haoyi Fan 1, Fengbin Zhang 1, Ruidong Wang 1, Liang Xi 1, Zuoyong Li 2

Harbin University of Science and Technology 1 Minjiang University 2 isfanhy@hrbust.edu.cn

• 2-

Background

Latent Space

Anomaly

Anomaly

Normal

Observed Space

• 3-

Background

https://www.explosion.com/135494/5-effective-strategies-of-fraud- detection-and-prevention-for-ecommerce/ https://blog.exporthub.com/working-with-chinese-manufacturers/ https://planforgermany.com/switching-private-public-health-insurance- germany/

Fraud Detection Disease Detection Fault Detection Intrusion Detection

https://towardsdatascience.com/building-an-intrusion-detection-system- using-deep-learning-b9488332b321

• 4-

Background

Latent Space

Unsupervised Anomaly Detection

– From the Density Estimation Perspective Data samples: 𝑌𝑢𝑠𝑏𝑗𝑜= 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑜 , 𝑦𝑗 is assumed normal.

• 5-

Background

Latent Space

Unsupervised Anomaly Detection

– From the Density Estimation Perspective Data samples: 𝑌𝑢𝑠𝑏𝑗𝑜 = 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑜 , 𝑦𝑗 is assumed normal. Model: 𝑞(𝑦)

• 6-

Background

Latent Space

Unsupervised Anomaly Detection

– From the Density Estimation Perspective Data samples: 𝑌𝑢𝑠𝑏𝑗𝑜 = 𝑦1, 𝑦2, … , 𝑦𝑜 , 𝑦𝑗 is assumed normal. Model: 𝑞(𝑦) Test samples: 𝑌𝑢𝑓𝑡𝑢 = 𝑦1, 𝑦2, … , 𝑦𝑜 , 𝑦𝑢 is unknow. if 𝑞(𝑦𝑢) < 𝜇, 𝑦𝑢 is abnormal. if 𝑞(𝑦𝑢) ≥ 𝜇, 𝑦𝑢 is normal.

• 7-

Background

Unsupervised Anomaly Detection

– From the Density Estimation Perspective Data samples: 𝑌𝑢𝑠𝑏𝑗𝑜 = 𝑦1, 𝑦2, … , 𝑦𝑜 , 𝑦𝑗 is assumed normal. Model: 𝑞(𝑦) Test samples: 𝑌𝑢𝑓𝑡𝑢 = 𝑦1, 𝑦2, … , 𝑦𝑜 , 𝑦𝑢 is unknow. if 𝑞(𝑦𝑢) < 𝜇, 𝑦𝑢 is abnormal. if 𝑞(𝑦𝑢) ≥ 𝜇, 𝑦𝑢 is normal.

Latent Space

Anomalies reside in the low probability density areas.

• 8-

Background

Correlation among data samples

Conventional Feature Learning Correlation-aware Feature Learning Anomaly Detection Anomaly Detection Feature Space Structure Space Graph Modeling

How to discover the normal pattern from both the feature level and structural level ?

• 9-

Problem Statement

Anomaly Detection

Given a set of input samples 𝓨 = {𝑦𝑗|𝑗 = 1, . . . , 𝑂}, each of which is associated with a 𝐺 dimension feature 𝐘𝑗 ∈ ℝ𝐺, we aim to learn a score function 𝑣(𝐘𝑗): ℝ𝐺 ↦ ℝ, to classify sample 𝑦𝑗 based on the threshold 𝜇 : 𝑧𝑗 = {1, 𝑗𝑔 𝑣(𝐘𝑗) ≥ 𝜇, 0, 𝑝𝑢ℎ𝑓𝑠𝑥𝑗𝑡𝑓. where 𝑧𝑗 denotes the label of sample 𝑦𝑗, with 0 being the normal class and 1 the anomalous class.

Notations

𝓗 : Graph. 𝓦 : Set of nodes in a graph. 𝓕 : Set of edges in a graph. 𝑂: Number of nodes. 𝐺 : Dimension of attribute. 𝐁 ∈ ℝ𝑂×𝑂 : Adjacency matrix

• f a network.

𝐘 ∈ ℝ𝑂×𝐺 : Feature matrix of all nodes.

• 10-

Method

CADGMM

Dual-Encoder Estimation network Feature Decoder Graph Construction

• 11-

Method

CADGMM

Graph Construction K-Nearest Neighbor e.g. K=5

Original feature: 𝓨 = {𝑦𝑗|𝑗 = 1, . . . , 𝑂} Find neighbors by K-NN: ൟ 𝓞𝑗 = {𝑦𝑗𝑙|𝑙 = 1, . . . , 𝐿 Model correlation as graph: 𝓗 = {𝓦, 𝓕, 𝐘} 𝓦 = {𝑤𝑗 = 𝑦𝑗|𝑗 = 1, . . . , 𝑂} 𝓕 = {𝑓𝑗𝑙 = (𝑤𝑗, 𝑤𝑗𝑙)|𝑤𝑗𝑙 ∈ 𝓞𝑗}

• 12-

Method

CADGMM

Feature Encoder e.g. MLP , CNN, LSTM Graph Encoder e.g. GAT Feature Decoder

• 13-

Method

CADGMM

Estimation network

Gaussian Mixture Model

Initial embedding: Z Membership: Z𝓝(𝑚ℳ) = 𝜏 Z𝓝 𝑚ℳ−1 W𝓝 𝑚ℳ−1 + b𝓝 𝑚ℳ−1 , Z𝓝(0) = Z 𝓝 = Softmax(Z𝓝(𝑀ℳ)), 𝓝 ∈ ℝ𝑂×𝑁 Parameter Estimation: 𝝂𝒏 =

෌𝑗=1

𝑂

𝓝𝑗,𝑛Z𝑗 ෌𝑗=1

𝑂

𝓝𝑗,𝑛

, 𝚻𝒏 =

෍

𝑗=1 𝑂

𝓝𝑗,𝑛(Z𝑗−𝝂𝒏)(Z𝑗−𝝂𝒏)T ෌𝑗=1

𝑂

𝓝𝑗,𝑛

Energy: EZ = −log σ𝑛=1

𝑁

σ𝑗=1

𝑂 𝓝𝑗,𝑛 𝑂

exp(−1

2(Z−𝝂𝒏)T𝚻𝑛 −1(Z−𝝂𝒏))

|2𝜌𝚻𝑛|

1 2

• 14-

ℒ = ||X − ෡ X||2

2 + 𝜇1EZ + 𝜇2 σ𝑛=1 𝑁

σ𝑗=1

𝑂 1 (𝚻𝒏)𝑗𝑗 + 𝜇3||Z||2 2

Method

Loss and Anomaly Score

𝑇𝑑𝑝𝑠𝑓 = EZ Loss Function: Anomaly Score:

• Rec. Error

Energy Covariance Penalty Embedding Penalty

𝑧𝑗 = {1, 𝑗𝑔 𝑣(𝐘𝑗) ≥ 𝜇, 0, 𝑝𝑢ℎ𝑓𝑠𝑥𝑗𝑡𝑓. 𝜇=Distribution(𝑇𝑑𝑝𝑠𝑓) Solution for Problem:

• 15-

Experiment

Datasets Baselines Evaluation Metrics

Precision Recall F1-Score OC-SVM Chen et al. 2001 IF Liu et al. 2008 DSEBM Zhai et al. 2016 DAGMM Zong et al. 2018 AnoGAN Schlegl et al. 2017 ALAD Zenati et al. 2018

• 16-

Experiment

Results

Consistent performance improvement!

• 17-

Experiment

Results

Less sensitive to noise data! More robust!

• 18-

Experiment

Results

• Fig. Impact of different K values of K-NN

algorithms in graph construction.

Less sensitive to hyper-parameters! Easy to use!

• 19-

Experiment

Results

Explainable and Effective!

• Fig. Embedding visualization on KDD99 (Blue indicates

the normal samples and orange the anomalies). (a). DAGMM (b). CADGMM

• 20-

Conclusion and Future Works

• Conventional feature learning models cannot

effectively capture the correlation among data samples for anomaly detection.

• We propose a general representation learning

framework to model the complex correlation among data samples for unsupervised anomaly detection.

• We plan to explore the correlation among samples

for extremely high-dimensional data sources like image or video.

• We plan to develop an adaptive and learnable graph

construction module for a more reasonable correlation modeling.

• 21-

Reference

• [OC-SVM] Chen, Y., Zhou, X.S., Huang, T.S.: One-class svm for learning in image
• retrieval. ICIP. 2001
• [IF] 8. Liu, F.T., Ting, K.M., Zhou, Z.H.: Isolation forest. ICDM. 2008.
• [DSEBM] Zhai, S., Cheng, Y., Lu, W., Zhang, Z.: Deep structured energy based

models for anomaly detection. ICML. 2016.

• [DAGMM] Zong, B., Song, Q., Min, M.R., Cheng, W., Lumezanu, C., Cho, D., Chen,

H.: Deep autoencoding gaussian mixture model for unsupervised anomaly

• detection. ICLR. 2018.
• [AnoGAN] Schlegl, T., Seeb•
• ck, P

., Waldstein, S.M., Schmidt-Erfurth, U., Langs, G.: Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. IPMI. 2017.

• [ALAD] Zenati, H., Romain, M., Foo, C.S., Lecouat, B., Chandrasekhar, V.:

Adversarially learned anomaly detection. ICDM. 2018.

• 22-

Thanks Thanks for listening!

Contact: isfanhy@hrbust.edu.cn Home Page: https://haoyfan.github.io/