Anomaly Detection Jia-Bin Huang Virginia Tech Spring 2019 - - PowerPoint PPT Presentation

Anomaly Detection

Jia-Bin Huang Virginia Tech

Spring 2019

ECE-5424G / CS-5824

Administrative

Anomaly Detection

• Motivation
• Developing an anomaly detection system
• Anomaly detection vs. supervised learning
• Choosing what features to use
• Multivariate Gaussian distribution

Anomaly Detection

• Motivation
• Developing an anomaly detection system
• Anomaly detection vs. supervised learning
• Choosing what features to use
• Multivariate Gaussian distribution

Anomaly detection example

• Dataset 𝑦(1), 𝑦(2), ⋯ , 𝑦(𝑛)
• New engine: 𝑦text
• Aircraft engine features:
• 𝑦1 = heat generated
• 𝑦2 = vibration intensity

𝑦2

vibration

𝑦1

heat

Density estimation

• Dataset 𝑦(1), 𝑦(2), ⋯ , 𝑦(𝑛)
• Is 𝑦text anomalous?

Model 𝒒 𝒚

• 𝑞 𝑦text < 𝜗 → flag anomaly
• 𝑞 𝑦text ≥ 𝜗 → OK

𝑦2

vibration

𝑦1

heat

Anomaly detection example

• Fraud detection
• 𝑦(𝑗)= features of user I’s activities
• Model 𝑞 𝑦 from data
• Identify unusual users by checking which have 𝑞 𝑦 < 𝜗
• Manufacturing
• Monitoring computers in a data center
• 𝑦(𝑗)= features of machine i
• 𝑦1= memory use, 𝑦2= number of disk accesses/sec
• 𝑦3= CPU load, 𝑦4 = CPU load/network traffic

Anomaly Detection

• Motivation
• Developing an anomaly detection system
• Anomaly detection vs. supervised learning
• Choosing what features to use
• Multivariate Gaussian distribution

Gaussian (normal) distribution

• Say 𝑦 ∈ 𝑆. If 𝑦 is a distributed Gaussian with mean 𝜈, variance 𝜏2.
• 𝑦 ∼ 𝑂(𝜈, 𝜏2)

𝜏 standard deviation

• 𝑞 𝑦; 𝜈, 𝜏2 =

1 2𝜌𝜏 exp − 𝑦−𝜈 2 2𝜏2

Gaussian distribution examples

Parameter estimation

• Dataset 𝑦(1), 𝑦(2), ⋯ , 𝑦(𝑛)
• 𝑦 ∼ 𝑂(𝜈, 𝜏2)
• Maximum likelihood estimation
• ො

𝜈 =

1 𝑛 σ𝑗=1 𝑛 𝑦(𝑗)

• ෢

𝜏2 =

1 𝑛 σ𝑗=1 𝑛 (𝑦 𝑗 − ො

𝜈)2

Density estimation

• Dataset 𝑦(1), 𝑦(2), ⋯ , 𝑦(𝑛)
• Each example 𝑦 ∈ 𝑆𝑜
• 𝑞 𝑦 = 𝑞(𝑦1; 𝜈1, 𝜏1

2) 𝑞 𝑦2; 𝜈2, 𝜏2 2 ⋯ 𝑞 𝑦𝑜; 𝜈𝑜, 𝜏𝑜 2

= Π𝑘 𝑞(𝑦𝑘; 𝜈𝑘, 𝜏

𝑘 2)

Anomaly detection algorithm

• 1. Choose features 𝑦𝑗 that you think might be indicative of anomalous

examples

• 2. Fit parameters 𝜈1, 𝜈2, ⋯ , 𝜈𝑜, 𝜏1

2, 𝜏2 2, ⋯ , 𝜏𝑜 2

• 𝜈𝑘 =

1 𝑛 σ𝑗=1 𝑛 𝑦𝑘 (𝑗)

• 𝜏

𝑘 2 = 1 𝑛 σ𝑗=1 𝑛 (𝑦𝑘 𝑗 − 𝜈𝑘)2

• 3. Given new example 𝑦, compute 𝑞 𝑦

𝑞 𝑦 = Π𝑘 𝑞(𝑦𝑘; 𝜈𝑘, 𝜏

𝑘 2)

Anomaly if 𝑞 𝑦 < 𝜗

Evaluation

• Assume we have some labeled data, of anomalous and non-

anomalous examples (𝑧 = 0 if normal, 𝑧 = 1 if anomalous)

• Training set 𝑦(1), 𝑦(2), ⋯ , 𝑦(𝑛) (assume normal examples)
• Cross-validation set: (𝑦𝑑𝑤

(1), 𝑧𝑑𝑤 (1)), (𝑦𝑑𝑤 (2), 𝑧𝑑𝑤 (2)), ⋯ , (𝑦𝑑𝑤 (𝑛𝑑𝑤), 𝑧𝑑𝑤 (𝑛𝑑𝑤))

• Test set: (𝑦𝑢𝑓𝑡𝑢

(1) , 𝑧𝑢𝑓𝑡𝑢 (1) ), (𝑦𝑢𝑓𝑡𝑢 (2) , 𝑧𝑢𝑓𝑡𝑢 (2) ), ⋯ , (𝑦𝑢𝑓𝑡𝑢 (𝑛𝑢𝑓𝑡𝑢), 𝑧𝑢𝑓𝑡𝑢 (𝑛𝑢𝑓𝑡𝑢))

Aircraft engines motivating example

• 10000 good (normal) engines
• 20 flawed engines (anomalous)
• Training set: 6000 good engines
• CV: 2000 good engines (y = 0), 10 anomalous (y = 1)
• Test: 2000 good engines (y = 0), 10 anomalous (y = 1)

Algorithm evaluation

• Fit model 𝑞(𝑦) on training set {𝑦 1 , ⋯ , 𝑦 𝑛 }
• On a cross-validation/test example 𝑦, predict
• 𝑧 = ቊ1

if 𝑞 𝑦 < 𝜗 (anomaly) if 𝑞 𝑦 ≥ 𝜗 (normal)

• Possible evaluation metrics:
• True positive, false positive, false negative, true negative
• Precision/Recall
• F1-score
• Can use cross-validation set to choose parameter 𝜗

Evaluation metric

• How about accuracy?
• Assume only 0.1% of the engines are anomaly (skewed classes)
• Declare every example as normal -> 99.9% accuracy!

Precision/Recall

• F1 score: 2

𝑄𝑆 𝑄+𝑆

Anomaly Detection

• Motivation
• Developing an anomaly detection system
• Anomaly detection vs. supervised learning
• Choosing what features to use
• Multivariate Gaussian distribution

Anomaly detection

• Very small number of positive

examples (y=1) (0-20 is common)

• Large number of negative (y=0)

examples

• Many different types of anomalies.

Hard for any algorithm to learn from positive examples what the anomalies look like

• Future anomalies may look nothing

like any of the anomalous examples we have seen so far Supervised learning Large number of positive and negative examples Enough positive examples for algorithm to get a sense of what positive are like, future positive examples likely to be similar to ones in training set.

Anomaly detection

• Fraud detection
• Manufacturing
• Monitoring machines in a data

center Supervised learning

• Email spam classification
• Weather prediction
• Cancer classification

Anomaly Detection

• Motivation
• Developing an anomaly detection system
• Anomaly detection vs. supervised learning
• Choosing what features to use
• Multivariate Gaussian distribution

Non-Gaussian features

log 𝑦

Error analysis for anomaly detection

Want 𝑞(𝑦) large for normal examples 𝑦 𝑞(𝑦) small for anomalous examples 𝑦 Most common problem: 𝑞(𝑦) is comparable (say both large) for normal and anomalous examples

Monitoring computers in a data center

• Choose features that might take on unusually large or small values in

the event of an anomaly

• 𝑦1 = memory use of computer
• 𝑦2 = number of dis accesses/sec
• 𝑦3 = CPU load
• 𝑦4 = network traffic
• 𝑦5 =

CPU load network traffic 𝑦5 = CPU load^2 network traffic

Anomaly Detection

• Motivation
• Developing an anomaly detection system
• Anomaly detection vs. supervised learning
• Choosing what features to use
• Multivariate Gaussian distribution

Motivating example: Monitoring machines in a data center

𝑦1 (CPU load) 𝑦2 (Memory use) 𝑦1 (CPU load) 𝑦2 (Memory use)

Multivariate Gaussian (normal) distribution

• 𝑦 ∈ 𝑆𝑜. Don’t model 𝑞 𝑦1 , 𝑞 𝑦2 , ⋯ separately
• Model 𝑞 𝑦 all in one go.
• Parameters: 𝜈 ∈ 𝑆𝑜, Σ ∈ 𝑆𝑜×𝑜 (covariance matrix)
• 𝑞 𝑦; 𝜈, Σ =

1 2𝜌 𝑜/2 Σ 1/2 exp − 𝑦 − 𝜈 ⊤Σ−1(𝑦 − 𝜈)

Multivariate Gaussian (normal) examples

Σ = 1 1 Σ = 0.6 0.6 Σ = 2 2 𝑦1 𝑦2 𝑦1 𝑦2 𝑦1 𝑦2

Multivariate Gaussian (normal) examples

Σ = 1 1 Σ = 0.6 1 Σ = 2 1 𝑦1 𝑦2 𝑦1 𝑦2 𝑦1 𝑦2

Multivariate Gaussian (normal) examples

Σ = 1 1 Σ = 1 0.5 0.5 1 Σ = 1 0.8 0.8 1 𝑦1 𝑦2 𝑦1 𝑦2 𝑦1 𝑦2

Anomaly detection using the multivariate Gaussian distribution

• 1. Fit model 𝑞 𝑦 by setting

𝜈 = 1 𝑛 ෍

𝑗=1 𝑛

𝑦(𝑗) Σ = 1 𝑛 ෍

𝑗=1 𝑛

(𝑦(𝑗)−𝜈)(𝑦(𝑗) − 𝜈)⊤ 2 Give a new example 𝑦, compute 𝑞 𝑦; 𝜈, Σ = 1 2𝜌 𝑜/2 Σ 1/2 exp − 𝑦 − 𝜈 ⊤Σ−1(𝑦 − 𝜈) Flag an anomaly if 𝑞 𝑦 < 𝜗

Original model 𝑞 𝑦1; 𝜈1, 𝜏1

2 𝑞 𝑦2; 𝜈2, 𝜏2 2 ⋯ 𝑞 𝑦𝑜; 𝜈𝑜, 𝜏𝑜 2

Manually create features to capture anomalies where 𝑦1, 𝑦2 take unusual combinations of values Computationally cheaper (alternatively, scales better) OK even if training set size is small

Original model

𝑞 𝑦; 𝜈, Σ = 1 2𝜌 𝑜/2 Σ 1/2 exp(− 𝑦 − 𝜈 ⊤Σ−1(𝑦

Things to remember

• Motivation
• Developing an anomaly detection system
• Anomaly detection vs. supervised learning
• Choosing what features to use
• Multivariate Gaussian distribution