SLIDE 1
Automatically Evading Classifiers
A Case Study on PDF Malware Classifiers
Weilin Xu David Evans Yanjun Qi University of Virginia
Machine Learning is Solving Our Problems
2
Fake
Spam IDS Malware Fake Accounts … …
Automatically Evading Classifiers A Case Study on PDF Malware - - PowerPoint PPT Presentation
Automatically Evading Classifiers A Case Study on PDF Malware - - PowerPoint PPT Presentation
Automatically Evading Classifiers A Case Study on PDF Malware Classifiers Weilin Xu David Evans Yanjun Qi University of Virginia Machine Learning is Solving Our Problems Fake Fake Spam IDS
SLIDE 2
SLIDE 3
3
SLIDE 4
4
SLIDE 5
Machine Learning is Eating the World
Data Scientist
Security Expert 5
?
SLIDE 6
Machine Learning is Eating the World
Data Scientist
Security Expert 6
No! Security is different.
SLIDE 7
Goal: Understand classifiers under attack. Results: Vulnerable to automated evasion.
Security Tasks are Different: Adversary Adapts
7
SLIDE 8
Building Machine Learning Classifiers
8
Trained Classifier Labelled Training Data ML Algorithm
Training (Supervised Learning)
Feature Extraction Vectors
SLIDE 9
Assumption: Training Data is Representative
9
Labelled Training Data ML Algorithm Feature Extraction Vectors
Deployment
Malicious / Benign Operational Data Trained Classifier
Training (Supervised Learning)
SLIDE 10
Results: Evaded PDF Malware Classifiers
PDFrate* [ACSAC’12] Hidost [NDSS’13]
Accuracy
0.9976 0.9996
False Negative Rate
0.0000 0.0056
False Negative Rate with Adversary
1.0000 1.0000
10
* Mimicus [Oakland ’14], an open source reimplementation of PDFrate.
SLIDE 11
Results: Evaded PDF Malware Classifiers
PDFrate* [ACSAC’12] Hidost [NDSS’13]
Accuracy
0.9976 0.9996
False Negative Rate
0.0000 0.0056
False Negative Rate with Adversary
1.0000 1.0000
11
Very robust against “strongest conceivable mimicry attack”.
* Mimicus [Oakland ’14], an open source reimplementation of PDFrate.
SLIDE 12
Variants
12
Clone Benign PDFs Malicious PDF
Mutation
01011001101
Variants Variants
Select Variants
✓ ✓ ✗ ✓
Based on Genetic Programming
Automated Evasion Approach
SLIDE 13
Variants
13
Clone Benign PDFs Malicious PDF
Mutation
01011001101
Variants Variants
Select Variants
✓ ✓ ✗ ✓
Based on Genetic Programming
Automated Evasion Approach
/Catalog /Pages /JavaScript eval(‘…’); /Root
Modified Parser
Extract Me If You Can:
Abusing PDF Parsers in Malware Detectors
Curtis Carmony,et al.
SLIDE 14
Variants
14
Clone Benign PDFs Malicious PDF
Mutation
01011001101
Variants Variants
Select Variants
✓ ✓ ✗ ✓
Based on Genetic Programming
Automated Evasion Approach
/Catalog /Pages /JavaScript eval(‘…’); /Root
Mutation
Variants From Benign
Insert / Replace / Delete
SLIDE 15
Variants
15
Clone Benign PDFs Malicious PDF
Mutation
01011001101
Variants Variants
Select Variants
✓ ✓ ✗ ✓
Based on Genetic Programming
Automated Evasion Approach
/Catalog /Pages /JavaScript eval(‘…’); /Root
Mutation
Variants From Benign
128 546
Insert / Replace / Delete
SLIDE 16
Variants
16
Clone Benign PDFs Malicious PDF
Mutation
01011001101
Variants Variants
Select Variants
✓ ✓ ✗ ✓
Based on Genetic Programming
Automated Evasion Approach
/Catalog /Pages /JavaScript eval(‘…’); /Root
Mutation
Variants From Benign
128 546
Insert / Replace / Delete
SLIDE 17
Variants
17
Clone Benign PDFs Malicious PDF
Mutation
01011001101
Variants Variants
Select Variants
✓ ✓ ✗ ✓
Based on Genetic Programming
Automated Evasion Approach
/Catalog /Pages /JavaScript eval(‘…’); /Root
Mutation
Variants From Benign
128 546 128
Insert / Replace / Delete
SLIDE 18
Variants
18
Clone Benign PDFs Malicious PDF
Mutation
01011001101
Variants Variants
Select Variants
✓ ✓ ✗ ✓
Based on Genetic Programming
Automated Evasion Approach
/Catalog /Pages /JavaScript eval(‘…’); /Root
Mutation
Variants From Benign
128
Insert / Replace / Delete
SLIDE 19
Variants
19
Clone Benign PDFs Malicious PDF
Mutation
01011001101
Variants Variants
Select Variants
✓ ✓ ✗ ✓
Based on Genetic Programming
Automated Evasion Approach
/Catalog /Pages /JavaScript eval(‘…’); /Root
Mutation
Variants From Benign
128
Insert / Replace / Delete
SLIDE 20
Variants
20
Clone Benign PDFs Malicious PDF
Mutation
01011001101
Variants Variants
Select Variants
✓ ✓ ✗ ✓
Based on Genetic Programming
Automated Evasion Approach
SLIDE 21
Variants
21
Clone Benign PDFs Malicious PDF
Mutation
01011001101
Variants Variants
Select Variants
✓ ✓ ✗ ✓
Based on Genetic Programming
Automated Evasion Approach
Fitness Function Oracle Target Classifier
f(x)
Malicious? Score Fitness Score Variants
SLIDE 22
Variants
22
Clone Benign PDFs Malicious PDF
Mutation
01011001101
Variants Variants
Select Variants
✓ ✓ ✗ ✓
Based on Genetic Programming
Automated Evasion Approach
Fitness Function Oracle Target Classifier
f(x)
Malicious? Score Fitness Score Variants Malicious Benign
SLIDE 23
Variants
23
Clone Benign PDFs Malicious PDF
Mutation
01011001101
Variants Variants
Select Variants
✓ ✓ ✗ ✓
Based on Genetic Programming
Automated Evasion Approach
SLIDE 24
Results: Evaded PDFrate 100%
24
Original Malware Seeds
SLIDE 25
Results: Evaded PDFrate 100%
25
Original Malware Seeds Evasive Variants
SLIDE 26
Evaded PDFrate with Adjusted Threshold
26
Original Malware Seeds Evasive Variants Evasive Variants with lower threshold
SLIDE 27
Results: Evaded Hidost 100%
27
Original Malware Seeds
SLIDE 28
Results: Evaded Hidost 100%
28
Original Malware Seeds Evasive Variants
SLIDE 29
29
Difficulty varies by seed Simple mutations often work Complex mutations sometimes needed. Difficulty varied by targets: PDFrate: 6 days to evade all Hidost: 2 days to evade all
Results: Accumulated Evasion Rate
SLIDE 30
Cross-Evasion Effects
30
PDF Malware Seeds
Hidost
Evasive PDF Malware (against Hidost) Automated Evasion
PDFrate
387/500 Evasive (77.4%) 3/500 Evasive (0.6%)
Gmail’s classifier is secure?
SLIDE 31
Cross-Evasion Effects
31
PDF Malware Seeds
Hidost
Evasive PDF Malware (against Hidost) Automated Evasion
PDFrate
387/500 Evasive (77.4%) 3/500 Evasive (0.6%)
Gmail’s classifier is secure? different.
SLIDE 32
Evading Gmail’s Classifier
32
Evasion rate on : 135/380 (35.5%)
SLIDE 33
Evading Gmail’s Classifier
33
Evasion rate on : 179/380 (47.1%)
SLIDE 34
Conclusion
34
Source Code: http://EvadeML.org