practical anomaly detection based on classifying frequent
play

Practical Anomaly Detection based on Classifying Frequent Traffic - PowerPoint PPT Presentation

Aug 17, 2022 •240 likes •616 views

Practical Anomaly Detection based on Classifying Frequent Traffic Patterns Ignasi Paredes-Oliva 1 Ismael Castell-Uroz 1 Pere Barlet-Ros 1 Xenofontas Dimitropoulos 2 Josep Sol-Pareta 1 1 UPC BarcelonaTech, Spain

  1. Practical Anomaly Detection based on Classifying Frequent Traffic Patterns Ignasi Paredes-Oliva 1 Ismael Castell-Uroz 1 Pere Barlet-Ros 1 Xenofontas Dimitropoulos 2 Josep Solé-Pareta 1 1 UPC BarcelonaTech, Spain {iparedes,icastell,pbarlet,pareta}@ac.upc.edu 2 ETH Zürich, Switzerland fontas@tik.ee.ethz.ch 15 th IEEE Global Internet Symposium (GI) Orlando, FL, United States March 30 th , 2012

  2. Introduction Related Work Our Proposal Performance Evaluation Conclusions Outline Introduction 1 Related Work 2 Our Proposal 3 Performance Evaluation 4 Conclusions 5 2 / 21

  3. Introduction Related Work Our Proposal Performance Evaluation Conclusions Outline Introduction 1 Related Work 2 Our Proposal 3 Performance Evaluation 4 Conclusions 5 3 / 21

  4. Introduction Related Work Our Proposal Performance Evaluation Conclusions The problem Growth of cyber-attacks 1 Anomaly detection systems not widely deployed e.g., too many false positives, complex black boxes Anomaly classification and root-cause analysis are still open issues e.g., manual analysis → error-prone, complex, slow and expensive 2 Our goal Simple system for automatic anomaly detection and classification High classification accuracy and low false positives Conceptually simple working scheme 1 Kim-Kwang Raymond Choo, The cyber threat landscape: Challenges and future research directions, Computers & Security, 2011. 2 M. Molina et al., Anomaly Detection in Backbone Networks: Building a Security Service Upon an Innovative Tool. TNC 2010. 4 / 21

  5. Introduction Related Work Our Proposal Performance Evaluation Conclusions The problem Growth of cyber-attacks 1 Anomaly detection systems not widely deployed e.g., too many false positives, complex black boxes Anomaly classification and root-cause analysis are still open issues e.g., manual analysis → error-prone, complex, slow and expensive 2 Our goal Simple system for automatic anomaly detection and classification High classification accuracy and low false positives Conceptually simple working scheme 1 Kim-Kwang Raymond Choo, The cyber threat landscape: Challenges and future research directions, Computers & Security, 2011. 2 M. Molina et al., Anomaly Detection in Backbone Networks: Building a Security Service Upon an Innovative Tool. TNC 2010. 4 / 21

  6. Introduction Related Work Our Proposal Performance Evaluation Conclusions Outline Introduction 1 Related Work 2 Our Proposal 3 Performance Evaluation 4 Conclusions 5 5 / 21

  7. Introduction Related Work Our Proposal Performance Evaluation Conclusions Related work and contributions Many proposals on anomaly detection Anomaly classification marginally studied Contributions of this paper Novel approach for automatic anomaly detection and classification based on classifying frequent traffic patterns Evaluated using data from two large networks High classification accuracy and low false positives ratio System deployed in the Catalan NREN 6 / 21

  8. Introduction Related Work Our Proposal Performance Evaluation Conclusions Related work and contributions Many proposals on anomaly detection Anomaly classification marginally studied Contributions of this paper Novel approach for automatic anomaly detection and classification based on classifying frequent traffic patterns Evaluated using data from two large networks High classification accuracy and low false positives ratio System deployed in the Catalan NREN 6 / 21

  9. Introduction Related Work Our Proposal Performance Evaluation Conclusions Outline Introduction 1 Related Work 2 Our Proposal 3 Performance Evaluation 4 Conclusions 5 7 / 21

  10. Introduction Related Work Our Proposal Performance Evaluation Conclusions System Overview Two phases: Offline: build model to classify anomalies Online: use model to classify incoming traffic Freq. Machine Feature Model Item-Set Learning Extraction Mining Freq. Feature Classification Item-Set Extraction Mining 8 / 21

  11. Introduction Related Work Our Proposal Performance Evaluation Conclusions Frequent Item-Set Mining Originally used in market basket analysis to find out products that were frequently bought together and make appealing offers ( e.g., beer and chips) What is an item-set? compact summarization of elements occurring together Why is it useful for anomaly detection? Many attacks involve high volume of flows with common features e.g., Port Scan: many flows with same sIP and dIP 9 / 21

  12. Introduction Related Work Our Proposal Performance Evaluation Conclusions Frequent Item-Set Mining Originally used in market basket analysis to find out products that were frequently bought together and make appealing offers ( e.g., beer and chips) What is an item-set? compact summarization of elements occurring together Why is it useful for anomaly detection? Many attacks involve high volume of flows with common features e.g., Port Scan: many flows with same sIP and dIP 9 / 21

  13. Introduction Related Work Our Proposal Performance Evaluation Conclusions Frequent Item-Set Mining Originally used in market basket analysis to find out products that were frequently bought together and make appealing offers ( e.g., beer and chips) What is an item-set? compact summarization of elements occurring together Why is it useful for anomaly detection? Many attacks involve high volume of flows with common features e.g., Port Scan: many flows with same sIP and dIP 9 / 21

  14. Introduction Related Work Our Proposal Performance Evaluation Conclusions Frequent Item-Set Mining Originally used in market basket analysis to find out products that were frequently bought together and make appealing offers ( e.g., beer and chips) What is an item-set? compact summarization of elements occurring together Why is it useful for anomaly detection? Many attacks involve high volume of flows with common features e.g., Port Scan: many flows with same sIP and dIP 9 / 21

  15. Introduction Related Work Our Proposal Performance Evaluation Conclusions Frequent Item-Set Mining Port Scan example sIP dIP sPort dPort 1 st flow X.77.17.59 Y.88.243.209 41393 21209 2 nd flow X.77.17.59 Y.88.243.209 41393 54766 3 rd flow X.77.17.59 Y.88.243.209 41393 31448 4 th flow X.77.17.59 Y.88.243.209 41393 58514 ... 2911 th flow X.77.17.59 Y.88.243.209 41393 48732 sIP dIP sPort dPort item-set X.77.17.59 Y.88.243.209 41393 * Need further information per item-set in order to classify it 10 / 21

  16. Introduction Related Work Our Proposal Performance Evaluation Conclusions Frequent Item-Set Mining Port Scan example sIP dIP sPort dPort 1 st flow X.77.17.59 Y.88.243.209 41393 21209 2 nd flow X.77.17.59 Y.88.243.209 41393 54766 3 rd flow X.77.17.59 Y.88.243.209 41393 31448 4 th flow X.77.17.59 Y.88.243.209 41393 58514 ... 2911 th flow X.77.17.59 Y.88.243.209 41393 48732 sIP dIP sPort dPort item-set X.77.17.59 Y.88.243.209 41393 * Need further information per item-set in order to classify it 10 / 21

  17. Introduction Related Work Our Proposal Performance Evaluation Conclusions Frequent Item-Set Mining Port Scan example sIP dIP sPort dPort 1 st flow X.77.17.59 Y.88.243.209 41393 21209 2 nd flow X.77.17.59 Y.88.243.209 41393 54766 3 rd flow X.77.17.59 Y.88.243.209 41393 31448 4 th flow X.77.17.59 Y.88.243.209 41393 58514 ... 2911 th flow X.77.17.59 Y.88.243.209 41393 48732 sIP dIP sPort dPort item-set X.77.17.59 Y.88.243.209 41393 * Need further information per item-set in order to classify it 10 / 21

  18. Introduction Related Work Our Proposal Performance Evaluation Conclusions Feature Extraction Computed features for each frequent item-set Value Defined Undefined Defined Src IP/Dst IP True False Src/Dst Port Port Number NaN Protocol Protocol Number NaN URG/ACK/PSH/RST/SYN/FIN True False # Bytes / # Packets Bytes per Packet (bpp) # Packets / # Flows Packet per Flow (ppf) 11 / 21

  19. Introduction Related Work Our Proposal Performance Evaluation Conclusions Building the classifier (offline) Goal: build model taking into account manually labeled frequent item-sets Output classes Anomalous: DoS (DDoS, SYN/ACK/UDP/ICMP floods), Network Scans (ICMP/Other Network Scans), Port Scans (SYN/ACK/UDP Port Scans) Normal (legitimate traffic) Unknown (not normal and did not fit in any anomalous class) Labeled item-sets + features + output classes are given to the C5.0 algorithm (machine learning) → output: classification model 12 / 21

  20. Introduction Related Work Our Proposal Performance Evaluation Conclusions Building the classifier (offline) Goal: build model taking into account manually labeled frequent item-sets Output classes Anomalous: DoS (DDoS, SYN/ACK/UDP/ICMP floods), Network Scans (ICMP/Other Network Scans), Port Scans (SYN/ACK/UDP Port Scans) Normal (legitimate traffic) Unknown (not normal and did not fit in any anomalous class) Labeled item-sets + features + output classes are given to the C5.0 algorithm (machine learning) → output: classification model 12 / 21

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Anomaly Detection of Trajectories Junier B. Oliva Anomaly Detection An anomaly (or outlier)

Anomaly Detection of Trajectories Junier B. Oliva Anomaly Detection An anomaly (or outlier)

Anomaly Detection of Trajectories Junier B. Oliva Anomaly Detection An anomaly (or outlier) in a dataset is an instance that is abnormal or unlikely based on the rest of the dataset If the instances in the dataset are labeled as

760 views • 19 slides
Detecting Attacks Anomaly-based Detection Signature-based Signature-based (Misuse)

Detecting Attacks Anomaly-based Detection Signature-based Signature-based (Misuse)

Detecting Attacks Anomaly-based Detection Signature-based Signature-based (Misuse) Detection Intrusion Detection Host-based Network-based Boriana Ditcheva and Lisa Fowler Active/Passive University of North Carolina at

401 views • 12 slides
In Incorporating Feedback in into Tree-based Anomaly Detection Shubhomoy Das, Weng-Keen Wong,

In Incorporating Feedback in into Tree-based Anomaly Detection Shubhomoy Das, Weng-Keen Wong,

In Incorporating Feedback in into Tree-based Anomaly Detection Shubhomoy Das, Weng-Keen Wong, Alan Fern, Thomas G. Dietterich and Md Amran Siddiqui School of EECS Anomaly Detection Goal: Identify rare or strange objects 2 Anomaly

756 views • 51 slides
Anomaly Based Network Intrusion Detection with Unsupervised Outlier Detection Jiong Zhang and

Anomaly Based Network Intrusion Detection with Unsupervised Outlier Detection Jiong Zhang and

Anomaly Based Network Intrusion Detection with Unsupervised Outlier Detection Jiong Zhang and Mohammad Zulkernine School of Computing Queens University, Kingston Ontario, Canada K7L 3N6 {zhang, mzulker} @cs.queensu.ca Abstract- Anomaly

284 views • 6 slides
Opprentice: Towards Practical and Automatic Anomaly Detection Through Machine Learning Dapeng Liu

Opprentice: Towards Practical and Automatic Anomaly Detection Through Machine Learning Dapeng Liu

Opprentice: Towards Practical and Automatic Anomaly Detection Through Machine Learning Dapeng Liu , Youjian Zhao, Haowen Xu, Yongqian Sun, Dan Pei, Jiao Luo, Xiaowei Jing, Mei Feng 2015/12/3 Dapeng Liu (liudp10@mails.tsinghua.edu.cn) KPIs and

545 views • 42 slides
An Empirical Evaluation of Entropy- based Traffic Anomaly Detection George Nychis, Vyas Sekar,

An Empirical Evaluation of Entropy- based Traffic Anomaly Detection George Nychis, Vyas Sekar,

An Empirical Evaluation of Entropy- based Traffic Anomaly Detection George Nychis, Vyas Sekar, David Andersen, Hyong Kim, Hui Zhang Carnegie Mellon University Entropy-based Anomaly Detection Goal: detect abnormal behavior scan activity,

447 views • 34 slides
What is an anomaly? Alastair Rushworth Data Scientist DataCamp Anomaly Detection in R Defining

What is an anomaly? Alastair Rushworth Data Scientist DataCamp Anomaly Detection in R Defining

DataCamp Anomaly Detection in R ANOMALY DETECTION IN R What is an anomaly? Alastair Rushworth Data Scientist DataCamp Anomaly Detection in R Defining the term anomaly Anomaly: a data point or collection of data points that do not follow the

870 views • 21 slides
Anomaly Detection Jia-Bin Huang Virginia Tech Spring 2019 ECE-5424G / CS-5824 Administrative

Anomaly Detection Jia-Bin Huang Virginia Tech Spring 2019 ECE-5424G / CS-5824 Administrative

Anomaly Detection Jia-Bin Huang Virginia Tech Spring 2019 ECE-5424G / CS-5824 Administrative Anomaly Detection Motivation Developing an anomaly detection system Anomaly detection vs. supervised learning Choosing what features

514 views • 34 slides
Data Mining II Anomaly Detection Heiko Paulheim Anomaly Detection Also known as Outlier

Data Mining II Anomaly Detection Heiko Paulheim Anomaly Detection Also known as Outlier

Data Mining II Anomaly Detection Heiko Paulheim Anomaly Detection Also known as Outlier Detection Automatically identify data points that are somehow different from the rest Working assumption: There are considerably

984 views • 72 slides
Data Mining II Anomaly Detection Heiko Paulheim Anomaly Detection Also known as Outlier

Data Mining II Anomaly Detection Heiko Paulheim Anomaly Detection Also known as Outlier

Data Mining II Anomaly Detection Heiko Paulheim Anomaly Detection Also known as Outlier Detection Automatically identify data points that are somehow different from the rest Working assumption: There are considerably

905 views • 74 slides
Anomaly-based Bot Server (and more!) Detection Jim Binkley jrb@cs.pdx.edu Portland State

Anomaly-based Bot Server (and more!) Detection Jim Binkley jrb@cs.pdx.edu Portland State

Anomaly-based Bot Server (and more!) Detection Jim Binkley jrb@cs.pdx.edu Portland State University Computer Science outline background experimental flow tuples botnet server mesh detection botnet client mesh detection

364 views • 23 slides
PANACEA: AUTOMATING ATTACK CLASSIFICATION FOR ANOMALY-BASED NETWORK INTRUSION DETECTION SYSTEMS

PANACEA: AUTOMATING ATTACK CLASSIFICATION FOR ANOMALY-BASED NETWORK INTRUSION DETECTION SYSTEMS

PANACEA: AUTOMATING ATTACK CLASSIFICATION FOR ANOMALY-BASED NETWORK INTRUSION DETECTION SYSTEMS DAMIANO BOLZONI, SANDRO ETALLE AND PIETER HARTEL DISTRIBUTED AND EMBEDDED SECURITY GROUP TWENTE SECURITY LAB 10+ YEARS OF RESEARCH OVER ANOMALY

160 views • 14 slides
Learning Rules for Anomaly Detection (LERAD) of Hostile Network Traffic Matt Mahoney Overview

Learning Rules for Anomaly Detection (LERAD) of Hostile Network Traffic Matt Mahoney Overview

Learning Rules for Anomaly Detection (LERAD) of Hostile Network Traffic Matt Mahoney Overview Prior Work in Network Anomaly Detection The 1999 DARPA Intrusion Detection Evaluation Packet Header Anomaly Detection (PHAD) Application

575 views • 18 slides
15-388/688 - Practical Data Science: Anomaly detection and mixture of Gaussians J. Zico Kolter

15-388/688 - Practical Data Science: Anomaly detection and mixture of Gaussians J. Zico Kolter

15-388/688 - Practical Data Science: Anomaly detection and mixture of Gaussians J. Zico Kolter Carnegie Mellon University Spring 2018 1 Outline Anomalies and outliers Multivariate Gaussian Mixture of Gaussians 2 Outline Anomalies and

511 views • 34 slides
Advanced Analytics in Business [D0S07a] Big Data Platforms & Technologies [D0S06a]

Advanced Analytics in Business [D0S07a] Big Data Platforms & Technologies [D0S06a]

Advanced Analytics in Business [D0S07a] Big Data Platforms & Technologies [D0S06a] Unsupervised Learning Anomaly Detection Overview Frequent itemset and association rule mining Other itemset extensions Clustering Anomaly detection 2

1.01k views • 85 slides
<Title> Yiqun Hu, SP Group Agenda Condition monitoring & anomaly detection

<Title> Yiqun Hu, SP Group Agenda Condition monitoring & anomaly detection

Data Council Singapre 2019 Autoencoder Forest for Anomaly Detection from IoT Time Series <Title> Yiqun Hu, SP Group Agenda Condition monitoring & anomaly detection Autoencoder for anomaly detection Autoencoder

540 views • 27 slides
Hardware Modeling 3 Timing Anomalies Peter Puschner slides credits: P. Puschner, R. Kirner, B.

Hardware Modeling 3 Timing Anomalies Peter Puschner slides credits: P. Puschner, R. Kirner, B.

Hardware Modeling 3 Timing Anomalies Peter Puschner slides credits: P. Puschner, R. Kirner, B. Huber VU 2.0 182.101 SS 2015 Timing Anomalies Obstacles to building

469 views • 33 slides
Lecture Two: Working with high dimensional data In ancient times they had no statistics so

Lecture Two: Working with high dimensional data In ancient times they had no statistics so

Lecture Two: Working with high dimensional data In ancient times they had no statistics so they had to fall back on lies. Stephen Leacock Recommended books The Elements of Statistical Learning: Data Pattern Recognition and Machine

380 views • 34 slides
Self Learning Networks An Overview Alvaro Retana aretana@cisco.com Distinguished Engineer,

Self Learning Networks An Overview Alvaro Retana aretana@cisco.com Distinguished Engineer,

Self Learning Networks An Overview Alvaro Retana aretana@cisco.com Distinguished Engineer, Cisco Services Slides by JP Vasseur and Jeff Apcar. What Self Learning Networks is About SLN is fundamentally a hyper-distributed analytics

456 views • 28 slides
Intrusion Detection Systems CSE497b - Spring 2007 Introduction Computer and Network Security

Intrusion Detection Systems CSE497b - Spring 2007 Introduction Computer and Network Security

Intrusion Detection Systems CSE497b - Spring 2007 Introduction Computer and Network Security Professor Jaeger www.cse.psu.edu/~tjaeger/cse497b-s07/ CSE497b Introduction to Computer and Network Security - Spring 2007 - Professor Jaeger

328 views • 18 slides
Insider Threat Insider Threat (Database Intrusion Detection) 1 Insider Threats: Motivation and

Insider Threat Insider Threat (Database Intrusion Detection) 1 Insider Threats: Motivation and

Insider Threat Insider Threat (Database Intrusion Detection) 1 Insider Threats: Motivation and Challenges Challenges Mission critical information = High value target Threatens Government organizations and large corporations

435 views • 39 slides
Motivation Both human- and computer-generated programs sometimes contain data-flow anomalies .

Motivation Both human- and computer-generated programs sometimes contain data-flow anomalies .

Motivation Both human- and computer-generated programs sometimes contain data-flow anomalies . These anomalies result in the program being worse, in some sense, than it was intended to be. Data-flow analysis is useful in locating, and sometimes

409 views • 40 slides
Machine Learning-based Anomaly Detection for Post-silicon Bug Diagnosis Andrew DeOrio , Qingkun

Machine Learning-based Anomaly Detection for Post-silicon Bug Diagnosis Andrew DeOrio , Qingkun

Machine Learning-based Anomaly Detection for Post-silicon Bug Diagnosis Andrew DeOrio , Qingkun Li, Matthew Burgess and Valeria Bertacco University of Michigan University of Illinois Verification trends Wilson Research Group and Mentor

201 views • 16 slides
Patient and Public Engagement Group New Congenital Heart Disease review 14 th January 2015 New

Patient and Public Engagement Group New Congenital Heart Disease review 14 th January 2015 New

New Congenital Heart Disease Review Patient and Public Engagement Group New Congenital Heart Disease review 14 th January 2015 New Congenital Heart Disease Review Agenda Welcome and overview of the afternoon Since we last met

618 views • 26 slides

More recommend