an empirical evaluation of entropy based traffic anomaly
play

An Empirical Evaluation of Entropy- based Traffic Anomaly Detection - PowerPoint PPT Presentation

Sep 24, 2022 •96 likes •447 views

An Empirical Evaluation of Entropy- based Traffic Anomaly Detection George Nychis, Vyas Sekar, David Andersen, Hyong Kim, Hui Zhang Carnegie Mellon University Entropy-based Anomaly Detection Goal: detect abnormal behavior scan activity,

  1. An Empirical Evaluation of Entropy- based Traffic Anomaly Detection George Nychis, Vyas Sekar, David Andersen, Hyong Kim, Hui Zhang Carnegie Mellon University

  2. Entropy-based Anomaly Detection  Goal: detect abnormal behavior  scan activity, DDoS, bandwidth floods ...  Traditional: raw traffic volume ( insufficient)  e.g., total number of packets in an epoch  Modern : entropy-based traffic metrics  e.g., relative randomness in distribution of packets across ports Example Anomaly Entropy: Detectable Traffic Volume: Undetected 2

  3. Motivation Anomaly Detection Traffic Feature Timeseries NetFlow Alarm! Detection Data 3

  4. Motivation Anomaly Detection Traffic Feature Timeseries NetFlow sum(packets) A(pkts) Detection Data 3

  5. Motivation Anomaly Detection Traffic Feature Timeseries NetFlow H(addresses) A(addr) Detection Data Entropy-based Features: Dist. of packets across addresses 3

  6. Motivation Anomaly Detection Traffic Feature A(addr) Timeseries NetFlow A(port) H(ports) Detection Data Entropy-based Features: Distribution of packets across ports H(addresses) 3

  7. Motivation Anomaly Detection A(addr) Traffic Feature A(port) Timeseries NetFlow H(flow-size) A(FSD) Detection Data Entropy-based Features: Distribution of flow-sizes (in packets) H(addresses) H(ports) 3

  8. Motivation Anomaly Detection A(addr) A(port) Traffic Feature A(FSD) Timeseries NetFlow H(degree) A(deg) Detection Data Entropy-based Features: Distribution of host communication H(addresses) H(ports) H(flow-size) 3

  9. Motivation Anomaly Detection A(addr) A(port) Traffic Feature A(FSD) Timeseries NetFlow ???????? A(deg) Detection Data Entropy-based Features: H(addresses) H(ports) H(flow-size) H(degree) 3

  10. Motivation Anomaly Detection A(addr) A(port) Traffic Feature A(FSD) Timeseries NetFlow ???????? A(deg) Detection Data Entropy-based Features: H(addresses) H(ports) H(flow-size) H(degree)  Goal: understanding the features 3

  11. Motivation Anomaly Detection A(addr) A(port) Traffic Feature A(FSD) Timeseries NetFlow ???????? A(deg) Detection Data Entropy-based Features: H(addresses) H(ports) H(flow-size) H(degree)  Goal: understanding the features 1. How unique are their detection capabilities? 2. How effective are they? 3

  12. Analysis Method 5 one-month-long traces: NetFlow CMU-2005, CMU-2008, GATech-2008, Data GEANT-2005, Internet2-2006 4

  13. Analysis Method 5 one-month-long traces: NetFlow CMU-2005, CMU-2008, GATech-2008, Data GEANT-2005, Internet2-2006 H(addresses) H(ports) Entropy Timeseries H(flow-size) H(degree) 4

  14. Analysis Method 5 one-month-long traces: NetFlow CMU-2005, CMU-2008, GATech-2008, Data GEANT-2005, Internet2-2006 H(addresses) H(ports) Entropy Timeseries H(flow-size) H(degree) Are the distributions structurally similar? Timeseries Correlation 4

  15. Analysis Method 5 one-month-long traces: NetFlow CMU-2005, CMU-2008, GATech-2008, Data GEANT-2005, Internet2-2006 H(addresses) H(ports) Entropy Timeseries H(flow-size) H(degree) Are the A(addr) distributions A(port) structurally Anomaly Detection A(FSD) similar? A(deg) Timeseries Correlation 4

  16. Analysis Method 5 one-month-long traces: NetFlow CMU-2005, CMU-2008, GATech-2008, Data GEANT-2005, Internet2-2006 H(addresses) H(ports) Entropy Timeseries H(flow-size) H(degree) Are the A(addr) distributions A(port) structurally Anomaly Detection A(FSD) similar? A(deg) Anomaly Correlation Timeseries Correlation Goal(1): Uniqueness 4

  17. Analysis Method 5 one-month-long traces: NetFlow CMU-2005, CMU-2008, GATech-2008, Data GEANT-2005, Internet2-2006 H(addresses) H(ports) Entropy Timeseries H(flow-size) H(degree) Are the A(addr) distributions A(port) structurally Anomaly Detection A(FSD) similar? A(deg) Anomaly Correlation Timeseries Correlation Goal(1): Uniqueness 4

  18. Entropy Timeseries (February 2005) In-degree Out-degree Flow-size Src. Address Dst. Address Src. Port Dst. Port Raw traffic volume 5

  19. Entropy Timeseries (February 2005) In-degree Out-degree Flow-size Src. Address Dst. Address Src. Port Dst. Port Raw traffic volume 5

  20. Entropy Timeseries (February 2005) In-degree test  Out-degree Flow-size Src. Address Dst. Address Src. Port Dst. Port Raw traffic volume 5

  21. Entropy Timeseries (February 2005) In-degree test  Out-degree Flow-size Src. Address Dst. Address Src. Port Dst. Port Raw traffic volume 5

  22. Entropy Timeseries (February 2005) In-degree test  Out-degree Flow-size Src. Address Dst. Address Src. Port Dst. Port Raw traffic volume 5

  23. Analysis Method 5 one-month-long traces: NetFlow CMU-2005, CMU-2008, GATech-2008, Data GEANT-2005, Internet2-2006 H(addresses) H(ports) Entropy Timeseries H(flow-size) H(degree) Are the A(addr) distributions A(port) structurally Anomaly Detection A(FSD) similar? A(deg) Anomaly Correlation Timeseries Correlation Goal(1): Uniqueness 6

  24. Correlation in Entropy Timeseries  Pairwise correlation-scores for CMU-2005  All 4 other traces exhibit similar behavior! 7

  25. Why Entropy is Structurally Correlated 1. Port / Address Correlation  Properties of Network Traffic: - contribute X packets to address A - contribute X packets to port B … if hosts have few connections, and ports are uniformly random → similar distributions 8

  26. Why Entropy is Structurally Correlated 1. Port / Address Correlation  Properties of Network Traffic 2. Source / Destination Correlation  Flow accounting: - Bi-directional: Addr1(23) → Addr2(53) Bi-directional Saddr(23) Daddr(53) 8

  27. Why Entropy is Structurally Correlated 1. Port / Address Correlation  Properties of Network Traffic 2. Source / Destination Correlation  Flow accounting: - Uni-directional: Addr1 → Addr2 (23) Addr2 → Addr1 (53) Bi-directional Uni-directional Saddr(23) Saddr(23), Daddr(23) Daddr(53) Saddr(53), Daddr(53) Uni-directionality destroys 2 unique distributions 8

  28. Why Anomalies are Correlated  Root-cause analysis approach: no Remove Recompute Anomaly Analyze top-k flows entropy subsides? yes, cause!  Our results:  Ports & addresses: only detect alpha flows (correlation)  FSD: detects scans, Degree: SYN flood  FSD & Degree are unique ( no correlation ) 9

  29. Why Anomalies are Correlated  Root-cause analysis approach: no Remove Recompute Anomaly Analyze top-k flows entropy subsides? yes, cause! Traffic volume  Our results:  Ports & addresses: only detect alpha flows (correlation)  FSD: detects scans, Degree: SYN flood  FSD & Degree are unique ( no correlation ) 9

  30. Summary of Goal(1): Uniqueness  Strong correlation in ports and addresses  Flow-size and degree: unique  Structural correlation : properties of traffic  Anomaly correlation : types of anomalies seen 10

  31. Understanding Effectiveness Inject Synthetic Anomalies NetFlow Data Entropy Timeseries Anomaly Detection Anomaly Correlation Timeseries Correlation 11

  32. Best Distribution for an Anomaly?  Anomalies: BW Flood, Scanner, Multiple Scanners, Port Scan, and SYN Flood  Other Results:  BW Flood :  ports & addresses  already detectable FSD best by traffic volume detector  Scans:  difficult to detect  … FSD and degree 12

  33. Implications and Conclusions  Look beyond ports and addresses  Select complementary traffic distributions  Uni-directional accounting introduces biases in traffic distributions  Future Work: Can correlations be leveraged?  during anomalies found in flow-size & degree, correlation drops between ports & addresses 13

  34. Questions? 14

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

On Entropy in Network Traffic Anomaly Detection Jayro Santiago-Paz, Deni Torres-Roman.

On Entropy in Network Traffic Anomaly Detection Jayro Santiago-Paz, Deni Torres-Roman.

Introduction Feature Extraction Entropy Calculation Anomaly detection Classification Open Issues On Entropy in Network Traffic Anomaly Detection Jayro Santiago-Paz, Deni Torres-Roman. Cinvestav, Campus Guadalajara, Mexico November 2015

690 views • 24 slides
Conformal anomaly, entanglement entropy and boundaries Sergey N. Solodukhin University of Tours

Conformal anomaly, entanglement entropy and boundaries Sergey N. Solodukhin University of Tours

Conformal anomaly, entanglement entropy and boundaries Sergey N. Solodukhin University of Tours Talk at Oxford Holography Seminar February 9, 2016 Plan of the talk: 1. Brief review: local Weyl anomaly, entangle- ment entropy 2. Integral

812 views • 35 slides
Learning Rules for Anomaly Detection (LERAD) of Hostile Network Traffic Matt Mahoney Overview

Learning Rules for Anomaly Detection (LERAD) of Hostile Network Traffic Matt Mahoney Overview

Learning Rules for Anomaly Detection (LERAD) of Hostile Network Traffic Matt Mahoney Overview Prior Work in Network Anomaly Detection The 1999 DARPA Intrusion Detection Evaluation Packet Header Anomaly Detection (PHAD) Application

575 views • 18 slides
Trace Anomaly Matching and Exact Results For Entanglement Entropy Shamik Banerjee Kavli IPMU

Trace Anomaly Matching and Exact Results For Entanglement Entropy Shamik Banerjee Kavli IPMU

Trace Anomaly Matching and Exact Results For Entanglement Entropy Shamik Banerjee Kavli IPMU Based On arXiv: 1405.4876, arXiv: 1406.3038, SB July 22, 2014 Introduction Entanglement entropy is an important and useful quantity which finds

288 views • 14 slides
Practical Anomaly Detection based on Classifying Frequent Traffic Patterns Ignasi Paredes-Oliva 1

Practical Anomaly Detection based on Classifying Frequent Traffic Patterns Ignasi Paredes-Oliva 1

Practical Anomaly Detection based on Classifying Frequent Traffic Patterns Ignasi Paredes-Oliva 1 Ismael Castell-Uroz 1 Pere Barlet-Ros 1 Xenofontas Dimitropoulos 2 Josep Sol-Pareta 1 1 UPC BarcelonaTech, Spain

616 views • 36 slides
Evaluation of Anomaly Detection Method based on Pattern Recognition Romain Fontugne The

Evaluation of Anomaly Detection Method based on Pattern Recognition Romain Fontugne The

Evaluation of Anomaly Detection Method based on Pattern Recognition Romain Fontugne The Graduate University for Advanced Studies Yosuke Himura The University of Tokyo Kensuke Fukuda National Institute of Informatics CNRS-Wide

443 views • 18 slides
User Interface Evaluation Empirical evaluation Heuristic evaluation 1 CS 349 - UI evaluation

User Interface Evaluation Empirical evaluation Heuristic evaluation 1 CS 349 - UI evaluation

User Interface Evaluation Empirical evaluation Heuristic evaluation 1 CS 349 - UI evaluation When does UI evaluation happen? Design Testing and Implementation Development evaluation Testing 2 CS 349 - UI evaluation Types of tests

820 views • 20 slides
Entropy, Relative Entropy, Cross Entropy Entropy Entropy, H(x) is a measure of the uncertainty of

Entropy, Relative Entropy, Cross Entropy Entropy Entropy, H(x) is a measure of the uncertainty of

Entropy, Relative Entropy, Cross Entropy Entropy Entropy, H(x) is a measure of the uncertainty of a discrete random variable. Properties: H(x) >= 0 Entropy Entropy Lesser the probability for an event, larger the entropy.

471 views • 21 slides
Sensitivity of PCA for Traffic Anomaly Detection Evaluating the robustness of current best

Sensitivity of PCA for Traffic Anomaly Detection Evaluating the robustness of current best

Sensitivity of PCA for Traffic Anomaly Detection Evaluating the robustness of current best practices Haakon Ringberg 1 , Augustin Soule 2 , Jennifer Rexford 1 , Christophe Diot 2 1 Princeton University, 2 Thomson Research Outline Context

598 views • 15 slides
ICS 667 Advanced HCI Design Methods 09. Empirical Evaluation Dan Suthers Spring 2005 Methods

ICS 667 Advanced HCI Design Methods 09. Empirical Evaluation Dan Suthers Spring 2005 Methods

ICS 667 Advanced HCI Design Methods 09. Empirical Evaluation Dan Suthers Spring 2005 Methods Continued Analytic Evaluation From expert or theory driven Models, Metrics, Heuristics Empirical Evaluation (Testing)

742 views • 27 slides
An Evaluation of Effect of Packet Sampling on Anomaly Detection Method Takuya Motodate

An Evaluation of Effect of Packet Sampling on Anomaly Detection Method Takuya Motodate

An Evaluation of Effect of Packet Sampling on Anomaly Detection Method Takuya Motodate April 25, 2010 The 3rd CAIDA-WIDE-CASFI Joint Measurement Workshop @Osaka 1 2010 4 25 Background Anomaly Detection:

331 views • 20 slides
In Incorporating Feedback in into Tree-based Anomaly Detection Shubhomoy Das, Weng-Keen Wong,

In Incorporating Feedback in into Tree-based Anomaly Detection Shubhomoy Das, Weng-Keen Wong,

In Incorporating Feedback in into Tree-based Anomaly Detection Shubhomoy Das, Weng-Keen Wong, Alan Fern, Thomas G. Dietterich and Md Amran Siddiqui School of EECS Anomaly Detection Goal: Identify rare or strange objects 2 Anomaly

756 views • 51 slides
Anomaly Detection Algorithms for Malware Traffic Analysis using Tamper Resistant Features Dr.

Anomaly Detection Algorithms for Malware Traffic Analysis using Tamper Resistant Features Dr.

Anomaly Detection Algorithms for Malware Traffic Analysis using Tamper Resistant Features Dr. Patrick McDaniel Berkay Celik Fall 2015 Introduction Motivation Related Work Data Approach Experimental Results Comparison

391 views • 25 slides
Anomaly Detection of Trajectories Junier B. Oliva Anomaly Detection An anomaly (or outlier)

Anomaly Detection of Trajectories Junier B. Oliva Anomaly Detection An anomaly (or outlier)

Anomaly Detection of Trajectories Junier B. Oliva Anomaly Detection An anomaly (or outlier) in a dataset is an instance that is abnormal or unlikely based on the rest of the dataset If the instances in the dataset are labeled as

760 views • 19 slides
Empirical Evaluation of an Approach to Resource Constrained Test Suite Execution Gregory M.

Empirical Evaluation of an Approach to Resource Constrained Test Suite Execution Gregory M.

Empirical Evaluation of an Approach to Resource Constrained Test Suite Execution Gregory M. Kapfhammer Department of Computer Science Allegheny College (in conjunction with Mary Lou Soffa and Daniel Moss) Empirical Evaluation of an Approach

755 views • 30 slides
Internet traffic measurements Renata Teixeira (Inria) Why measure traffic? Performance

Internet traffic measurements Renata Teixeira (Inria) Why measure traffic? Performance

Internet traffic measurements Renata Teixeira (Inria) Why measure traffic? Performance analysis Anomaly and intrusion detec=on Network engineering Traffic at different granulari=es IP-level packets Capture per-packet

539 views • 40 slides
CDA 4253 FPGA System Design The PicoBlaze Microcontroller Hao Zheng Comp Sci & Eng U of

CDA 4253 FPGA System Design The PicoBlaze Microcontroller Hao Zheng Comp Sci & Eng U of

CDA 4253 FPGA System Design The PicoBlaze Microcontroller Hao Zheng Comp Sci & Eng U of South Florida Overview of PicoBlaze So:-core microcontroller in VHDL: portable to other plaAorms. Small: occupies ~20 CLBs. Respectable

424 views • 30 slides
SYSC3601 Microprocessor Systems Unit 6: Input/Output (I/O) Systems SYSC3601 1 Microprocessor

SYSC3601 Microprocessor Systems Unit 6: Input/Output (I/O) Systems SYSC3601 1 Microprocessor

SYSC3601 Microprocessor Systems Unit 6: Input/Output (I/O) Systems SYSC3601 1 Microprocessor Systems Topics/Reading 1. I/O Ports, design, and address decoding. 2. Programmed I/O structures 3. 82C55 - Programmable peripheral interface chip.

687 views • 35 slides
Device Programming Nima Honarmand (Based on slides by Don Porter and Mike Ferdman) Fall 2014::

Device Programming Nima Honarmand (Based on slides by Don Porter and Mike Ferdman) Fall 2014::

Fall 2014:: CSE 506:: Section 2 (PhD) Device Programming Nima Honarmand (Based on slides by Don Porter and Mike Ferdman) Fall 2014:: CSE 506:: Section 2 (PhD) Talking to Devices Device interface consists of registers and memories plus

549 views • 23 slides
Resource Elasticity in Distributed Deep Learning Andrew Or , Haoyu Zhang * , Michael J. Freedman

Resource Elasticity in Distributed Deep Learning Andrew Or , Haoyu Zhang * , Michael J. Freedman

Resource Elasticity in Distributed Deep Learning Andrew Or , Haoyu Zhang * , Michael J. Freedman Princeton University, *Google AI MLSys 2020 Resource allocation today 16 GPUs 2200 images/sec 32 GPUs 4000 images/sec 64 GPUs 5000 images/sec

943 views • 29 slides
Next Generation Microprocessor Functional Prototype SpaceWire Router Validation Results Jonas

Next Generation Microprocessor Functional Prototype SpaceWire Router Validation Results Jonas

Next Generation Microprocessor Functional Prototype SpaceWire Router Validation Results Jonas Ekergarn, Jan Andersson, Andreas Larsson, Daniel Hellstrm, Magnus Hjorth Aeroflex Gaisler AB Roland Weigand ESA SpaceWire Conference 2013

283 views • 13 slides
MACsec Encryption for the wired LAN Networking Services Team, Red Hat Sabrina Dubroca

MACsec Encryption for the wired LAN Networking Services Team, Red Hat Sabrina Dubroca

Proceedings of NetDev 1.1: The Technical Conference on Linux Networking (February 10th-12th 2016. Seville, Spain) MACsec Encryption for the wired LAN Networking Services Team, Red Hat Sabrina Dubroca sd@queasysnail.net Netdev1.1, Seville, 2016

1.33k views • 47 slides
The new FCNB portal Bringing pension regulatory processes online Jake van der Laan, CIO At the

The new FCNB portal Bringing pension regulatory processes online Jake van der Laan, CIO At the

The new FCNB portal Bringing pension regulatory processes online Jake van der Laan, CIO At the end of this webinar, you should: have a good sense of how our new portal works understand the new functionality to be added soon relating to

769 views • 19 slides
From Furlough to Funding The portal is read y... but are YOU ? A MUST WATCH GUIDE FOR BUSINESSES

From Furlough to Funding The portal is read y... but are YOU ? A MUST WATCH GUIDE FOR BUSINESSES

From Furlough to Funding The portal is read y... but are YOU ? A MUST WATCH GUIDE FOR BUSINESSES ACCESSING THE HMRC CORONAVIRUS JOB RETENTION SCHEME PORTAL Providing 150+ SME s with co - sourcing solutions for their Numbers , People & S y

380 views • 36 slides

More recommend