Autom atic anom aly detection using NfSen Wim Biemolt, SURFnet - - PowerPoint PPT Presentation

▶
autom atic anom aly detection using nfsen
SMART_READER_LITE
LIVE PREVIEW

Autom atic anom aly detection using NfSen Wim Biemolt, SURFnet - - PowerPoint PPT Presentation

May 22, 2023 323 likes •596 views

Autom atic anom aly detection using NfSen Wim Biemolt, SURFnet Werner Schram, SURFnet 14/ 12/ 2007 Autom atic anom aly detection using NfSen - SURFnet and netflow anomaly detection - NERD - NfSen - PeakFlow SP - Currently used detection

slide-1
SLIDE 1

14/ 12/ 2007

Autom atic anom aly detection using NfSen

Wim Biemolt, SURFnet Werner Schram, SURFnet

slide-2
SLIDE 2

SURFnet – Automatic anomaly detection using NfSen 1

Autom atic anom aly detection using NfSen

  • SURFnet and netflow anomaly detection
  • NERD
  • NfSen
  • PeakFlow SP
  • Currently used detection methods
  • DDos
  • Botnet
  • Holt-Winters aberrant behavior
slide-3
SLIDE 3

SURFnet – Automatic anomaly detection using NfSen 2

SURFnet and netflow anom aly detection

  • NERD v1
  • Developed by TNO
  • Based on cflowd
  • cflowd is no longer supported
  • NERD v2
  • Initially developed by TNO
  • Has serious performance problems
  • NfSen can do the same but without the

performance problems

slide-4
SLIDE 4

SURFnet – Automatic anomaly detection using NfSen 3

NfSen

  • Netflow Sensor (NfSen) is a
  • network statistics tool
  • Developed by Peter Haag
  • Currently in active development
  • Alert plug-in system
  • Generic plug-in system
  • Some plug-ins already available
slide-5
SLIDE 5

SURFnet – Automatic anomaly detection using NfSen 4

NfSen

slide-6
SLIDE 6

SURFnet – Automatic anomaly detection using NfSen 5

DDos detection

  • Simple flow analysis
  • based on NERD v1 DDos detection
  • using a low threshold and a high threshold
  • Rules for traffic between those thresholds
  • Custom thresholds for high load services
slide-7
SLIDE 7

SURFnet – Automatic anomaly detection using NfSen 6

Expected traffic

slide-8
SLIDE 8

SURFnet – Automatic anomaly detection using NfSen 7

Definitively Conspicuous Traffic

slide-9
SLIDE 9

SURFnet – Automatic anomaly detection using NfSen 8

Border cases

slide-10
SLIDE 10

SURFnet – Automatic anomaly detection using NfSen 9

High load servers

slide-11
SLIDE 11

SURFnet – Automatic anomaly detection using NfSen 10

Custom thresholds

slide-12
SLIDE 12

SURFnet – Automatic anomaly detection using NfSen 11

DDos interface: report

slide-13
SLIDE 13

SURFnet – Automatic anomaly detection using NfSen 12

DDos interface: Details

slide-14
SLIDE 14

SURFnet – Automatic anomaly detection using NfSen 13

Botnet detection

  • Hosts infected by viruses connect to hosts known as

botnet controllers

  • List of botnet controllers are available, for example:

http: / / www.bleedingthreats.net/ rules/ bleeding-botcc.rules

  • Our plug-in logs all hosts that connect to known botnet

controllers

  • Automatically reports to incident report system using

IODEF

slide-15
SLIDE 15

SURFnet – Automatic anomaly detection using NfSen 14

<?xml version="1.0" encoding="iso-8859-1"?> <io:IODEF-Document xmlns:io="urn:ietf:params:xml:ns:iodef-1.0” lang="en"> <io:Incident purpose="reporting"> <io:IncidentID name="overflow.surfnet.nl&#10;">#33408</io:IncidentID> <io:StartTime>2007-08-13T15:07:47+02:00</io:StartTime> <io:EndTime>2007-08-13T21:06:12+02:00</io:EndTime> <io:ReportTime>2007-08-13T21:12:07+02:00</io:ReportTime> <io:Assessment> <io:Impact type="user"/> </io:Assessment> <io:Contact> <io:ContactName>Werner Schram</io:ContactName> </io:Contact> <io:EventData> <io:Method> <io:Reference> <io:ReferenceName>botnet</io:ReferenceName> </io:Reference> </io:Method> <io:Flow> <io:System category="source"> <io:Node> <io:Address category="ipv4-addr">192.168.1.1</io:Address> <io:Counter type="flow">20</io:Counter> </io:Node> </io:System> <io:System category="target"> <io:Node> <io:Address category="ipv4-addr">192.168.1.2</io:Address> </io:Node> <io:Service ip_version="4" ip_protocol="6"> <io:Port>80</io:Port> </io:Service> </io:System> </io:Flow> </io:EventData> <io:AdditionalData dtype="string">Generated by NFSen</io:AdditionalData> </io:Incident> </io:IODEF-Document>

Botnet I ODEF reports

slide-16
SLIDE 16

SURFnet – Automatic anomaly detection using NfSen 15

Holt-W inters aberrant behavior detection

  • Uses information about periodic data to predict

aberrant behavior.

slide-17
SLIDE 17

SURFnet – Automatic anomaly detection using NfSen 16

Holt-W inters: Exam ple

slide-18
SLIDE 18

SURFnet – Automatic anomaly detection using NfSen 17

Holt-W inters: Original im plem entation

Noise Periodic information Trend Prediction

slide-19
SLIDE 19

SURFnet – Automatic anomaly detection using NfSen 18

Lim itations of the

  • riginal im plem entation
  • The original algorithm has three parameters which

define:

  • the weight of historical data
  • the weight of the trend
  • the amount of expected noise
  • The original algorithm has a constant learning rate
  • If a low learning rate is used, the selection of the initial values is critical.

This will introduce false positives for a long time.

  • With a high learning rate, the model will likely be overfitted. This will

introduce false negatives

  • The trend parameter has no significant influence with

the resolution we are using

slide-20
SLIDE 20

SURFnet – Automatic anomaly detection using NfSen 19

Holt-W inters: Multiple trends

Network traffic time series often show multiple recurring patterns, for example a weekly trend:

slide-21
SLIDE 21

SURFnet – Automatic anomaly detection using NfSen 20

Holt-W inters: Multiple periods

Noise Weekly period Daily Period

slide-22
SLIDE 22

SURFnet – Automatic anomaly detection using NfSen 21

Learning rate

Fixed learning rate: The first pattern is overweighted Adaptive learning rate: The weight of the first pattern is relative to the rest

slide-23
SLIDE 23

SURFnet – Automatic anomaly detection using NfSen 22

Real data exam ple

slide-24
SLIDE 24

SURFnet – Automatic anomaly detection using NfSen 23

Holt W inters: Usage Exam ple

Normal ICMP Traffic Aberrant ICMP Traffic: Caused by DDos attack by Stormworm botnet

slide-25
SLIDE 25

SURFnet – Automatic anomaly detection using NfSen 24

Holt W inters: Other possible uses

Common SMTP Traffic Last week SMTP Traffic

slide-26
SLIDE 26

SURFnet – Automatic anomaly detection using NfSen 25

Wim Biemolt Wim.Biemolt@surfnet.nl www.surfnet.nl Werner Schram Werner.Schram@surfnet.nl www.surfnet.nl