On the Impact of Flow Monitoring Configuration Petr Velan et al. - - PowerPoint PPT Presentation

▶
on the impact of flow monitoring configuration
SMART_READER_LITE
LIVE PREVIEW

On the Impact of Flow Monitoring Configuration Petr Velan et al. - - PowerPoint PPT Presentation

Apr 13, 2023 127 likes •387 views

On the Impact of Flow Monitoring Configuration Petr Velan et al. velan@ics.muni.cz Institute of Computer Science, Masaryk University April 20, 2020 Flow Monitoring Recapitulation Network Flow Monitoring Network Flow Monitoring Used for

slide-1
SLIDE 1

On the Impact of Flow Monitoring Configuration

Petr Velan et al. velan@ics.muni.cz

Institute of Computer Science, Masaryk University

April 20, 2020

slide-2
SLIDE 2

Flow Monitoring Recapitulation

Network Flow Monitoring

Network Flow Monitoring Used for monitoring of large networks Scales better than DPI Supported by network equipment (NetFlow and IPFIX protocols) Use of Flow Monitoring Data Network management

Network planning (using long term statistics) Network debugging

Security

Incident handling Policy verification Attack detection Anomaly detection

  • P. Velan· On the Impact of Flow Monitoring Configuration· April 20, 2020

2 / 19

slide-3
SLIDE 3

Flow Monitoring Recapitulation

Flow Record Creation

A B

Individual packet Receiving side Sending side Time

  • P. Velan· On the Impact of Flow Monitoring Configuration· April 20, 2020

3 / 19

slide-4
SLIDE 4

Flow Monitoring Recapitulation

Flow Record Creation

A B

Individual packet Receiving side Sending side Time Active Timeout

  • P. Velan· On the Impact of Flow Monitoring Configuration· April 20, 2020

3 / 19

slide-5
SLIDE 5

Flow Monitoring Recapitulation

Flow Record Creation

A B

Individual packet Receiving side Sending side Time Active Timeout Active Timeout Active Timeout

  • P. Velan· On the Impact of Flow Monitoring Configuration· April 20, 2020

3 / 19

slide-6
SLIDE 6

Flow Monitoring Recapitulation

Flow Record Creation

A B

Individual packet Receiving side Sending side Time Active Timeout Active Timeout Active Timeout

A B

Active Timeout Active Timeout

  • P. Velan· On the Impact of Flow Monitoring Configuration· April 20, 2020

3 / 19

slide-7
SLIDE 7

Flow Monitoring Recapitulation

Flow Record Creation

A B

Individual packet Receiving side Sending side Time Active Timeout Active Timeout Active Timeout

A B

Active Timeout Active Timeout

A B

Inactive Timeout Active Timeout Inactive Timeout

  • P. Velan· On the Impact of Flow Monitoring Configuration· April 20, 2020

3 / 19

slide-8
SLIDE 8

Flow Monitoring Recapitulation

Flow Record Creation

Flow Expiration Conditions Active timeout Inactive timeout Protocol specific reasons (e.g. end of TCP connection) Resource restrictions (e.g. limited flow cache size) Exporter shutdown

  • P. Velan· On the Impact of Flow Monitoring Configuration· April 20, 2020

4 / 19

slide-9
SLIDE 9

The Important Lesson

Configuration of flow monitoring is essential!

If you are publishing results based on flow data:

Always include description of flow monitoring configuration.

  • P. Velan· On the Impact of Flow Monitoring Configuration· April 20, 2020

5 / 19

slide-10
SLIDE 10

Flow Expiration Configuration Impact

Flow Expiration Configuration Impact

What is affected by flow expiration timeouts? Flow export

Larger inactive timeout causes flows to be cached longer which increases computing and memory requirements Smaller timeouts increase number of generated flow records, which increases computing and export bandwidth requirements

Flow collection

Larger number of flow records increases computing and storage space requirements

Flow analysis

Larger number of flow records increases computing requirements Different number of flow records with different properties influences analysis results

  • P. Velan· On the Impact of Flow Monitoring Configuration· April 20, 2020

6 / 19

slide-11
SLIDE 11

Flow Expiration Configuration Impact

Number of Created Flows

How large is impact of flow expiration timeouts on flow creation? To find out, we: Selected datasets

The CAIDA Anonymized Internet Traces 2015 Dataset (1.1 billion of packets) A Realistic Cyber Defense Dataset (CSE-CIC-IDS2018) (3.6 million of packets)

Computed flows using a range of different timeouts

We were interested only in number of flows Python tool – unsuitable for the CAIDA dataset (too slow, large memory consumption) C++ tool, fast computation the flow records

Analysed the results

  • P. Velan· On the Impact of Flow Monitoring Configuration· April 20, 2020

7 / 19

slide-12
SLIDE 12

Analysis of Flow Expiration Timeouts Impact

Impact of the Inactive Timeout (CAIDA Dataset, TCP)

10 20 30 40 50 60 70 80 90 100 110 120 130 1.5x107 2x107 2.5x107 3x107 3.5x107 4x107 # of flows Interpacket gap frequency 0.07 0.06 0.05 0.04 0.03 0.02 0.00 Interpacket gap / inactive timeout length (seconds) # of flows Interpacket gap frequency

  • P. Velan· On the Impact of Flow Monitoring Configuration· April 20, 2020

8 / 19

slide-13
SLIDE 13

Analysis of Flow Expiration Timeouts Impact

Impact of the Inactive Timeout (CAIDA Dataset, TCP)

Changing inactive timeout setting: 30 s -> 10 s causes an increase of almost 26% flows records 60 s -> 30 s causes an increase of almost 16% flows records 60 s -> 10 s causes an increase of almost 44% flows records A 45 second interpacket gap is quite common. Number of generated flows increases by 1.2% for 46 s and 45 s inactive timeout setting.

  • P. Velan· On the Impact of Flow Monitoring Configuration· April 20, 2020

9 / 19

slide-14
SLIDE 14

Analysis of Flow Expiration Timeouts Impact

Impact of the Active Timeout (CSE-CIC Dataset, TCP)

10 20 30 40 50 60 70 80 90 100 450000 500000 550000 600000 650000 700000 750000 # of flows 0.175 0.150 0.125 0.100 0.075 0.050 0.025 0.000 Connection length frequency # of flows Connection length frequency Connection / Active timeout length (seconds)

  • P. Velan· On the Impact of Flow Monitoring Configuration· April 20, 2020

10 / 19

slide-15
SLIDE 15

Analysis of Flow Expiration Timeouts Impact

Impact of the Active Timeout (CSE-CIC Dataset, TCP)

The impact of active timeout is more complicated to evaluate Correlation between active timeout and connection length is weaker Number of multiples of active timeout that can fit into a connection length is also important (e.g. 10 s active timeout) Interpacket gaps influence the result as well Decreasing the active timeout from 300 seconds to 120 seconds increases the number of flow records only by 3% (for this dataset).

  • P. Velan· On the Impact of Flow Monitoring Configuration· April 20, 2020

11 / 19

slide-16
SLIDE 16

Analysis of Flow Expiration Timeouts Impact

Impact of the Combination of Both Timeouts (CAIDA Dataset, TCP)

50 100 150 200 250 300 Active timeout 20 40 60 80 100 120 Inactive timeout 1.6*107 1.7*107 1.8*107 1.9*107 2.0*107 2.1*107 2.2*107 2.3*107 2.4*107

  • P. Velan· On the Impact of Flow Monitoring Configuration· April 20, 2020

12 / 19

slide-17
SLIDE 17

Analysis of Flow Expiration Timeouts Impact

Impact of the Combination of Both Timeouts (CAIDA Dataset, TCP)

Following can be derived from analysing the timeouts: Specifics of used transport protocols such as timeouts and common connection lengths (e.g. HTTP keepalive) can be observed as faster changes in the number of flows (colour changes) Different protocols (UDP, TCP, ICMP) and networks (datasets) behave differently Decreasing the active timeout from 300 seconds to 120 seconds and the inactive timeout from 30 seconds to 10 seconds increases the number of flow records by 26% (for this dataset).

  • P. Velan· On the Impact of Flow Monitoring Configuration· April 20, 2020

13 / 19

slide-18
SLIDE 18

Analysis of Flow Expiration Timeouts Impact

Impact on Flow Data Analysis

The magnitude of impact of flow timeouts depends on a type of analysis, for example: Port scan detection will not be affected because port scans always generate only short flow records Covert dictionary attack can run slowly from multiple attackers to avoid detection. Their detection might be be affected by flow timeout settings DDoS attacks detection such as Slowloris detection will be affected the most

  • P. Velan· On the Impact of Flow Monitoring Configuration· April 20, 2020

14 / 19

slide-19
SLIDE 19

Analysis of Flow Expiration Timeouts Impact

Impact on a Slowloris Attack Detection

There is a Slowloris attack in the CSE-CIC dataset, which we analysed: Successful attack establishes a TCP connection, sends first part of HTTP header and continues to send a small part of request header every 100 seconds (first after 78 s). The server responds with Bad Request response after approximately 2470 seconds. When the attack is successful, attacker does not get response for initial part of GET request and closes connection after 108 seconds In the most severe case, TCP connection cannot be established and attacker gives up after sending tree SYN packet in 3 seconds.

  • P. Velan· On the Impact of Flow Monitoring Configuration· April 20, 2020

15 / 19

slide-20
SLIDE 20

Analysis of Flow Expiration Timeouts Impact

Impact on a Slowloris Attack Detection

20 40 60 80 100 120 140 160 180 200 220 240 Active timeout 20 40 60 80 100 120 Inactive timeout 1.0*104 2.0*104 3.0*104 4.0*104 5.0*104 6.0*104 7.0*104 8.0*104 9.0*104 100s interpacket gap 108s failed HTTP GET 54s fraction 78s request chunk

  • P. Velan· On the Impact of Flow Monitoring Configuration· April 20, 2020

16 / 19

slide-21
SLIDE 21

Analysis of Flow Expiration Timeouts Impact

Impact on a Slowloris Attack Detection

Caution is required when relying on flow data for Slowloris detection Large enough timeouts should be used Preprocessing using flow aggregation might be needed When using machine learning, ensure that flow expiration conditions remain the same throughout the whole process

  • P. Velan· On the Impact of Flow Monitoring Configuration· April 20, 2020

17 / 19

slide-22
SLIDE 22

Closing Remarks

Recommended Flow Expiration Configuration

When determining the flow expiration timeouts, the following should be taken into consideration: Timeouts should be tuned for different protocols (e.g. TCP, UDP) separately Timeouts must be based on processing delay requirements (data freshness) The number of generated flows must be within performance limitations of the monitoring system Timeouts must be accounted for by all network data analysis tools

  • P. Velan· On the Impact of Flow Monitoring Configuration· April 20, 2020

18 / 19

slide-23
SLIDE 23

Closing Remarks

Take Away Message – Reminder

Configuration of flow monitoring is essential!

If you are publishing results based on flow data:

Always include description of flow monitoring configuration. Thank you for your attention

  • P. Velan· On the Impact of Flow Monitoring Configuration· April 20, 2020

19 / 19

slide-24
SLIDE 24