Distributed Summary Statistics with Bro Vlad Grigorescu 1 > - - PowerPoint PPT Presentation

▶
distributed summary statistics with bro
SMART_READER_LITE
LIVE PREVIEW

Distributed Summary Statistics with Bro Vlad Grigorescu 1 > - - PowerPoint PPT Presentation

Dec 30, 2023 45 likes •307 views

Distributed Summary Statistics with Bro Vlad Grigorescu 1 > whoami Member of the Bro development team Senior Developer at Broala LLC Senior Information Security Engineer at Carnegie Mellon University

slide-1
SLIDE 1

Distributed Summary Statistics with Bro

Vlad Grigorescu

1

slide-2
SLIDE 2

> whoami

  • Member of the Bro development team
  • Senior Developer at Broala LLC
  • Senior Information Security Engineer at

Carnegie Mellon University

https://github.com/grigorescu @0f010d

2

slide-3
SLIDE 3

Goal

To develop statistics that can efficiently summarize network activity distributed

  • ver a large number of sensors, while

minimizing memory usage.

3

slide-4
SLIDE 4

Outline

  • 1. Observation examples
  • 2. What types of questions can we answer?
  • 3. SumStats Framework
  • 1. Overview
  • 2. Available Reducers
  • 4. Real-world usage

4

slide-5
SLIDE 5

Observation Examples

  • 192.168.2.13 received an NXDOMAIN

reply for a DNS A query of: host.244.ipoe2.subnets.khb.ttkdv.ru

5

slide-6
SLIDE 6

Observation Examples

  • 192.168.2.14 received a 403 Forbidden

when performing a POST to: http://sqm.microsoft.com/sqm/ Windows/sqmserver.dll

6

slide-7
SLIDE 7

Observation Examples

  • 192.168.2.15 sent an e-mail with an

application/x-dosexec attachment, with MD5 hash c84a46850de0a29483ed1f7a0b9897ab

7

slide-8
SLIDE 8

What types of questions can we answer?

  • Which source/dest IP pairs have the

lowest variance in TCP session byte counts?

  • Which ASNs have the highest number of

connections into your network?

  • Which IP source has connected to the

highest number of unique destinations?

8

slide-9
SLIDE 9

What types of questions can we answer?

  • In the past 24 hours, which clients have

sent the most failed DNS queries?

  • Which servers have received the most

failed DNS queries?

  • If we look at each IP’s ratio of failed to

total DNS queries, which IPs have had

  • ver 90% failures?

9

slide-10
SLIDE 10

SumStats Framework

  • A set of Bro scripts for generating

summary statistics

  • Tie into the existing Bro scripts to make
  • bservations about events in layers 2-7
  • Can threshold values to create notices,

which can prompt automated responses

  • Can query the current values for more

advanced use-cases scripts

10

slide-11
SLIDE 11

SumStats Framework: Philosphy

All summary statistics must be:

  • Highly memory efficient,
  • Streaming (the data is only seen once),
  • Mergable (distributable across thousands
  • f nodes, each of which see a subset of the

total traffic)

11

slide-12
SLIDE 12

SumStats Framework: Design

12

Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation!

Reducer

SumStat Notice!

slide-13
SLIDE 13

SumStats Framework: Design

13

Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation! Observation!

Reducer

SumStat Notice!

slide-14
SLIDE 14

SumStats Framework: Design

14

Reducer Reducer Reducer

Observation!

slide-15
SLIDE 15

SumStats Framework: Reducers

15

“Classic” Stats:

  • Average
  • Min
  • Max
  • Last
  • Sum
  • Std Dev
  • Variance
  • Cardinality

“Memory Efficient” Stats:

  • HyperLogLog
  • Top-k
  • Reservoir Sampling
slide-16
SLIDE 16

Reducers: HyperLogLog

16

  • Streaming algorithm for calculating

cardinality of huge datasets

  • Can calculate cardinality of 1 billion

elements with a relative accuracy of 2% using 1.5 KB of memory

  • Mergeable without any loss in accuracy
slide-17
SLIDE 17

Reducers: HyperLogLog

17

Which IP source has connected to the highest number of unique destinations? Let’s assume that you have a fully populated /8 network (16.5M hosts). We want to know the cardinality of destinations for each host. 16.5M ₒ1.5 KB ≈ 24 GB of RAM

slide-18
SLIDE 18

Reducers: Top-k

18

  • Streaming algorithm for finding the most

frequent elements in a dataset, in a space-saving way

  • Implementation of:

Metwally A, Agrawal D, El Abbadi A (2005) Efficient computation of frequent and top-k elements in data streams.

slide-19
SLIDE 19

Reducers: Top-k

19

Which IP source has connected to the highest number of unique destinations? Connect our HyperLogLog reducer to a Top-k reducer. Still assuming /8 network and 2% error; top talker connected to 1000 destinations ≈ 6 GB of RAM.

slide-20
SLIDE 20

Real-World Usage: Writing a SumStat Script

Which source/dest IP pairs have the lowest variance in TCP session byte counts?

20

slide-21
SLIDE 21

Real-World Usage: Writing a SumStat Script

  • 1. Observation:

event connection_state_remove(c: connection)

{ SumStats::observe("end_of_conn", [$key=cat(c$id$orig_h,c$id$resp_h)], [$num=c$orig$size+c$resp$size]); }

21

slide-22
SLIDE 22

Real-World Usage: Writing a SumStat Script

  • 2. Reducers:

local r1 = SumStats::Reducer( $stream="end_of_conn", $apply=set(SumStats::VARIANCE, SumStats::SUM) );

22

slide-23
SLIDE 23

Real-World Usage: Writing a SumStat Script

  • 3. SumStat:

SumStats::create( [$name="variance_of_orig_bytes", $epoch=5min, $reducers=set(r1), $threshold_val=(1-variance), #See note $threshold=0.9, $threshold_crossed=doNotice()#See note ]); Note: Slightly simplified for brevity where commented.

23

slide-24
SLIDE 24

Real-World Usage: scan.bro

Tracks the number of failed connection attempts (“port scans”) by source IP. Generates a notice when:

  • A source scans over 25 unique IPs on the

same port within 5 minutes, or

  • A source scans over 25 unique ports on

the same destination IP within 5 minutes.

24

slide-25
SLIDE 25

Real-World Usage: scan.bro

  • Carnegie Mellon sees approximately

3000-6000 failed connection attempts per second

  • scan.bro uses approx. 150 MB of RAM

and has detected 49,500 scans from July-November 2013

25

slide-26
SLIDE 26

Ongoing Work

  • Writing more SumStats scripts to detect:
  • DNS amplification attacks
  • Beaconing
  • Behavioral changes

26