S A V A N T Security Analytics & Visualisation for Advanced - - PowerPoint PPT Presentation

▶
s a v a n t
SMART_READER_LITE
LIVE PREVIEW

S A V A N T Security Analytics & Visualisation for Advanced - - PowerPoint PPT Presentation

Jun 23, 2023 562 likes •1.17k views

S A V A N T Security Analytics & Visualisation for Advanced Network Threats Paul D. Hood & Kristian Kocher OxCERT OxCERT Paul D. Hood Security Operations Lead Kristian Kocher UNIX Security Systems Administrator

slide-1
SLIDE 1

S A V A N T

Security Analytics & Visualisation for

Paul D. Hood & Kristian Kocher

OxCERT

Advanced Network Threats

slide-2
SLIDE 2

Paul D. Hood Security Operations Lead Kristian Kocher UNIX Security Systems Administrator

OxCERT

paul.hood@it.ox.ac.uk kristian.kocher@it.ox.ac.uk

slide-3
SLIDE 3
slide-4
SLIDE 4

S A V A N T

The ElasticSIEM

slide-5
SLIDE 5

SAVANT

As network speeds increase, NSM data balloons to multi-GB per day We are at 40GB+ NetFlow per day

2.5Gbps 2002

10Gbps 2008

40Gbps 2018 (?)

NSM Trends

slide-6
SLIDE 6

SAVANT

Traditional logging methods aggregate data into large compressed archive files Traditional search techniques rely on decompression on the CLI (ie, zgrep)

NSM Trends

slide-7
SLIDE 7

SAVANT

NSM Trends

slide-8
SLIDE 8

SAVANT

This method scales very poorly as data size continues to increase

NSM Trends

slide-9
SLIDE 9

SAVANT

NSM Trends

Individual analyses are taking longer Number of sources are expanding Analyst time is a precious resource We are losing this war

slide-10
SLIDE 10

SAVANT

Aggregated and parallelised search has emerged as the only viable option

NSM Trends

slide-11
SLIDE 11

Our solution

slide-12
SLIDE 12

SAVANT

SAVANT is built on a stack of interlocking software components Each performs a vital function

The Stack

E L K

lasticSearch

  • gstash

ibana

slide-13
SLIDE 13

SAVANT

ELASTICSEARCH is a high-speed indexing engine, able to store and retrieve data as JSON objects Anything can be indexed

The Stack

slide-14
SLIDE 14

SAVANT

LOGSTASH is a flexible log shipping and storage application. Logstash translates log entries from near-any source into a JSON

  • bject for storage in ElasticSearch

The Stack

slide-15
SLIDE 15

SAVANT

KIBANA is the front-end, forming the user interface and search functionality Kibana can visualize huge quantities of data at extreme speed, thanks to Python Lucene

The Stack

slide-16
SLIDE 16

SAVANT

The three components allow:

  • JSON data objects
  • Resilient storage
  • Search, retrieval,

analytics The Stack

slide-17
SLIDE 17

SAVANT

nBox

Elastic Elastic Elastic Logstash Kibana Search

NetFlow

slide-18
SLIDE 18

SAVANT

NSM

Elastic Elastic Elastic Logstash Kibana Search

NSM/logs/alerts

FileBeat

slide-19
SLIDE 19

SAVANT

PacketBeat

Elastic Elastic Elastic Kibana Search

Protocols (DNS)

slide-20
SLIDE 20

Proof of Concept

slide-21
SLIDE 21

SAVANT

Hardware is required to handle each major functional stage; Tool Server / Appliance Data Node Replica Node Search Node

PoC

slide-22
SLIDE 22

SAVANT

PoC

slide-23
SLIDE 23

SAVANT

PoC

slide-24
SLIDE 24

SAVANT

PoC Insights

In general, when building a cluster

  • f this magnitude it will require;
  • Data nodes: High I/O, multiple

cores, 32GB+ of RAM, RAID-1

  • Search nodes: maximum CPU and

RAM, system on SSD storage

  • Replica nodes: can be practically

anything, but better hardware contributes more to search metrics

slide-25
SLIDE 25

SAVANT

PoC Insights

There are a few ‘gotchas’ which persist when building these clusters:

Each ElasticSearch node can have a maximum of 31GB RAM due to JVM pointer compression limitations Assigning the full 31GB causes huge ‘stop the world’ garbage collection

BUT…

slide-26
SLIDE 26

SAVANT

PoC Insights

0.3Tbit/sec NetFlow is a big ask… Snapshotting takes time and resource… GeoIP is not terribly performant…. Build your own Logstash codec Schedule for low-usage hours Only enable it for logs/alerts, not NetFlow…

slide-27
SLIDE 27

SAVANT

30-60 days

Online, searchable data

6-12 months

Snapshotted archives

<60 secs

Search performance target

Design Metrics

slide-28
SLIDE 28

Scaling

slide-29
SLIDE 29

SAVANT

4 fibre taps 40Gb/s line rate ~320Gb/s total

Evolved Scaling

slide-30
SLIDE 30

SAVANT

Very few (FLOSS/cheap) analysis tools can handle 40G+ line rates

We have a theoretical 0.3TBit/sec to fully monitor and analyse… L

Evolved Scaling

The best we can do is ~10G…

slide-31
SLIDE 31

SAVANT

Evolved Scaling 10Gbps output streams

40Gb + 40Gb + 40Gb + 40Gb

slide-32
SLIDE 32

SAVANT

Tool Servers/Appliances

40Gb + 40Gb + 40Gb + 40Gb

Evolved Scaling

slide-33
SLIDE 33

SAVANT

NetFlow

40Gb + 40Gb + 40Gb + 40Gb

Evolved Scaling NSM Protocols

slide-34
SLIDE 34

SAVANT

Effectively we can compartmentalise capability into ~10G units (Rx/Tx)

Following this scaling principle, we can scale this tech to 100G line rates

Evolved Scaling

A 40G-capable cluster is composed

  • f the same fundamentals as a 10G
slide-35
SLIDE 35

The SIEM

slide-36
SLIDE 36
slide-37
SLIDE 37

SAVANT

Aggregation

slide-38
SLIDE 38

SAVANT

Aggregation…

slide-39
SLIDE 39

SAVANT

The SIEM

Single unified interface Fully aggregated Multi-TB index search capacity

slide-40
SLIDE 40

SAVANT

The SIEM

External intelligence Internal investigations Arbitrary IoC sources

slide-41
SLIDE 41

SAVANT

The SIEM

slide-42
SLIDE 42

Case Studies

slide-43
SLIDE 43

Use Case 1 –Threat Hunting

slide-44
SLIDE 44

Use Case 1 –Threat Hunting

slide-45
SLIDE 45

Use Case 1 –Threat Hunting

slide-46
SLIDE 46

Use Case 1 –Threat Hunting

slide-47
SLIDE 47

Use Case 1 –Threat Hunting

slide-48
SLIDE 48

Use Case 1 –Threat Hunting

Total Investigation time:3 minutes

slide-49
SLIDE 49

Use Case 2 – Host Identification

slide-50
SLIDE 50

Use Case 2 – Host Identification

slide-51
SLIDE 51

Use Case 2 – Host Identification

slide-52
SLIDE 52

Use Case 3 – Strategic NSM

slide-53
SLIDE 53

Use Case 4 – Deep Analysis

slide-54
SLIDE 54

Use Case 4 – Deep Analysis

slide-55
SLIDE 55

Use Case 4 – Deep Analysis

Total Investigation time:

2 minutes

slide-56
SLIDE 56

Use Case 5 – All of the above

slide-57
SLIDE 57

Thank Y

  • u!
slide-58
SLIDE 58

https://www.infosec.ox.ac.uk/