1 in a Nutshell 2019 Pass the S ALT Workshop Overview 2 - - PowerPoint PPT Presentation

▶
1
SMART_READER_LITE
LIVE PREVIEW

1 in a Nutshell 2019 Pass the S ALT Workshop Overview 2 - - PowerPoint PPT Presentation

Apr 07, 2024 257 likes •810 views

Elastic S tack for S ecurity Monitoring 1 in a Nutshell 2019 Pass the S ALT Workshop Overview 2 Introduction to Elastic S tack Beats Logstash Elasticsearch Kibana Elastic S tack Alerting and S ecurity Introductory Workshop! 3

slide-1
SLIDE 1

Elastic S tack for S ecurity Monitoring in a Nutshell

2019 Pass the S ALT Workshop

1

slide-2
SLIDE 2

Overview

Introduction to Elastic S tack Beats Logstash Elasticsearch Kibana Elastic S tack Alerting and S ecurity

2

slide-3
SLIDE 3

Introductory Workshop!

  • This is an introductory workshop
  • Y
  • u probably won’ t hear/ see a

lot of new things if you have:

  • Used Elastic S

tack in the past;

  • Took the Elastic training…

;

  • Followed S

ANS S EC455, S EC555, FOR572, etc.;

  • If you are stuck, please

do not suffer in silence!

3

slide-4
SLIDE 4

Workshop VM

  • ais_workshop_xubuntu-18.04.2-desktop-amd64
  • VMware Workstation, Player, or Fusion
  • Y
  • u can try VirtualBox too, but you are on your own with that…

sorry! 

  • 8 GB RAM
  • 30-50 GB disk space
  • Keyboard layout: EN-US !!!
  • Workshop VM (Ubuntu) user/ pass: user / Workshop1234%
  • Normally, it should not require password for login and sudo

4

slide-5
SLIDE 5

About David

  • Managing partner at Alzette Information S

ecurity (@ AlzetteInfoS ec)

  • Network penetration testing, security architectures, security

monitoring, incident response

  • Instructor at S

ANS Institute: FOR572

  • BS

ides Luxembourg organizer https:/ / bsideslux.lu

  • Twitter: @

DavidS zili

  • E-mail: david.szili@

alzetteinfosec.com

  • Blog: http:/ / j umpespj ump.blogspot.com

5

slide-6
SLIDE 6

About Eva

  • Managing partner at Alzette Information S

ecurity (@ AlzetteInfoS ec)

  • Web application penetration testing, source code review, security

monitoring

  • CyberWayFinder
  • BS

ides Luxembourg organizer https:/ / bsideslux.lu

  • Twitter: @

EvaS zilagyiS ec

  • E-mail: eva.szilagyi@

alzetteinfosec.com

  • Blog: http:/ / j umpespj ump.blogspot.com

6

slide-7
SLIDE 7

Introduction to Elastic S tack

2019 Pass the S ALT Workshop

7

slide-8
SLIDE 8

About Elastic S tack

What is Elastic Stack?

  • 4 main components:
  • Elasticsearch
  • Logstash
  • Kibana
  • Beats
  • And several other smaller

components

  • Elastic S

tack Features (X-Pack)

  • APM (Application Performance

Monitoring)

Why Elastic Stack?

  • (Free) Open S
  • urce S
  • ftware
  • Distributed, real-time search and

analytics (very scalable)

  • Parsing and data enrichment
  • Large Community
  • InfoS

ec Proj ects built around it:

  • S

ecurity Onion

  • Moloch (Elasticsearch)
  • S

OF-ELK

  • S

ELKS

  • HELK
  • ROCK NS

M

8

slide-9
SLIDE 9

Elastic S tack History

Early 2000s: S hay Banon’s Recipe App 2012: Elasticsearch Inc. 2015: "Release Bonanza“ , Beats, Elastic Cloud (AWS ) 2016: Elastic S tack 5.0 2017: Elastic Cloud Enterprise (ECE) 2018: Open source X-Pack, New Y

  • rk S

tock Exchange 2019: Core security features (TLS , RBAC) are free, S IEM, EndGame

9

S

  • urce: https:/ / www.elastic.co/ about/ history-of-elasticsearch
slide-10
SLIDE 10

Elastic S tack (Very) High-Level Overview 10

  • Beats: single-purpose data shippers
  • Logstash: server-side data processing pipeline
  • Elasticsearch: distributed search and analytics engine
  • Kibana: visualization and dashboards

S ee also: https:/ / www.elastic.co/ assets/ blt2614227bb99b9878/ architecture-best-practices.pdf

slide-11
SLIDE 11

Beats

2019 Pass the S ALT Workshop

11

slide-12
SLIDE 12

Beats: Lightweight Data S hippers

  • Lightweight log agents
  • Written in Go
  • Can send to Logstash or directly to

Elasticsearch

  • Beats Family:
  • Filebeat
  • Winlogbeat
  • Auditbeat
  • Packetbeat
  • Heartbeat
  • Metricbeat
  • Functionbeat
  • Etc.

12

S ee also: https:/ / www.elastic.co/ guide/ en/ beats/ libbeat/ current/ index.html

slide-13
SLIDE 13

Beats Configuration Examples

Winlogbeat Filebeat

13

slide-14
SLIDE 14

Beats Hands-On

2019 Pass the S ALT Workshop

14

slide-15
SLIDE 15

Logstash

2019 Pass the S ALT Workshop

15

slide-16
SLIDE 16

Logstash Overview

  • LOTS AND LOTS of plugins!
  • Input: tcp, udp, syslog, beats, jdbc, kafka, rabbitmq, file, exec, cloudwatch, etc.
  • Filter: csv, json, xml, kv, grok, date, mutate, split, useragent, ruby, drop, etc.
  • Output: elasticsearch, graphite, nagios, kafka, rabbitmq, radis, file, email, irc, etc.
  • Easy to learn and use

Input Filter Output

16

S ee also: https:/ / www.elastic.co/ guide/ en/ logstash/ current/ index.html

slide-17
SLIDE 17

Input Plugin Examples

Plugin Description beats Events from Elastic Beats cloudwatch Events from AWS CloudWatch file S treams events from files jdbc Events from JDBC data kafka Reads events from Kafka rabbitmq Pulls events from RabbitMQ s3 Events from files in S 3 snmp Polls devices using S NMP syslog Reads syslog messages

17

i nput { s t di n { } } i nput { be a t s { por t => 5044 } } i nput { s ys l og { por t => 5514 } }

S ee also: https:/ / www.elastic.co/ guide/ en/ logstash/ current/ input-plugins.html

slide-18
SLIDE 18

Filter Plugin Examples

Plugin Description cidr Check IP against net blocks csv Parses CS V data into fields date Parses dates from fields dissect Extracts unstructured data drop Drops all events elasticsearch Gets data from Elasticsearch geoip Geo info about an IP grok Parses unstructured data json Parses JS ON data Plugin Description kv Parses key-value pairs mutate Performs mutations on fields ruby Executes Ruby code split S plits multi-line messages translate Replaces field contents truncate Truncates fields urldecode Decodes URL-encoded fields useragent Parses user agent strings xml Parses XML data

18

S ee also: https:/ / www.elastic.co/ guide/ en/ logstash/ current/ filter-plugins.html

slide-19
SLIDE 19

Filters - The Easy S tuff

JSON CSV

19

f i l t e r { . . . j s on { s our c e => " m e s s a ge “ } . . . m ut a t e { r e m

  • ve _f i e l d => [ " m

e s s a ge " ] } }

f i l t e r { . . . c s v { c ol um ns => [ " t s " , " ui d" , " i d. or i g_h" , " i d. or i g_p" , " i d. r e s p_h" , " i d. r e s p_p" , " pr ot o" , " s e r vi c e " , " dur a t i on" , " or i g_byt e s " , " r e s p_byt e s c onn_s t a t e " , " l oc a l _or i g" , " l oc a l _r e s p" , " m i s s e d_byt e s " , " hi s t or y" , " or i g_pkt s " , " or i g_i p_byt e s " , " r e s p_pkt s " , " r e s p_i p_byt e s " , " t unne l _pa r e nt s " ] s e pa r a t or => " “ } . . . m ut a t e { r e m

  • ve _f i e l d => [ " m

e s s a ge " ] } }

slide-20
SLIDE 20

Filters - RegExp vs. Grok, Dissect (1)

RegExp

  • (?

<![0-9])(? :(? :25[0-5]| 2[0- 4][0-9]| [0-1]? [0- 9]{1,2})[.](? :25[0-5]| 2[0-4][0- 9]| [0-1]? [0-9]{1,2})[.](? :25[0- 5]| 2[0-4][0-9]| [0-1]? [0- 9]{1,2})[.](? :25[0-5]| 2[0-4][0- 9]| [0-1]? [0-9]{1,2}))(? ![0-9]) Dissect

  • S

tring-based split operation

  • Very fast

Grok

  • %

{IPV4:source_ip}

  • Pre-cooked RegExp patterns
  • Custom Patterns:
  • (?

<queue_id>[0-9A-F]{10,11})

20

Grok Debuggers:

  • Heroku App: http:/ / grokdebug.herokuapp.com
  • S
  • urce: https:/ / github.com/ nickethier/ grokdebug
  • Docker: https:/ / hub.docker.com/ r/ fdrouet/ grokdebug
  • Kibana / Dev Tools / Grok Debugger
slide-21
SLIDE 21

Filters - RegExp vs. Grok, Dissect (2)

dissect grok

21

f i l t e r { . . . di s s e c t { m a ppi ng => { " m e s s a ge " => " % {t s } % {+t s } % {+t s } % {s r c } % {pr og}[ % {pi d}] : % {m s g}“ } } . . . } f i l t e r { . . . gr ok { m a t c h => { " m e s s a ge " => " % {SYSLOGTI M ESTAM P: s ys l og_t i m e s t a m p} % {SYSLOGHOST: s ys l og_hos t na m e } % {DATA: s ys l og_pr ogr a m }( ? : \ [ % {POSI NT: s ys l og_pi d}\ ] ) ? : % {GREEDYDATA: s ys l og_m e s s a ge }" #" m e s s a ge " => " % {SYSLOGBASE2} % {GREEDYDATA: m e s s a ge }“ } } . . . }

slide-22
SLIDE 22

Filters - Enrichment Examples

ruby geoip

22

f i l t e r { . . . i f [ pr ogr a m ] == " br o_dns " { r uby { c ode => " e ve nt . s e t ( ' que r y_l e ngt h' , e ve nt . ge t ( ' que r y' ) . l e ngt h) “ } } . . . } f i l t e r { . . . i f [ r e s p_h_r out a bl e ] == " t r ue " { ge oi p { s our c e => " i d. r e s p_h“ t a r ge t => " ge oi p“ de f a ul t _da t a ba s e _t ype => " Ci t y“ } ge oi p { s our c e => " i d. r e s p_h“ t a r ge t => " ge oi p“ de f a ul t _da t a ba s e _t ype => " ASN“ } } . . . }

slide-23
SLIDE 23

Output Plugin Examples

Plugin Description csv Writes events to disk in CS V elasticsearch S tores logs in Elasticsearch email S ends email to an address exec Runs a command file Writes events to files graphite Writes metrics to Graphite kafka Writes events to Kafka rabbitmq Pushes events to RabbitMQ redis S ends events to Redis

23

  • ut put {

e l a s t i c s e a r c h { hos t s => [ " l oc a l hos t : 9200" ] } }

  • ut put {

s t dout { c ode c => r ubyde bug } }

S ee also: https:/ / www.elastic.co/ guide/ en/ logstash/ current/ output-plugins.html

slide-24
SLIDE 24

Elastic Common S chema (ECS )

  • Specification that provides a

consistent and customizable way to structure your data in Elasticsearch

  • S

earches can be created more narrowly

  • Field names are easier to

remember

  • ECS Reference:

https:/ / www.elastic.co/ guide/ en / ecs/ current/ index.html

  • ECS GitHub:

https:/ / github.com/ elastic/ ecs

Level Description ECS Core Fields Fully defined set of field names that exists under a defined set of ECS top-level obj ects ECS Extended Fields Partially defined set of field names that exists under the same set of ECS top-level obj ects Custom Fields Undefined and unnamed set of fields that exists under a user- supplied set of non-ECS top-level

  • bj ects that must not conflict with

ECS fields or obj ects

24

slide-25
SLIDE 25

Logstash Hands-On

2019 Pass the S ALT Workshop

25

slide-26
SLIDE 26

Elasticsearch

2019 Pass the S ALT Workshop

26

slide-27
SLIDE 27

Elasticsearch Overview

  • S

torage and S earch

  • Built on Apache Lucene
  • “ wrapper” written in Java
  • RES

T API

  • JS

ON over HTTP

  • Distributed
  • Real-time

More info: John Hubbard - The Elastic Stack as a SIEM: https://www.youtube.com/watch?v=v69kyU5XMFI

27

slide-28
SLIDE 28

Elasticsearch Terms

  • Cluster: All nodes
  • Node: Elasticsearch

instance

  • Index: Set of documents

(group of shards)

  • Shard:
  • S

ubset of documents in an index

  • Apache Lucene instance
  • Primary (like RAID 0) and

Replica (like RAID 1)

  • Document: JSON obj ect in

Elasticsearch

Node1

Index1 Index2 Index3

Node2

Index

Node3

Index

28

S hard S hard S hard

S hard

Documents

Cluster

slide-29
SLIDE 29

Elasticsearch vs. Relational Database

  • Mapping:
  • Defines field names and datatypes

in documents

  • Can add new fields, but existing

fields cannot be changed!

  • Field:
  • Key-value pair in a document
  • Metadata like: _index, _id, etc.
  • WORM (Write Once Read Many)
  • vs. ACID (Atomicity, Consistency,

Isolation, Durability)

Elasticsearch Relational Database Index Database Mapping S chema Document Row Field Column

29

slide-30
SLIDE 30

Data Types (Few Examples)

Core

  • text
  • keyword
  • long, integer, short, byte
  • double, float, half_float,

scaled_float

  • boolean
  • binary

Geo

  • geo_point
  • geo_shape

Specialized

  • date
  • ip

Complex

  • array
  • obj ect
  • nested

Multi-fields

  • Indexed as more one type

Etc.

30

S ee also: https:/ / www.elastic.co/ guide/ en/ elasticsearch/ reference/ current/ mapping-types.html

slide-31
SLIDE 31

Text vs. Keyword

Text type

  • “ Full-text value”
  • Payload, message, etc.
  • Analyzed and tokenized
  • Cannot be used for
  • S
  • rting
  • Aggregations

Keyword type

  • “ Exact value”
  • IP

, port, protocol, user, etc.

  • Exact match / not match
  • Can be used for
  • S
  • rting
  • Aggregations

31

slide-32
SLIDE 32

Elasticsearch Hands-On

2019 Pass the S ALT Workshop

32

slide-33
SLIDE 33

Kibana

2019 Pass the S ALT Workshop

33

slide-34
SLIDE 34

Kibana Overview

  • Web-based analytic interface
  • S

earches

  • Apache Lucene syntax
  • Filters
  • Visualizations, Dashboards
  • S

tored in JS ON

  • Plugins
  • Reporting, Alerting, etc.

34

slide-35
SLIDE 35

Kibana Features

  • Discover: S

earch

  • Visualize: Graphs, charts
  • Vega, Vega-Lite
  • Dashboard: Visualizations and saved

searches

  • Timelion: Time series visualizations
  • Canvas: Presentation
  • Machine Learning (Paid)
  • Graph (Paid)
  • Infrastructure: Metricbeats

monitoring

  • Logs: Filebeat monitoring
  • APM: Application Performance

Monitoring

  • Uptime: Monitor the status of

network endpoints

  • SIEM: Interactive workspace for

security investigations

  • Dev Tools: API access
  • Monitoring: Cluster health
  • Management: Cluster management
  • etc.

35

slide-36
SLIDE 36

Index Patterns

  • Must choose an index pattern
  • Discovery (S

earches)

  • Visualization
  • Limits the indices searched
  • Relates to index naming

scheme

  • Can use the * wildcard
  • “ logstash-* ”

S teps:

  • 1. Create Elasticsearch index
  • 2. “ Create index pattern”
  • 3. S

elect index/ indices

  • 4. Define @

timestamp field

36

slide-37
SLIDE 37

S earch –Apache Lucene Query S yntax (1)

Search Type Syntax Example Single Term <term> hello Phrase “ <term>” “ hello world” Fields <field>:<term> title:hello AND <term-a> AND <term-b> hello AND world / / hello world OR <term-a> OR <term-b> hello OR world NOT NOT <term-a> !<term-a> NOT “ hello world” !“ hello world” Must match +<term> +hell!o Must not match

  • <term>
  • hello

37

slide-38
SLIDE 38

S earch –Apache Lucene Query S yntax (2)

Search Type Syntax Example Field exists _exists_:<field> _exists_:title Field does not exists NOT _exists_:<field> ! _exists_:<field>

  • _exists_:<field>

NOT _exists_:title ! _exists_:title

  • _exists_:title

Wildcard search ? , * h? llo, hell* Fuzzy search <term>~[<number>] hello~2 Proximity search “ <term>” ~[<number>] “ hello world” ~5 Range <field>:[<value-a> TO <value-b>] <field>:{<value-a> TO <value-b>} port:[1 TO 1024] title:{hello TO world}

38

slide-39
SLIDE 39

S earch vs. Filters And Time Range

  • Search: Using the Query bar and the Apache Lucene Query S

yntax

  • Filter: Using the Filters Box and the Elasticsearch Query DS

L (Domain S pecific Language)

39

slide-40
SLIDE 40

Visualizations

Visualization Type Area Basic Charts Heat Map Basic Charts Horizontal Bar Basic Charts Line Basic Charts Pie Basic Charts Vertical Bar Basic Charts Data Table Data Gauge Data Goal Data Visualization Type Metric Data Coordinate Map Maps Region Map Maps Timelion Time S eries Visual Builder (E) Time S eries Controls (E) Other Markdown Other Tag Cloud Other Vega (E) Other

40

slide-41
SLIDE 41

Visualizations use Elasticsearch Aggregations

Metrics: value to calculate

  • Count
  • Average
  • Sum
  • Min
  • Max
  • Unique Count
  • Standard

Deviation

  • Top Hit
  • Percentiles
  • etc.

Bucket: aggregation (grouping)

  • Date

Histogram (by time)

  • Date Range
  • Filter
  • Geo Distance
  • IP Range
  • Range
  • S

ampler

  • S

ignificant Text

  • Terms (by

field)

  • etc.

41

slide-42
SLIDE 42

Vega and VegaLite

  • Vega Graphs
  • Visualization grammar
  • Declarative language
  • JS

ON format

  • S

upported from Elastic 6

  • Vega vs VegaLite
  • VegaLite: simplified Vega
  • https:/ / vega.github.io/ vega/
  • https:/ / vega.github.io/ vega-lite/

42

Based on: https:/ / www.elastic.co/ blog/ sankey- visualization-with-vega-in-kibana

slide-43
SLIDE 43

Canvas 43

slide-44
SLIDE 44

Elastic S IEM 44

S

  • urce: https:/ / www.elastic.co/ blog/ introducing-elastic-siem
slide-45
SLIDE 45

Kibana Hands-On S cenario 45

slide-46
SLIDE 46

Kibana Hands-On

2019 Pass the S ALT Workshop

46

slide-47
SLIDE 47

Elastic S tack Alerting and S ecurity

2019 Pass the S ALT Workshop

47

slide-48
SLIDE 48

Watcher vs. ElastAlert

Watcher

  • Part of X-Pack
  • https:/ / www.elastic.co/ guide/ en

/ x-pack/ current/ xpack- alerting.html

  • Elasticsearch API
  • JSON format
  • Watches: Triggers, Inputs,

Conditions, Transforms, Actions

ElastAlert

  • Developed by Y

elp

  • https:/ / github.com/ Y

elp/ elast alert

  • S

imple framework for alerting

  • Y

AML format

  • Components: Rules and Alerts

48

slide-49
SLIDE 49

ElastAlert Overview

  • 1. Elasticsearch is periodically

queried

  • 2. Data is passed to the rules
  • 3. When a match occurs, one or

more alerts are triggered

  • 4. Alerts take action based on

the match

  • Rule types: Any, Blacklist,

Whitelist, Change, Frequency, S pike, Flatline, New Term, Cardinality, Metric Aggregation, Percentage Match

  • Alert types: Command, Email,

JIRA, S erviceNow, S lack, PagerDuty, GoogleChat, Mattermost, Telegram, etc.

  • https:/ / elastalert.readthedocs.io

49

slide-50
SLIDE 50

ElastAlert Examples 50

slide-51
SLIDE 51

ElastAlert Hands-On

2019 Pass the S ALT Workshop

51

slide-52
SLIDE 52

S ecurity 52

  • Elastic Stack Security: https:/ / www.elastic.co/ products/ stack/ security
  • Part of Elastic Stack Features (formerly X-Pack)
  • “ S

tarting in version 6.8 and 7.1, core security features like TLS , file and native realm authentication, and role-based access control are now free.”

  • ReadonlyREST: https:/ / readonlyrest.com
  • 3rd party
  • Free community version
  • Search Guard: https:/ / search-guard.com
  • 3rd party
  • Free community version
  • NGINX reverse proxy + Basic Auth: https:/ / www.nginx.com
  • No RBAC at all
slide-53
SLIDE 53

Questions and Answers

2019 Pass the S ALT Workshop

53

slide-54
SLIDE 54

References

  • Elastic Website
  • https:/ / www.elastic.co
  • Elastic Documentation
  • https:/ / www.elastic.co/ guide/ index.html
  • John Hubbard - The Elastic S

tack as a S IEM

  • https:/ / www.youtube.com/ watch?

v=v69kyU5XMFI

  • ElastAlert
  • https:/ / github.com/ Y

elp/ elastalert

54