Behavioral Analysis Using Network traffic, DNS and logs JOSH PYORRE - - PowerPoint PPT Presentation

▶
behavioral analysis
SMART_READER_LITE
LIVE PREVIEW

Behavioral Analysis Using Network traffic, DNS and logs JOSH PYORRE - - PowerPoint PPT Presentation

Feb 06, 2023 175 likes •1.46k views

Behavioral Analysis Using Network traffic, DNS and logs JOSH PYORRE Security Researcher Previously: Threat Analyst at NASA Threat Analyst at Mandiant @joshpyorre rootaccesspodcast.com Behavioral Analysis VIDEO analyzing website visitors

slide-1
SLIDE 1

Behavioral Analysis

Using Network traffic, DNS and logs

slide-2
SLIDE 2

Previously: Threat Analyst at NASA

JOSH PYORRE

Security Researcher @joshpyorre Threat Analyst at Mandiant rootaccesspodcast.com

slide-3
SLIDE 3

Behavioral Analysis

VIDEO

analyzing website visitors

slide-4
SLIDE 4

Behavioral Analysis

slide-5
SLIDE 5

We’re Working With Data

slide-6
SLIDE 6

Current detection methods

slide-7
SLIDE 7

IDS

  • Based on signatures
slide-8
SLIDE 8

IDS

#alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"ET WEB_SERVER PHP Generic Remote File Include Attempt (HTTP)"; flow:to_server,established; content:".php"; nocase; http_uri; content:"=http|3a|/"; nocase; http_uri; pcre:"/ \x2Ephp\x3F.{0,300}\x3Dhttp\x3A\x2F[^\x3F\x26]+\x3F/ Ui"; reference:url,doc.emergingthreats.net/2009151; classtype:web-application-attack; sid:2009151; rev:7;)

slide-9
SLIDE 9

IDS

  • Based on signatures
  • Not adaptable / requires human intervention
  • Catches known threats / some unknown
  • if you do your reg-ex right
slide-10
SLIDE 10

Antivirus

  • Host based file detection
  • Based on signatures of known threats
slide-11
SLIDE 11

What We’re Working With

slide-12
SLIDE 12

CSV files

slide-13
SLIDE 13

PCAP’s

slide-14
SLIDE 14

AD logs

slide-15
SLIDE 15

DNS logs

slide-16
SLIDE 16

VIDEO

Graphing with Excel

slide-17
SLIDE 17

VIDEO

Graphing with Excel

slide-18
SLIDE 18

We Want to Find Normal

  • What is normal?
  • How do you find it?
slide-19
SLIDE 19

VIDEO

DNS Tunneling Domains

slide-20
SLIDE 20

VIDEO

  • ne unusual domain
slide-21
SLIDE 21
slide-22
SLIDE 22

How do you find that in this?

VIDEO

Streaming data

slide-23
SLIDE 23

Or this?

slide-24
SLIDE 24

Looking at just the domains

slide-25
SLIDE 25

Most visited domains in this list

slide-26
SLIDE 26

Just focusing on Facebook domains

slide-27
SLIDE 27

Facebook subdomains graphed

slide-28
SLIDE 28

Subdomains Removed

slide-29
SLIDE 29

Remove “Normal”

slide-30
SLIDE 30

Remove most visited

slide-31
SLIDE 31

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

Looking at some (a manual process)

slide-32
SLIDE 32

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

Take out obvious known good

slide-33
SLIDE 33

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

What about these?

slide-34
SLIDE 34

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

We can tell they were fine

slide-35
SLIDE 35

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

Look at each one

slide-36
SLIDE 36

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

This one is ok…

slide-37
SLIDE 37

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

Suspicious…

slide-38
SLIDE 38

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

slide-39
SLIDE 39

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

This one is ok…

slide-40
SLIDE 40

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

Suspicious…

slide-41
SLIDE 41

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

Suspicious…

slide-42
SLIDE 42

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

Suspicious…

slide-43
SLIDE 43

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

Suspicious…

slide-44
SLIDE 44

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

slide-45
SLIDE 45

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

slide-46
SLIDE 46

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

This one is actually ok…

slide-47
SLIDE 47

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

slide-48
SLIDE 48

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

This one is ok too…

slide-49
SLIDE 49

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

slide-50
SLIDE 50

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

Also ok…

slide-51
SLIDE 51

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

slide-52
SLIDE 52

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

slide-53
SLIDE 53

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

This one is ok…

slide-54
SLIDE 54

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

slide-55
SLIDE 55

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

slide-56
SLIDE 56

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

slide-57
SLIDE 57

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

slide-58
SLIDE 58

masterbrand.com mathads.com mdswanson.com medicaresupplement.com metanetwork.net mfg.com mhthemes.com military.com ministerial5.com mixpanel.com mobileapptracking.com mom.me monarchads.com mrsteam.com msftncsi.com mshcdn.com myfonts.net myvzw.com newsinc.com nexage.com nextadvisor.com norcraftcompanies.com notanpest.net obtrk.xyz office.net

Found a bad one!

slide-59
SLIDE 59

That process is really tedious

slide-60
SLIDE 60

Tedious

  • Mostly Manual Process
  • Searching and Personal Expertise
slide-61
SLIDE 61

Automate Things

  • Auto-process Logs / Streams
  • Save it so it’s Workable
slide-62
SLIDE 62

Getting data

  • Log files
  • DNS, AD, System, Web, etc
  • Network Capture
slide-63
SLIDE 63

Have to Write Parsers

  • We all have custom data
  • Use whatever is easiest for you
  • Python
  • Excel
  • R

U g h

slide-64
SLIDE 64

Creating a baseline

  • Counting Total Number of Domains
  • Getting “Scores”
  • Categorization
slide-65
SLIDE 65

DNS

  • Starting with data from OpenDNS
  • Over a billion queries per day - approximately

3% of the internet

  • How do you baseline that? (You need a LOT of

that data - as in TB)

  • Can you baseline that?
slide-66
SLIDE 66

DNS looks like this

slide-67
SLIDE 67

Let’s try anyway

  • Count how many times a domain is seen
  • Over 10 times: Write to normal_traffic.txt
  • Under 3 times: Write to suspicious_traffic.txt
slide-68
SLIDE 68

08/25/2016 17:20:00,s1.mohito.com,A,360,78.24.161.76 08/25/2016 17:29:00,hotspotcostablanca.ath.cx,A,60,81.36.140.179

9 Minutes of DNS

slide-69
SLIDE 69

wc -l 2016-08-25-17-20.myzPMsaJ 664,938 2016-08-25-17-20.myzPMsaJ

664,938 domains looked up

slide-70
SLIDE 70

File size of original logs, the normal_traffic and the suspicious_traffic output files

slide-71
SLIDE 71

Normal Traffic

slide-72
SLIDE 72

Suspicious Traffic

slide-73
SLIDE 73

Suspicious Traffic

Picking one ‘suspicious’ site

slide-74
SLIDE 74

Looking at DNS requests for that ‘suspicious’ site

slide-75
SLIDE 75

It’s probably fine…

slide-76
SLIDE 76

More Suspicious Traffic

Picking another ‘suspicious’ site

slide-77
SLIDE 77

Looking at DNS requests:

slide-78
SLIDE 78

Looking at the site (no longer available)

slide-79
SLIDE 79

Categorization

Looking at categorization:

slide-80
SLIDE 80

Categorization

We still have a lot that is uncategorized

slide-81
SLIDE 81

Look at the Bandwidth

slide-82
SLIDE 82

Total traffic by Minute

slide-83
SLIDE 83

Doesn’t mean much

It’s random data from unrelated sources

slide-84
SLIDE 84

We must Narrow the Focus

slide-85
SLIDE 85

DNS One Organization

slide-86
SLIDE 86

19,801,469 DNS Requests: For a 17 hour period:

slide-87
SLIDE 87

Looks like this. Lots of visits to directv.com

slide-88
SLIDE 88

Remove unneeded data, still a large file:

slide-89
SLIDE 89

Get rid of the obviously normal: Down to 493,351 DNS Requests:

slide-90
SLIDE 90

Get rid of additional normal domains Down to 1,320 DNS Requests:

slide-91
SLIDE 91

Graphing on a timeline, but it’s not too clean

slide-92
SLIDE 92

Graph by domain and visit count (with some mistakes)

slide-93
SLIDE 93

2 GB File: Taking forever

Auto processing through past data can take a toll

slide-94
SLIDE 94

DNS

  • Looking at DNS traffic on the wire
  • Processing with various tools
  • Python
  • Pandas, etc…
slide-95
SLIDE 95

DNS

  • Ran this on a system at home:

tcpdump -i eth1 -j host port 53 -tttt >> tcpdump.log eth1 is hooked up to a network tap watching traffic between the routers internet port and the cable modem

slide-96
SLIDE 96

It looks like this:

12:56:37.854306 IP 73.202.157.15.53018 > 208.67.222.222.53: 46044+ A? apple.com. (27) E..7..@.@...I....C..... 5.#.w.............apple.com..... 12:56:37.854517 IP 73.202.157.15.2461 > 208.67.222.222.53: 18586+ A? calendar.google.com. (37) E..A..@.@...I....C.. .. 5.-.xH............calendar.google.com..... 12:56:37.854681 IP 73.202.157.15.25959 > 208.67.222.222.53: 17850+ A? 1-courier.push.apple.com. (42) E..F..@.@...I....C..eg.5.2V.E........... 1- courier.push.apple.com..... 12:56:37.854906 IP 73.202.157.15.15125 > 208.67.222.222.53: 63415+ A? 14-lvl3-pdl.vimeocdn.com. (42)

slide-97
SLIDE 97

Just the time, domain and visit count

slide-98
SLIDE 98

Graph by domain and visit count

slide-99
SLIDE 99

This is anomalous:

slide-100
SLIDE 100

Compare Traffic

See if we can get an idea of network behavior from patterns

slide-101
SLIDE 101

0 hr 24 hr

Thursday

slide-102
SLIDE 102

Thursday

At a Die Antwoord concert

slide-103
SLIDE 103

Friday

At work Home

slide-104
SLIDE 104

Saturday

slide-105
SLIDE 105

Sunday

slide-106
SLIDE 106

Monday

slide-107
SLIDE 107

Tuesday

slide-108
SLIDE 108

Wednesday

slide-109
SLIDE 109

Thursday

Ignore the color change…

slide-110
SLIDE 110

Friday

slide-111
SLIDE 111

Saturday

not a lot of traffic…

slide-112
SLIDE 112

Saturday

Went out of town

slide-113
SLIDE 113

Sunday

Still out of town

slide-114
SLIDE 114

Sunday

Still out of town …and I’m home

slide-115
SLIDE 115

Monday

slide-116
SLIDE 116

Monday

slide-117
SLIDE 117

Tuesday

At work

slide-118
SLIDE 118

Tuesday

At work At a Peaches Concert

slide-119
SLIDE 119

Tuesday

At work At a Peaches Concert

Not ready for bed (amped up from concert, most likely)

slide-120
SLIDE 120

Creating a dashboard

  • Scripts auto-push streaming data into DB
  • MongoDB holds the data
  • Scripts autoprocess from MongoDB
  • Flask serves graphs and statistics
slide-121
SLIDE 121

DNS Traffic (demo)

slide-122
SLIDE 122

Network Traffic/PCAP

  • Live capture or saved PCAP
  • Using DPKT python packet library
  • Select all the HTTP requests
  • Generate changing stats and information
slide-123
SLIDE 123

Network traffic (demo)

slide-124
SLIDE 124

AD Logs

  • Successful vs Failed logins
slide-125
SLIDE 125

AD Logs (demo)

slide-126
SLIDE 126

In Progress

  • Machine Learning to process data (to replace me)
  • ASN categorization
  • Dashboard with live/streaming data
  • Looking at authlogs
  • Looking at Honeypot data
  • Using opensource categorization
slide-127
SLIDE 127

jpyorre@gmail.com @cisco.com @opendns.com https://jpyorre.com @joshpyorre rootaccesspodcast.com