What Is This #4 Introduction About Flow Unix Beginning Analysis Basic SiLK Tools sIP| dIP|sPort|dPort|pkts|flags| sTime| rwfilter 72.24.129.20|82.80.30.150| 80| 1220| 152| S PA|00:00:23.602| Printing and Sorting Tools 82.80.30.150|72.24.129.20| 1220| 80| 90| SRPA|00:00:23.602| Counting Tools 72.24.129.20|82.80.30.150| 80| 1221|1126| S PA|00:00:23.710| Other Tools 82.80.30.150|72.24.129.20| 1221| 80| 413| SRPA|00:00:23.710| Advanced 72.24.129.20|82.80.30.150| 80| 1223| 63| S PA|00:00:26.341| Sets Bags 82.80.30.150|72.24.129.20| 1223| 80| 39| S PA|00:00:26.341| Prefix Maps 72.24.129.20|82.80.30.150| 80| 1224| 8| S PA|00:00:26.883| Unix Scripting 82.80.30.150|72.24.129.20| 1224| 80| 7| SRPA|00:00:26.883| 82.80.30.150|72.24.129.20| 1223| 80| 1| R A|00:01:33.068| Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 15
It’s all a matter of timing Introduction About Flow The flow buffer has to be kept manageable Unix Beginning Analysis Inactivity timeout: Basic SiLK Tools ◮ If there’s no activity within [30] seconds, flush the rwfilter Printing and Sorting flow Tools Counting Tools Other Tools Active timeout: Advanced ◮ Flush all flows open for [30] minutes Sets Bags Prefix Maps Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 16
What Is This #5 Introduction About sIP| dIP|sPort|dPort|pro|Pkt|byte| sTime| dur| Flow Unix 8.97.138.194|72.24.145.68| 500| 500| 17| 1| 112|00:02:31| 0.000| Beginning Analysis 72.24.145.68|8.97.138.194| 500| 500| 17| 1| 112|00:02:31| 0.000| Basic SiLK Tools 8.97.138.194|72.24.145.68| 500| 500| 17| 2| 224|00:13:53|40.498| rwfilter 72.24.145.68|8.97.138.194| 500| 500| 17| 2| 224|00:13:53|40.498| Printing and Sorting Tools 8.97.138.194|72.24.145.68| 500| 500| 17| 2| 224|00:25:10|45.582| Counting Tools 72.24.145.68|8.97.138.194| 500| 500| 17| 2| 224|00:25:10|45.582| Other Tools 8.97.138.194|72.24.145.68| 500| 500| 17| 1| 112|00:36:03| 0.000| Advanced 72.24.145.68|8.97.138.194| 500| 500| 17| 1| 112|00:36:03| 0.000| Sets Bags 8.97.138.194|72.24.145.68| 500| 500| 17| 1| 112|00:43:19| 0.000| Prefix Maps 72.24.145.68|8.97.138.194| 500| 500| 17| 1| 112|00:43:19| 0.000| Unix Scripting 8.97.138.194|72.24.145.68| 500| 500| 17| 3| 336|00:47:30|46.088| Visualization 72.24.145.68|8.97.138.194| 500| 500| 17| 3| 336|00:47:30|46.088| Basic Graphs 72.24.145.68|8.97.138.194| 500| 500| 17| 1| 112|00:53:32| 0.000| Excel 8.97.138.194|72.24.145.68| 500| 500| 17| 1| 112|00:53:32| 0.000| Gnuplot Advanced Graphs 72.24.145.68|8.97.138.194| 500| 500| 17| 2| 208|00:58:42| 0.000| 8.97.138.194|72.24.145.68| 500| 500| 17| 20|2232|00:58:49|90.095| Conclusion The Community Page 17
What Is This #6 Introduction About sIP| dIP|sPort|dPort| pkts|flg| sTime| dur| Flow 72.24.147.6|58.210.70.72|35282| 22|29640|PA |00:00:11.361|1800.63| Unix Beginning Analysis 58.210.70.72| 72.24.147.6| 22|35282|29633|PA |00:00:11.911|1800.08| Basic SiLK Tools 72.24.147.6|58.210.70.72|35282| 22|30824|PA |00:26:23.092|1800.82| rwfilter 58.210.70.72| 72.24.147.6| 22|35282|30825|PA |00:26:23.092|1800.82| Printing and Sorting 72.24.147.6|58.210.70.72|35282| 22|29346|PA |00:56:24.020|1800.90| Tools Counting Tools 58.210.70.72| 72.24.147.6| 22|35282|29347|PA |00:56:24.020|1800.90| Other Tools 72.24.147.6|58.210.70.72|35282| 22|31107|PA |01:00:10.783|1800.20| Advanced 58.210.70.72| 72.24.147.6| 22|35282|31113|PA |01:00:11.301|1800.68| Sets 72.24.147.6|58.210.70.72|35282| 22|29227|PA |01:26:25.036|1800.95| Bags Prefix Maps 58.210.70.72| 72.24.147.6| 22|35282|29228|PA |01:26:25.036|1800.95| Unix Scripting 72.24.147.6|58.210.70.72|35282| 22|30880|PA |01:56:26.096|1800.82| 58.210.70.72| 72.24.147.6| 22|35282|30878|PA |01:56:26.096|1800.82| Visualization 72.24.147.6|58.210.70.72|35282| 22|30302|PA |02:00:11.301|1800.65| Basic Graphs Excel 58.210.70.72| 72.24.147.6| 22|35282|30287|PA |02:00:11.843|1800.10| Gnuplot 72.24.147.6|58.210.70.72|35282| 22|31998|PA |02:26:27.028|1800.90| Advanced Graphs 58.210.70.72| 72.24.147.6| 22|35282|31999|PA |02:26:27.028|1800.90| Conclusion 72.24.147.6|58.210.70.72|35282| 22|32764|PA |02:56:28.040|1800.88| The Community Page 18
What Is This #7 sIP| dIP|sPort|dPort|pkt|flags| sTime| dur| Introduction 72.24.144.17|10.25.235.38|40395| 80| 45| S PA|1:59:34.81|1759.18| About 10.25.235.38|72.24.144.17| 80|40395| 44| S PA|1:59:34.81|1759.07| Flow Unix 10.25.235.38|72.24.144.17| 80|40395| 40| PA|2:29:39.82|1797.62| Beginning Analysis 72.24.144.17|10.25.235.38|40395| 80| 40| A|2:29:39.93|1797.51| Basic SiLK Tools 10.25.235.38|72.24.144.17| 80|40395| 40| PA|3:00:23.46|1800.17| rwfilter 72.24.144.17|10.25.235.38|40395| 80| 40| A|3:00:23.57|1800.17| Printing and Sorting Tools 10.25.235.38|72.24.144.17| 80|40395| 40| PA|3:31:09.83|1797.52| Counting Tools 72.24.144.17|10.25.235.38|40395| 80| 40| A|3:31:09.93|1797.52| Other Tools 10.25.235.38|72.24.144.17| 80|40395| 40| PA|4:01:53.42|1797.72| Advanced 72.24.144.17|10.25.235.38|40395| 80| 40| A|4:01:53.51|1797.64| Sets Bags 10.25.235.38|72.24.144.17| 80|40395| 35| RPA|4:32:37.18|1560.50| Prefix Maps 72.24.144.17|10.25.235.38|40395| 80| 34| A|4:32:37.29|1520.89| Unix Scripting 72.24.144.17|37.52.53.241|40395| 80| 13|FS PA|5:18:41.57| 0.48| 37.52.53.241|72.24.144.17| 80|40395| 18|FS PA|5:18:41.63| 0.43| Visualization Basic Graphs 72.24.144.17|42.15.190.19|40395| 80| 9|FS PA|8:21:01.15| 4.14| Excel 42.15.190.19|72.24.144.17| 80|40395| 6|FS PA|8:21:01.15| 4.14| Gnuplot 42.15.190.19|72.24.144.17| 80|40395| 1| A|8:21:05.29| 0.00| Advanced Graphs 72.24.144.17|10.46.227.72|40395| 80| 7|FS PA|9:21:24.36| 0.22| Conclusion 10.46.227.72|72.24.144.17| 80|40395| 6|FS PA|9:21:24.47| 0.22| The Community 72.24.144.17|18.113.57.14|40395| 80| 6|FS PA|9:39:43.67| 0.11| 18.113.57.14|72.24.144.17| 80|40395| 4|FS PA|9:39:43.67| 0.21| Page 19
Where do I collect flows? Flow is often collected at the Introduction About border Flow Unix ◮ Watch internal and Beginning Analysis Basic SiLK Tools external rwfilter communications Printing and Sorting Tools Counting Tools ◮ Identify services on your Other Tools Advanced network Sets Bags ◮ Identify resources your Prefix Maps machines use regularly Unix Scripting Visualization Most routers can generate Basic Graphs Excel flows Gnuplot Advanced Graphs Conclusion The Community Page 20
Flow vs. IDS Introduction About Flow Unix IDS Beginning Analysis Basic SiLK Tools + Content inspection rwfilter Printing and Sorting - Presents an interpretation of raw data Tools Counting Tools Other Tools - Tuning means discarding false positive data Advanced Flow Sets Bags Prefix Maps - No content available Unix Scripting + Gives direct observations Visualization Basic Graphs + No tuning, keep everything Excel Gnuplot Advanced Graphs Conclusion The Community Page 21
Flow vs. Firewall Introduction About Flow Unix Firewalls Beginning Analysis Basic SiLK Tools + Block unwanted traffic rwfilter Printing and Sorting Tools - Not intended as a historial record; logging is Counting Tools Other Tools secondary Advanced Flow Sets Bags Prefix Maps - Completely passive Unix Scripting + Logging is primary Visualization Basic Graphs + Audits the firewall Excel Gnuplot Advanced Graphs Conclusion The Community Page 22
Got a question? Flow can help. Introduction About Flow ◮ What’s on my network? Unix Beginning Analysis ◮ What happened before the event? Basic SiLK Tools rwfilter ◮ Where are policy violations occurring? Printing and Sorting Tools Counting Tools ◮ What are the most popular web sites? Other Tools Advanced ◮ How much volume would be reduced with a blacklist? Sets Bags ◮ Do my users browse to known infected web servers? Prefix Maps Unix Scripting ◮ Do I have a spammer on my network? Visualization ◮ When did my web server stop responding to queries? Basic Graphs Excel Gnuplot ◮ Who uses my public DNS server? Advanced Graphs Conclusion The Community Page 23
About ssh Introduction About ◮ ssh creates a secured connection between your Flow Unix computer and the ssh server Beginning Analysis Basic SiLK Tools ◮ ssh is your primary tool for moving things between rwfilter Printing and Sorting you and your analysis server Tools Counting Tools Other Tools Advanced Sets Bags Prefix Maps Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 24
Try It #1! Introduction About Flow Unix Beginning Analysis Basic SiLK Tools rwfilter Login Printing and Sorting Tools Check for access to data ( ls /data ) Counting Tools Other Tools Type “ which rwfilter ” Advanced Sets Type “ rwfilter --help | more ” Bags Logout (optional) Prefix Maps Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 25
Try It #2! Introduction About Flow Unix Beginning Analysis Basic SiLK Tools rwfilter Move a file from the server to your workstation: Printing and Sorting Tools scp server:/remote/path/to/file.ext Counting Tools Other Tools /local/directory/ Advanced Sets Move a file from your workstation to the server: Bags Prefix Maps scp /local/file.ext server:/remote/directory/ Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 26
Text Editors Introduction About Flow Unix Beginning Analysis Write only: echo "blah" > file Basic SiLK Tools Simple: vi rwfilter Printing and Sorting Flexible but not always available: emacs Tools Counting Tools Other simple text file tools Other Tools Advanced ◮ cat : print it out Sets Bags ◮ more , less : print it out one page at a time Prefix Maps Unix Scripting ◮ head , tail : print out just the beginning (or end) Visualization ◮ wc -l : count the number of lines Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 27
Getting around Introduction About Flow Unix Beginning Analysis Some other commands you may need: Basic SiLK Tools rwfilter ◮ cd : change directory Printing and Sorting Tools ◮ ls : list the current directory contents Counting Tools Other Tools ◮ mkdir : make a directory Advanced Sets ◮ rm : remove a file Bags Prefix Maps ◮ cp : copy a file Unix Scripting Visualization ◮ logout or exit : log out Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 28
Try It #3! Introduction About Flow Unix Beginning Analysis Basic SiLK Tools 1. Create a sample file on the server. rwfilter Printing and Sorting Tools 2. Move the file from the server to your local machine, Counting Tools Other Tools and open it in the local text editor. Change the file Advanced and move it back to the server. Sets Bags Prefix Maps 3. Use head and tail to display the second line of a file Unix Scripting which contains 5 lines. Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 29
Where’s the GUI command prompt? Introduction About Flow Unix Beginning Analysis Basic SiLK Tools ◮ It’s not quite the same as Windows: rwfilter Printing and Sorting Tools ◮ You’ll always be working from a command prompt Counting Tools Other Tools ◮ We’ll be doing lots of text manipulation Advanced Sets ◮ There’s occasional CR-LF messiness Bags Prefix Maps ◮ Data can get big, but that’s usually OK Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 30
Get used to using pipes Introduction About Flow Unix Beginning Analysis Basic SiLK Tools ◮ Pass output from one command as input to another rwfilter Printing and Sorting ◮ Stop things with ctrl+c Tools Counting Tools Other Tools ◮ Also watch out for ctrl+s (suspend), restart with Advanced ctrl+q Sets Bags ◮ Also watch out for ctrl+z (put in background), Prefix Maps Unix Scripting continue with fg Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 31
About SiLK Introduction About Flow Unix Beginning Analysis Basic SiLK Tools rwfilter ◮ The System for internet Level Knowledge Printing and Sorting Tools Counting Tools ◮ http://tools.netsa.cert.org Other Tools ◮ Packing System Advanced Sets ◮ Accepts Netflow Bags Prefix Maps ◮ Stores data in a very space-efficient binary flat file Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 32
Analysis Suite Introduction About Flow Unix Beginning Analysis ◮ Used to query binary flat files from the packing Basic SiLK Tools rwfilter system Printing and Sorting Tools Counting Tools ◮ Some mirror Unix text tools for operate on binary Other Tools flow files e.g., cut, uniq, sort, split Advanced Sets ◮ Some work with large IP data collections sets, bags Bags Prefix Maps and prefix maps Unix Scripting ◮ All support ad-hoc analysis needs Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 33
What SiLK Does Introduction About ◮ Optimized for extremely large data collections Flow Unix ◮ Very compact record format Beginning Analysis Basic SiLK Tools ◮ Large amount of history can stay on line rwfilter Printing and Sorting ◮ Command line interface Tools Counting Tools Other Tools ◮ Keep data in the native binary format as long as Advanced possible Sets Bags ◮ Retrospective analysis Prefix Maps Unix Scripting ◮ Most useful for analyzing past network events Visualization Basic Graphs ◮ May feed an automated report generator Excel Gnuplot ◮ Good for forensics (what happened before the Advanced Graphs Conclusion incident?) The Community Page 34
Flavoring your Flows Introduction About Flow Unix Beginning Analysis Without content data, flows often seem very bland Basic SiLK Tools rwfilter SiLK flavors flow data with add-ons: Printing and Sorting Tools ◮ Address sets e.g., blacklists Counting Tools Other Tools ◮ Address Bags give a value to an address Advanced Sets ◮ Prefix Maps give an arbitrary label to a group of Bags Prefix Maps addresses e.g., Country Code Mapping Unix Scripting ◮ Hooks for custom libraries Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 35
About Classes and Types Introduction SiLK assigns each flow record a CLASS and a TYPE About Flow Class Unix Beginning Analysis ◮ Duplicates the purpose of the router Basic SiLK Tools rwfilter ◮ Sample classes might be Border, Internal, Customer Printing and Sorting Tools Counting Tools ◮ We will simply use “All” Other Tools Advanced Type Sets Bags ◮ Separate inbound from outbound Prefix Maps Unix Scripting ◮ Queries often run against a single type to improve Visualization performance Basic Graphs Excel ◮ Other types are common also Gnuplot Advanced Graphs ◮ in, inweb, out, outweb, null Conclusion The Community Page 36
The Flow Repository The Repository: a directory structure holding binary flow Introduction About files Flow Unix Directory structure based on: Beginning Analysis Basic SiLK Tools ◮ Sensor rwfilter Printing and Sorting ◮ Type Tools Counting Tools ◮ Year Other Tools Advanced ◮ Month Sets Bags ◮ Day Prefix Maps Unix Scripting File name based on: Visualization Basic Graphs ◮ Type Excel Gnuplot ◮ Sensor Advanced Graphs Conclusion ◮ YYYYMMDD.HH The Community All times are GMT Page 37
Try It #4! Introduction About Flow Unix Beginning Analysis Basic SiLK Tools rwfilter We’re using anonymized flow in the repository at /data. Printing and Sorting Tools Counting Tools SSH in to the server and determine: Other Tools 1. Which dates is data available for? Advanced Sets Bags 2. What classes and types of data are available? Prefix Maps Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 38
The Training Repository Based on LBNL Anonymized data set Introduction About http://www.icir.org/enterprise-tracing/Overview.html Flow Unix Sensor name and date/time locates data within the Beginning Analysis repository Basic SiLK Tools rwfilter ◮ S0 – anonymized general flows Printing and Sorting Tools Counting Tools ◮ S1 – anonymized scanning flows, different Other Tools anonymization Advanced Sets ◮ Selected dates and times in 2004 and 2005 Bags Prefix Maps ◮ Avaliable data types: Unix Scripting ◮ out, outweb: source internal, destination not internal Visualization Basic Graphs ◮ in, inweb: source not internal, destination internal Excel Gnuplot Timeouts Advanced Graphs Conclusion ◮ 1800s (30 min) active timeout The Community ◮ 60s inactive timeout Page 39
Dates in Sample Data Introduction About Flow Unix Beginning Analysis ◮ 2004/10/04:20-22 Basic SiLK Tools rwfilter ◮ 2004/12/15:08-23 Printing and Sorting Tools Counting Tools ◮ 2004/12/16:01-06,16-23 Other Tools ◮ 2004/12/17:00-03 Advanced Sets Bags ◮ 2005/01/06:19-23 Prefix Maps Unix Scripting ◮ 2005/01/06:00-06,10-23 Visualization ◮ 2005/01/08:00-05 Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 40
Try It #5! Introduction About We’ve glossed over the nuance of how SiLK handles Flow Unix ICMP flows. Type in the following command and look at Beginning Analysis the output: Basic SiLK Tools rwfilter Printing and Sorting Tools rwfilter --type=in --start-date=2004/10/04 \ Counting Tools Other Tools --protocol=1 --max-pass-records=10 \ Advanced --pass-destination=stdout \ Sets Bags | rwcut --fields=sip,sport,dip,dport,icmptypecode Prefix Maps Unix Scripting Visualization 1. How does SiLK store ICMP type and code Basic Graphs Excel information? Gnuplot Advanced Graphs 2. What did this command actually do? Conclusion The Community Page 41
Introduction About Flow Unix Beginning Analysis What have we done so far? Basic SiLK Tools rwfilter ◮ An Introduction to Flow Printing and Sorting Tools Counting Tools ◮ A Brief Discussion of Unix Other Tools Advanced ◮ A Flow Analysis Teaser Sets Bags Prefix Maps Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 42
Section Outline Introduction About Flow Unix Beginning Analysis Basic SiLK Tools Basic SiLK Tools rwfilter Printing and Sorting Tools ◮ rwfilter Counting Tools Other Tools ◮ Printing and Sorting Tools Advanced Sets ◮ Counting Tools Bags Prefix Maps ◮ Other Tools Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 43
So much to do, so little time... Introduction About Flow Unix Beginning Analysis We can’t discuss all parameters for every tool Resources Basic SiLK Tools rwfilter ◮ Analyst’s handbook Printing and Sorting Tools Counting Tools ◮ SiLK Reference Guide (hardcopy man pages) Other Tools Advanced ◮ rw[something] --help Sets Bags ◮ man rw[something] Prefix Maps Unix Scripting ◮ http://tools.netsa.cert.org Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 44
rwfilter Introduction About Flow Unix Beginning Analysis Basic SiLK Tools rwfilter Printing and Sorting Tools Counting Tools Other Tools Advanced Sets Bags Prefix Maps Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 45
rwfilter Command Structure Introduction About Flow Unix Beginning Analysis ◮ Most of the time: any order of parameters Basic SiLK Tools rwfilter ◮ Parameters may be abbreviated to unique prefix Printing and Sorting Tools Counting Tools ◮ Five different groups of parameters: Other Tools ◮ Input – file, repository, pipe Advanced Sets ◮ Selection – which part of repositiory Bags ◮ Partitioning – which flows among the selected Prefix Maps Unix Scripting ◮ Output – going where (pipe, file) Visualization ◮ Other – IP version, filter statistics, etc. Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 46
rwfilter Command Flow Introduction About Flow Unix --class INPUT Beginning Analysis --type PARAMETERS --sensor PIPE --print-filenames --flowtypes Basic SiLK Tools rwfilter FILE Printing and Sorting PARTITIONING Tools PARAMETERS Counting Tools SELECTION Other Tools PARAMETERS Advanced OUTPUT Sets REPOSITORY PARAMETERS Bags PIPE Prefix Maps Unix Scripting FILE Visualization OTHER PARAMETERS Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 47
rwfilter Requirements Each rwfilter call must have: Introduction ◮ Somewhere to get records from: About Flow ◮ File name Unix Beginning Analysis ◮ --input-pipe=stdin or other pipe Basic SiLK Tools ◮ Repository (default or rwfilter --data-rootdir=./myarchive ) with selection Printing and Sorting Tools Counting Tools parameters (type, sensor, start-date, end-date, class) Other Tools ◮ Some description of what records are wanted Advanced Sets (partitioning parameters) Bags Prefix Maps ◮ Some description of where records should go: Unix Scripting ◮ --pass=myfile.rw Visualization ◮ --fail=stdout Basic Graphs Excel ◮ --print-statistics Gnuplot Advanced Graphs rwfilter --start-date=2008/12/05:00 \ Conclusion --end-date=2008/12/05:03 --type=all \ The Community --protocol=6 --packets=1-3 --pass=dec05.rw Page 48
Selection Parameters Introduction About These options control access to repository files Flow Unix Beginning Analysis ◮ --start-date=2007/10/03:00 Basic SiLK Tools ◮ --end-date=2007/10/03T03 rwfilter Printing and Sorting Tools ◮ --sensor=S0 Counting Tools Other Tools ◮ --class=all Advanced Sets ◮ --type=in Bags Prefix Maps Unix Scripting Alternatively, use a pipe or a file Visualization ◮ --input-pipe=stdin – Useful for chaining filters Basic Graphs Excel through stdin/stdout Gnuplot Advanced Graphs ◮ myfile.rw – Useful for filtering previous results Conclusion The Community Page 49
Partitioning Parameters Partioning is the most complex Introduction About Flow Unix Beginning Analysis Basic SiLK Tools rwfilter Flow Record Fields Printing and Sorting IP Sets Tools Counting Tools User pmaps and Country Codes Other Tools Tuples Advanced Dynamic Libs Sets PySiLK Bags Prefix Maps Unix Scripting Visualization Basic Graphs Excel ◮ Partitioning parameters form an “and” expression Gnuplot Advanced Graphs ◮ Too few parameters means too much output Conclusion ◮ Can refine partitioning with another rwfilter call The Community ◮ Some of these are beyond the scope of this course Page 50
Flow partitioning parameters: Record Fields Introduction Pass records based on flow record fields; one is required About Flow ◮ --[not-]saddress, --[not-]daddress : Wildcard Unix Beginning Analysis like 12.5,7,9.2-250.x Basic SiLK Tools rwfilter ◮ --protocol : IP protocol Printing and Sorting Tools ◮ --sport, --dport, --aport TCP, UDP ports Counting Tools Other Tools (caveat: ICMP) Advanced Sets Bags ◮ Prefix Maps --tcp-flags=SF; --flags-all=S/SA; --fin-flag ;... Unix Scripting ◮ --icmp-type; --icmp-code Visualization Basic Graphs ◮ --bytes, --packets, --bytes-per-packet Excel Gnuplot Advanced Graphs At least one partitioning parameter is required Conclusion ◮ Use --proto=0- to pass all The Community Page 51
Flow partitioning parameters: Flow Record Time Introduction About Flow Unix start-date, end-date choose repository files, but do not Beginning Analysis look at the actual flow records Basic SiLK Tools rwfilter Printing and Sorting ◮ --stime , --etime : choose flows that start (or end) Tools Counting Tools within a time range Other Tools Advanced ◮ --active-time : flows active in a time range Sets Bags ◮ Time format: YYYY/MM/DD:HH:MM:SS Prefix Maps Unix Scripting ◮ Time range format: [Time]-[Time] Visualization Basic Graphs Duration Excel Gnuplot Advanced Graphs ◮ --duration=1-10 : number of seconds the flow was Conclusion active The Community Page 52
Flow partitioning Parameters: Flags Introduction About Flow Unix Beginning Analysis Basic SiLK Tools ◮ --tcp-flags=[FSRPAUEC] rwfilter Printing and Sorting Tools ◮ --fin-flag, --syn-flag , etc. Counting Tools Other Tools ◮ --flags-all=[FSRPAUEC]/[FSRPAUEC] Advanced Sets ◮ --flags-initial=[FSRPAUEC]/[FSRPAUEC] Bags Prefix Maps ◮ --flags-session=[FSRPAUEC]/[FSRPAUEC] Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 53
Flow partitioning Parameters: Advanced Introduction About Flow Some of these will be discussed later: Unix Beginning Analysis ◮ --max-pass : limit the number of records passed Basic SiLK Tools rwfilter ◮ --sipset , --dipset , etc: limit to specfici IP Printing and Sorting Tools Counting Tools addresses Other Tools ◮ --ipport : IP/port pairs Advanced Sets Bags ◮ --pmap ; prefix map Prefix Maps Unix Scripting ◮ --dynamic-library : dynamically loaded library Visualization ◮ --scc , --dcc : country codes Basic Graphs Excel ◮ compression Gnuplot Advanced Graphs Conclusion The Community Page 54
Output Parameters Introduction About Flow rwfilter leaves the flows in binary(compact) form Unix Beginning Analysis ◮ --pass , --fail : direct the flows to a file or pipe Basic SiLK Tools rwfilter ◮ --all : destination for everything pulled from the Printing and Sorting Tools Counting Tools repository Other Tools Advanced ◮ One output is required but more than one can be Sets used Bags Prefix Maps Unix Scripting Other useful output: Visualization ◮ --print-statistics Basic Graphs Excel ◮ --print-volume-statistics Gnuplot Advanced Graphs Conclusion The Community Page 55
Other Parameters Introduction About Flow Unix ◮ --dry-run : test the command (useful for scripting) Beginning Analysis Basic SiLK Tools ◮ --ipversion=6 : process IPv6 data rwfilter Printing and Sorting Tools ◮ --print-filenames : print files from which flow Counting Tools Other Tools records came Advanced ◮ --help : print condensed help text Sets Bags Prefix Maps ◮ --man : print manual page Unix Scripting ◮ --version : print configuration info Visualization Basic Graphs ◮ --threads : parallelize rwfilter run Excel Gnuplot Advanced Graphs Conclusion The Community Page 56
Try It #6! Introduction About Flow Unix The time to run an initial query against the repository Beginning Analysis Basic SiLK Tools often depends on the number of files which will be rwfilter accessed. How many files in the repository will be opened Printing and Sorting Tools Counting Tools with this command? Other Tools Advanced Sets rwfilter --sensor=s0 \ Bags --start-date=2004/12/15:19 Prefix Maps Unix Scripting Visualization (note: you have to add extra parameters to this command Basic Graphs Excel to make it work) Gnuplot Advanced Graphs Conclusion The Community Page 57
Try It #7! Introduction About Flow Unix Beginning Analysis Often you will want to track an individual address or Basic SiLK Tools address block. Develop a filter command to retrieve: rwfilter Printing and Sorting ◮ Flows to the 131.243.10.0/24 CIDR block, Tools Counting Tools Other Tools ◮ Leaving our network, Advanced ◮ On 12/16/2004 at 17:00 GMT, Sets Bags Prefix Maps ◮ And save the flows in the file netblock.rw . Unix Scripting How many packets, bytes and flows were retrieved? Visualization Basic Graphs How many packets, bytes, and flows were retrieved? Excel Gnuplot Advanced Graphs Conclusion The Community Page 58
Try It #8! Introduction About Let’s look for short, bursty outbound ssh traffic. Develop Flow Unix a filter command that does the following: Beginning Analysis Basic SiLK Tools ◮ Pulls out all outbound ssh (TCP port 22) flows, rwfilter Printing and Sorting Tools ◮ On 12/17/2004, Counting Tools Other Tools ◮ Between 00:00 and 04:00 GMT, Advanced Sets ◮ That lasted less than 60 seconds, Bags Prefix Maps ◮ With an average of more than 60 bytes per packet, Unix Scripting ◮ And store the result in a file named short-ssh.raw Visualization Basic Graphs Excel How many records did you retrieve? How many files in Gnuplot Advanced Graphs the repository were opened? Conclusion The Community Page 59
Try It #9! Introduction About Flow Unix Beginning Analysis Basic SiLK Tools rwfilter Printing and Sorting Tools Examine traffic trends. What is the change in mail traffic Counting Tools Other Tools volume between 19:00 and 20:00 hours on 12/15/2004 for Advanced the mail server at 128.3.26.249? Sets Bags Prefix Maps Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 60
Chaining filters It is often very efficient to chain rwfilter Introduction commands together About Flow Unix ◮ Use --pass and --fail to Beginning Analysis segregate bins Basic SiLK Tools rwfilter ◮ Use --all so you only pull from the Printing and Sorting Tools Counting Tools repository once Other Tools Advanced Sets Bags Prefix Maps Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 61
What Is This #8 Introduction About Flow Unix rwfilter \ Beginning Analysis Basic SiLK Tools --start-date=2007/09/30 \ rwfilter --type=outweb \ Printing and Sorting Tools Counting Tools --bytes=100000- \ Other Tools --pass=stdout \ Advanced Sets | rwfilter \ Bags Prefix Maps --input-pipe=stdin \ Unix Scripting --duration=60- \ Visualization --pass=long-http.rw \ Basic Graphs Excel --fail=short-http.rw Gnuplot Advanced Graphs Conclusion The Community Page 62
Try It #10! Introduction About Flow Unix Beginning Analysis Let’s revisit the last example for some more analysis. For Basic SiLK Tools rwfilter the mail server at 128.3.26.249, and looking only at Printing and Sorting Tools outbound traffic for the 19:00 hour on 12/15/2004, use a Counting Tools Other Tools single command to find out both: Advanced Sets ◮ The total number of SMTP flows (TCP port 25), and Bags Prefix Maps ◮ The number of flows which were for outbound Unix Scripting messages Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 63
Common rwfilter Typos Introduction About Flow ◮ --port or --destport : not an option name Unix Beginning Analysis ◮ --saddress=file : pointing to a filename; should be Basic SiLK Tools rwfilter an IP Printing and Sorting Tools ◮ --sip=10.1.2.3 : sip specifies an IPSet; use saddr Counting Tools Other Tools for addresses Advanced Sets ◮ --start=2005/11/04:06:00:00 start-date and end Bags Prefix Maps date use only down to the hour Unix Scripting ◮ ---start-date : should be only two dashes Visualization Basic Graphs ◮ -- start=2007/05/22 : no space between -- and the Excel Gnuplot option Advanced Graphs Conclusion The Community Page 64
But I can’t read binary... Introduction About rwcut provides a way to display binary records as Flow Unix human-readable ASCII Beginning Analysis Basic SiLK Tools ◮ Useful for printing flows to the screen rwfilter Printing and Sorting ◮ Useful for input to text processing tools Tools Counting Tools Other Tools ◮ You’ll usually only need the --fields argument Advanced Sets sip packets sval flags application Bags Prefix Maps dip bytes dval initialflags icmptypecode Unix Scripting sport Sensor in, out sessionflags attributes Visualization Basic Graphs dport scc dur dur+msec type Excel protocol dcc stime stime+msec stype Gnuplot Advanced Graphs class nhip etime etime+msec dtype Conclusion The Community Page 65
Pretty Printing SiLK Output Introduction Default output is fixed-width pipe delimited data About Flow Unix Beginning Analysis sIP| dIP|pro|pkts|bytes| Basic SiLK Tools 207.240.215.71| 128.3.48.203| 1| 1| 60| rwfilter Printing and Sorting 207.240.215.71| 128.3.48.68| 1| 1| 60| Tools Counting Tools 207.240.215.71| 128.3.48.71| 1| 1| 60| Other Tools Advanced Sets Tools with text output have these formatting options Bags Prefix Maps ◮ --no-titles : suppress the first row Unix Scripting Visualization ◮ --no-columns : suppress the spaces Basic Graphs Excel ◮ --delimited ; --column-separator Gnuplot Advanced Graphs ◮ --legacy-timestamps : better for import to Excel Conclusion The Community Page 66
Try It #11! Introduction About Flow Unix Create a file ssh.rw that contains all outbound SSH flows Beginning Analysis from 12/16/2004:17. Experiment with rwcut and Unix Basic SiLK Tools rwfilter text tools to try and sort out records: Printing and Sorting Tools Counting Tools 1. Can you tell which flows are from internal SSH Other Tools servers, and which are from external SSH servers? Advanced Sets Bags 2. Which flows look like SSH keep-alives? Prefix Maps Unix Scripting 3. Which flows had the most data transfer? Visualization Try to write rwfilter commands against ssh.rw to query Basic Graphs Excel these records, and display them with rwcut Gnuplot Advanced Graphs Conclusion The Community Page 67
rwsort Introduction About Flow Why sort flow records? Unix Beginning Analysis ◮ Records are recorded as received, not in time order Basic SiLK Tools rwfilter (look at records from the last exercise) Printing and Sorting Tools ◮ Analysis often requires finding outliers Counting Tools Other Tools rwsort options Advanced Sets ◮ fields (same as rwcut) is required Bags Prefix Maps ◮ in, out (stdin / stdout are defaults) Unix Scripting Visualization ◮ For improved sorts, specify a buffer size Basic Graphs Excel ◮ For large sorts, specify a temporary directory Gnuplot Advanced Graphs Conclusion The Community Page 68
I only believe what I see Introduction About Flow Unix Beginning Analysis Basic SiLK Tools You’ll be tempted to work with text-based records rwfilter Printing and Sorting ◮ It’s easy to see the results and postprocess with other Tools Counting Tools Other Tools tools (e.g., perl) Advanced ◮ It takes a lot of space, and it’s much, much slower Sets Bags Prefix Maps Guiding Principle: Keep flows in binary format as long as Unix Scripting possible Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 69
Try It #12! Introduction About Flow Unix Beginning Analysis Often HTTP beaconing consists of very small HTTP Basic SiLK Tools requests. Let’s get a feel for what HTTP data looks like, rwfilter Printing and Sorting even before we start to find these beacons. Tools Counting Tools What do the smallest outbound HTTP web client flows Other Tools look like on 12/15/2004? Advanced Sets Bags ◮ First, find them using rwsort Prefix Maps Unix Scripting ◮ Second, find them using sort Visualization ◮ Which was faster? Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 70
Counting Tools Introduction About Flow Unix Beginning Analysis Basic SiLK Tools The suite contains several counting tools: rwfilter Printing and Sorting Tools ◮ rwcount - count across time Counting Tools Other Tools ◮ rwaddrcount - count across addresses Advanced Sets ◮ rwuniq - count on arbitrary field combinations Bags Prefix Maps ◮ rwstats - descriptive statistics and counts Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 71
rwcount Introduction About Flow Unix Beginning Analysis Basic counting: Basic SiLK Tools rwfilter Printing and Sorting ◮ rwcount myfile.rw > count_file Tools Counting Tools ◮ Produces byte, packet and flow totals by time Other Tools Advanced Sets Common Options: Bags Prefix Maps ◮ --bin-size : changes the size of each bin (in seconds) Unix Scripting ◮ --skip-zeroes : should empty bins be printed? Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 72
rwaddrcount Introduction About Flow Basic counting: Unix Beginning Analysis ◮ rwaddrcount --[print-option] myfile.rw Basic SiLK Tools rwfilter Printing and Sorting ◮ --use-dest to work with dIP; default is sIP Tools Counting Tools Other Tools Print Options: Advanced ◮ --print-stat : Lists total number of addresses found Sets Bags Prefix Maps ◮ --print-ips : Just print out the IP address, nothing Unix Scripting else Visualization Basic Graphs ◮ --print-recs : Lists bytes, packets, records, times Excel Gnuplot for each address Advanced Graphs Conclusion The Community Page 73
rwstats Introduction About Great for generating top-N, bottom-N lists Flow Unix Group by (choose one or two): Beginning Analysis Basic SiLK Tools ◮ Addresses rwfilter Printing and Sorting ◮ Ports Tools Counting Tools Other Tools ◮ Protocols Advanced Sets Output Limit Bags Prefix Maps ◮ Count Unix Scripting ◮ Top, Bottom Visualization Basic Graphs ◮ Threshold (specific value range) Excel Gnuplot Advanced Graphs ◮ Percentage Conclusion The Community Page 74
rwuniq Introduction About Flow Unix The more general case for rwstats Beginning Analysis Basic SiLK Tools Mirrors the unix “uniq -c” command rwfilter Printing and Sorting ◮ Creates a giant hash table where you define the key Tools Counting Tools ◮ Memory is expensive, so we can’t uniq everything Other Tools Advanced Common Options: Sets Bags Prefix Maps ◮ --fields : same as cutting and sorting Unix Scripting ◮ --all-counts : collect bytes, packets and flows Visualization Basic Graphs ◮ --bin-time : size the bins when uniq’ing on time Excel Gnuplot Advanced Graphs Conclusion The Community Page 75
Try It #13! Introduction About Flow Unix Find scans traffic in the sample data (while the Beginning Analysis Basic SiLK Tools anonymizer removed some of the simple scans, they didn’t rwfilter find them all). When you find one, answer the following Printing and Sorting Tools Counting Tools questions: Other Tools Advanced ◮ What type of scan was it? Sets Bags ◮ When did it start/end? Prefix Maps Unix Scripting ◮ How fast was it? Visualization ◮ What did the scanner discover? Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 76
Try It #14! Introduction About Flow Unix Beginning Analysis Basic SiLK Tools rwfilter Printing and Sorting This example showcases the very useful --dip-distinct Tools Counting Tools feature of rwuniq: Other Tools Advanced For 2004/12/15, how many clients connected to the Sets highest volume web servers? Bags Prefix Maps Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 77
Oops...I forgot where this came from... Introduction About Flow Unix Beginning Analysis rwfileinfo Basic SiLK Tools ◮ Each SiLK file (flows, sets, bags, prefix maps, etc.) rwfilter Printing and Sorting has a header which logs data Tools Counting Tools Other Tools ◮ rwfileinfo prints out that data Advanced ◮ For flow files, it also (usually) keeps a history of the Sets Bags commands used to generate the file Prefix Maps Unix Scripting Try It! Visualization Basic Graphs ◮ rwfileinfo *.rw Excel Gnuplot Advanced Graphs Conclusion The Community Page 78
When the files are LARGE Introduction About Flow Unix Beginning Analysis Basic SiLK Tools rwcat rwfilter ◮ Send a binary rw file to stdout Printing and Sorting Tools Counting Tools rwappend Other Tools Advanced ◮ Join multiple files together Sets Bags rwsplit Prefix Maps Unix Scripting ◮ Carve large files into pieces Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 79
How long will this really take? Introduction About Flow Unix Beginning Analysis Basic SiLK Tools rwfilter rwfglob Printing and Sorting Tools Counting Tools ◮ Find out which files will be pulled from the repository Other Tools Advanced ◮ Find out whats available and whats missing Sets Bags ◮ Use the output in other file-processing scripts Prefix Maps Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 80
Maintain anonymity Introduction About Flow Unix Beginning Analysis rwnetmask Basic SiLK Tools rwfilter ◮ Mask off low order bits of source and/or destination Printing and Sorting Tools addresses Counting Tools Other Tools rwrandomizeip Advanced Sets ◮ Randomly replace source and destination addresses Bags Prefix Maps rwtuc Unix Scripting ◮ Change text flow data into binary (opposite of rwcut) Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 81
Who was that? Introduction About Flow Unix Beginning Analysis Basic SiLK Tools rwfilter rwresolve Printing and Sorting Tools Counting Tools ◮ Perform a DNS lookup on text output Other Tools Advanced ◮ Caveat: it uses your analysis host’s DNS resolver Sets Bags ◮ Caveat: DNS is subject to change Prefix Maps Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 82
What have we done so far? Introduction About Flow Unix Beginning Analysis Basic SiLK Tools Basic SiLK Tools rwfilter Printing and Sorting Tools ◮ rwfilter Counting Tools Other Tools ◮ Printing and Sorting Tools Advanced Sets ◮ Counting Tools Bags Prefix Maps ◮ Other Tools Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 83
Section Outline Introduction About Flow Unix Beginning Analysis Basic SiLK Tools rwfilter Advanced SiLK Tools Printing and Sorting Tools Counting Tools ◮ Sets Other Tools Advanced ◮ Bags Sets Bags ◮ Prefix Maps Prefix Maps Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 84
Blacklists, Whitelists, Books of Lists... Introduction About Too many addresses for the command line? Flow Unix ◮ Spam block list Beginning Analysis Basic SiLK Tools ◮ Malicious web sites rwfilter Printing and Sorting Tools ◮ Arbitrary list of any type of addresses Counting Tools Other Tools Create an IP set! Advanced Sets ◮ Individual IP address in dotted decimal or integer Bags Prefix Maps ◮ CIDR blocks, 192.168.0.0/16 Unix Scripting ◮ Wildcards, 10.4,6.x.2-254 Visualization Basic Graphs Excel Use it directly within your filter commands Gnuplot Advanced Graphs ◮ --sip , --dip , --anyset Conclusion The Community Page 85
Set Tools Introduction About Flow Unix Beginning Analysis Basic SiLK Tools rwset : create sets from binary flows rwfilter Printing and Sorting rwsetbuild : create sets from text Tools Counting Tools Other Tools rwsetcat : print out an IP set into text ( very useful ) Advanced rwsetmember : test if IP is in given IP sets Sets Bags rwsettool : perform set algebra (set, union, intersection) Prefix Maps on multiple IP sets Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 86
Try It #15! Introduction About Flow Unix Beginning Analysis Basic SiLK Tools rwfilter Printing and Sorting Tools Flow is also very useful for creating network inventories. Counting Tools Other Tools What /24 net blocks are populated within my network? Advanced Which block has the densest population? Sets Bags Prefix Maps Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 87
Other uses of IP Sets Introduction About Flow Unix Beginning Analysis Basic SiLK Tools Perform set arithmetic on IP data rwfilter Printing and Sorting Tools ◮ What addresses on my spam blacklist are also bot Counting Tools Other Tools infected? Advanced Sets Randomly select items for sampling Bags Prefix Maps ◮ rwsettool --sample --size=100 Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 88
Bags: sets with attitude Introduction About Flow Unix Beginning Analysis Basic SiLK Tools rwfilter Bags are generally IPSets with an associated integer Printing and Sorting Tools ◮ Usually a count or sum Counting Tools Other Tools ◮ Could also be ports or protocols Advanced Sets Bags can make sets Bags Prefix Maps Math operations can be performed on bags Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 89
Try It #16! Let’s look for DNS clients that are using an external DNS Introduction About resolver. Flow Unix ◮ First, let’s take a moment to review DNS: Beginning Analysis Basic SiLK Tools ◮ When a client wants an address, it asks its local DNS rwfilter Printing and Sorting server Tools Counting Tools Other Tools ◮ The local DNS server does all the work Advanced Sets Bags Prefix Maps Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 90
Try It #16!(2) The Local DNS Server should be local Introduction ◮ Can be assigned manually or by DHCP About Flow Unix ◮ Up to three can be assigned, but often only one is Beginning Analysis used Basic SiLK Tools rwfilter Printing and Sorting Tools Counting Tools Other Tools Advanced Sets Bags Prefix Maps Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 91
Try It #16!(3) Once again, let’s look for DNS clients that are using an Introduction About external DNS resolver: Flow Unix ◮ Use bags to count the number of outbound DNS Beginning Analysis Basic SiLK Tools connections per address, rwfilter ◮ Create a candidate set from that bag of addresses Printing and Sorting Tools Counting Tools with more than 100 outbound flows, and Other Tools Advanced ◮ Find the number of unique destination addresses for Sets Bags the candidates. Prefix Maps Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 92
Prefix Maps(pmaps): sets with bling Assign an arbitrary label to address prefixes Introduction ◮ Start with a text file of IP ranges and labels About Flow ◮ Order from least to most specific Unix Beginning Analysis ◮ Compile the text file with rwpmapbuild Basic SiLK Tools rwfilter ◮ Print out the pmap with rwpmapcat Printing and Sorting Tools Counting Tools The input file: Other Tools Advanced Sets 10.0.0.0/8 Private Unassigned Bags Prefix Maps 192.168.0.0/16 Private Unassigned Unix Scripting 172.16.0.0/12 Private Unassigned Visualization 10.0.1.100 10.0.1.200 Workstation DHCP Basic Graphs Excel 10.0.1.1 10.0.1.50 Servers Gnuplot Advanced Graphs 10.0.2.1 10.0.2.50 Servers Conclusion 10.0.3.1 10.0.3.50 DMZ Servers The Community No other pmap tools (?!?) Page 93
Using pmaps Introduction About Flow pmaps don’t have their own tools, they fit in with existing Unix Beginning Analysis tools Basic SiLK Tools rwfilter ◮ rwfilter Printing and Sorting Tools Counting Tools ◮ rwsort Other Tools ◮ rwcut Advanced Sets Bags ◮ rwuniq Prefix Maps Unix Scripting This allows you to add your own fields to flow Visualization ◮ Query all your servers: Basic Graphs Excel rwfilter --sval="Servers","DMZ Servers" Gnuplot Advanced Graphs Conclusion The Community Page 94
Port-based pmaps Introduction About Flow Unix Beginning Analysis Basic SiLK Tools rwfilter Printing and Sorting It’s also possible to create prefix maps based on ports. Tools Counting Tools Other Tools ◮ Useful for well-known service ports; e.g., IRC, HTTP Advanced ◮ Also useful for ICMP Sets Bags Prefix Maps Unix Scripting Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 95
Try It #17! Introduction About Flow Unix Beginning Analysis Create an ICMP prefix map from the ICMP types (or Basic SiLK Tools types and codes). rwfilter Printing and Sorting Tools ◮ Look at unassigned ICMP type/code values that are Counting Tools Other Tools in use. Which ICMP types receive the most traffic? Advanced ◮ Note: ICMP type/code values are assigned by Sets Bags RFC792; a summary table is available from IANA at Prefix Maps Unix Scripting http: Visualization //www.iana.org/assignments/icmp-parameters Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 96
Are we there yet? Introduction About Flow Unix Beginning Analysis Basic SiLK Tools Advanced SiLK Tools rwfilter Printing and Sorting ◮ Splitting and merging Tools Counting Tools ◮ Sets Other Tools Advanced ◮ Bags Sets Bags ◮ Prefix Maps Prefix Maps Unix Scripting More reliance on examples to demonstrate these concepts Visualization Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 97
Scripting Introduction About Flow Unix Why script? Beginning Analysis ◮ Repeatable analyses Basic SiLK Tools rwfilter ◮ Encapsulating syntax Printing and Sorting Tools Counting Tools ◮ Composing complex commands Other Tools Advanced How to script? Sets Bags ◮ Shell scripting (we’ll use bash) Prefix Maps Unix Scripting ◮ Good reference: http://tldp.org/LDP/abs/html/ Visualization ◮ Python (beyond this class, but widely used) Basic Graphs Excel ◮ Good reference: http://docs.python.org/tut/ Gnuplot Advanced Graphs Conclusion The Community Page 98
Getting started Introduction About Flow Unix Beginning Analysis Put your typed commands into a file Basic SiLK Tools rwfilter ◮ From our second example, ls -lR /data |grep Printing and Sorting Tools "/data" Counting Tools Other Tools Run the file Advanced Sets ◮ bash script.sh or Bags Prefix Maps ◮ sh script.sh or Unix Scripting Visualization ◮ chmod +x script.sh ; ./script.sh Basic Graphs Excel Gnuplot Advanced Graphs Conclusion The Community Page 99
Recommend
More recommend
