Traffic Measurement Lothar Braun Outline Why do we need to measure - - PowerPoint PPT Presentation

Chair for Network Architectures and Services

Department of Informatics Technische Universität München

Traffic Measurement

Lothar Braun

2

Outline

 Why do we need to measure traffic in the Internet?  Active measurement vs. passive measurement  Passive measurement: Packet-level vs. flow-level  Protocols for transporting measurement data  What are you going to do in the Lab?

3

Outline

 Why do we need to measure traffic in the Internet?  Active measurement vs. passive measurement  Passive measurement: Packet-level vs. flow-level  Protocols for transporting measurement data  What are you going to do in the Lab?

4

Why do You Measure in the Lab?

 Task: Setup your lab environment  It doesn’t work!  What do you do to find the problem?

5

Why do We Need to Measure in the Internet?

 Problems get worse in large-scale networks  Openness

• No/little access control for new Internet participants
• Free deployment of new applications and services

 Heterogeneity

• Different technologies at lower layers
• Different protocols and services on top of IP

 different requirements / different failures

 Consequences for network operators

• Little control of utilization of network resources

 possible misuse: hackers, attacks, spam

• Little knowledge about applications and services

 unpredictable traffic

 Traffic measurements can help to understand our network

6

Applications of Traffic Measurements

 What information is useful for network operators?  Network monitoring

• Fault detection
• Connectivity and routing

 Performance measurements

• Link capacity and utilization
• Quality of Service parameters

(delay, jitter, throughput)

 Accounting and charging

• Traffic volume per customer or peering AS

 Network security

• Worm and attack detection

Map of the MWN (December 2010)

7

Outline

 Why do we need to measure traffic in the Internet?  Active measurement vs. passive measurement  Passive measurement: Packet-level vs. flow-level  Protocols for transporting measurement data  What are you going to do in the Lab?

8

Active Measurements

 Methodology

• Probe packets exchanged

between two end-systems

• Measurement of packet loss,
• ne-way delay, round-trip times,

packet interarrival times

 Analysis

• Complete packet loss  link down, invalid route, router defect
• Partial packet loss  available bandwidth, level of congestion
• Delay = propagation time + buffer time  distance, filling level of buffers
• Interarrival times of packet pairs/trains  path capacity

 Pros and cons

• Does not require access to internal network components
• Intrusive  existing traffic is disturbed
• No information about existing traffic

9

Passive Measurements (= Focus of this Lab)

 Methodology

• Observation of existing traffic using

monitoring probes in the network

• Measurement of traffic volume, traffic

composition, packet interarrival times

• Different levels of granularity:

packet-level, flow-level, link-level

 Analysis

• Measurement of network utilization for accounting and traffic engineering
• Measurement of quality-of-service parameters (e.g., throughput, delay)
• Detection of failures, traffic anomalies, flooding attacks and scans
• Traffic characterization with deep packet inspection

 Pros and cons

• Non-intrusive  existing traffic is not disturbed
• Installation of monitoring probes at appropriate locations in the network

10

Outline

 Why do we need to measure traffic in the Internet?  Active measurement vs. passive measurement  Passive measurement: Packet-level vs. flow-level  Protocols for transporting measurement data  What are you going to do in the Lab?

11

Packet-Level Measurements

 Gather information about individual packets:

• Observation time
• Location (interface and direction)
• Packet header information (link layer, IP header, transport header)
• Packet payload

31 16

Source Port Destination Port Sequence Number Acknowledgement

4 bit TCP header length

6 bit unused U R G A C K P S H R S T S Y N F I N

Window Checksum Urgent Pointer Options GET /index.html HTTP/1.1 ... Application ports Connection Information Application payload

12

Using Packet-Level Measurements

 You already used packet-level measurements

• For understanding the SCTP traffic in the SCTP lab

You can look at all the packets You can look into the packets You can identify problems with the packets You can look into the application protocol

13

Signature Detection in Packets

 Some applications try to look for patterns in the payload

• Application identification
• Search for attacks like worm or botnet traffic

 Application detection

• Idea: Identify an application by looking for specified protocol fields
• Example: HTTP
• GET /index.html HTTP/1.1

 Malicious traffic detection

• Idea: Security analysts know how malicious traffic looks like
• Signature detection systems try to find such patterns
• These systems are often called Intrusion Detection Systems

14

Example: IRC-based botnets

 Internet Relay Chat (IRC)

• Real-time chat
• Users join a chat room
• Also used by botnets to control

bot clients

 Important properties

• Every user must choose a

nickname

• Each nickname must be unique
• n a network

 Bot clients need to generate nicknames

 Intrusion Detection Systems

can look for such automatically generated nicks

15

Signature Detection: Snort

 Snort is an Intrusion Detection System  Operation

• Capture network packets like wireshark
• Perform signature matching on packets according to signature database

 Signature database must be configured by the users

alert tcp $HOME_NET any -> $EXTERNAL_NET any (flow:established,to_server; flowbits:isset,is_proto_irc; content:"USER XP-"; pcre:"/USER XP-[A-z0-9]{4,8} \* 0 \:.*/"; msg:"E4[rb] ET TROJAN Likely Bot Username in IRC (XP-..)"; classtype:trojan-activity; reference:url,doc.emergingthreats.net/2008123; reference:url,www.emergingthreats.net/cgi- bin/cvsweb.cgi/sigs/VIRUS/TROJAN_IRC_Bots; sid:2008123; rev:4;)

16

Using Packet-Level Measurements

 Also useful: tcpdump  It will give you network

and transport layer information about the traffic

 Important information:

• Packet loss

 Not a problem on my

desktop system, but …

17

Packet-Level Measurements in Large Networks

 … can be a big problem for larger systems

• Example: X-Win from MWN to DFN (Internet access)
• 10 Gigabit Ethernet link
• Connects more than 80.000 systems to the Internet

 Problem

• Processing of all packets requires a lot of resources
• Packet loss will occur if you do not have these resources

18

Packet Selection

 Most applications require only specific packets to be observed  Packet selection

• Capture only the interesting subset of packets that you can handle

 Packet Filtering

• Select packets with properties X and Y
• Goal: Select “interesting traffic” that helps to identify problems
• Example: tcpdump -i eth0 tcp and port 80
• Useful if you are interested in HTTP traffic on port 80

 Packet Sampling

• Systematic or probabilistic sampling algorithms
• Goal: Infer statistics of overall traffic from sampled packets
• Example: Select each packet with probability of 10 %

19

Flow-Level Measurements

 Some applications do not require packet-level information  Other information can be more important

• How much traffic is in my network?
• How much UDP traffic is there?
• Is there some client that sends a lot of mail traffic?
• E.g. a spammer in my network?

 Answering these questions does not require packet data

• Instead: It requires information about who communicates with whom

 Solution: Generate and analyze flow data

20

 Flows are often stored for a long time period

• Used for understanding and visualizing traffic

Using Flow Data

Source: NfSen- Homepage

21

Network Flows

 Definition of a flow

• Set of packets common properties called flow keys (often IP-quintuple)
• Observed at one point in the network (e.g., router interface)

 Measured properties

• Observation period (timestamp of first and last packet)
• Number of bytes and packets belonging to the flow

 Flow record = {(flow keys), (measured properties)}

• {(Host A, port a, Host B, port b, TCP), (200 packets, 5000 bytes, 11:00, 11:05)}

 Flow duration

• Start:

first packet with new flow key values

• End:

after timeout or based on signaling (e.g., TCP FIN)

Host A Host B

22

Flow Duration

 Flow expiration after timeout

• Inactive timeout

 maximum gap between two consecutive packets

• Active timeout

 maximum flow duration since flow start

23

Flow Measurement Deployment

 Flows are typically generated at a network router

• Analysis does not happen on the router
• Instead: Data is transmitted to a flow collecting machine

Internal Network Internal Network Internet Internet Flow-Collector Flow Data (e.g. encoded in IPFIX)

24

Outline

 Why do we need to measure traffic in the Internet?  Active measurement vs. passive measurements  Passive measurement: Packet-level vs. flow-level  Protocols for transporting measurement data  What are you going to do in the Lab?

25

Protocols for Transporting Measurement Data

 Motivation

• Monitoring probes have limited

analysis capabilities  transport measurement data to remote traffic analyzer

 IETF protocols

• Simple Network Management Protocol (SNMP)
• Measurement data stored in Management Information Base (MIB) of the

monitoring probe (e.g., using RMON-MIB)

• Traffic analyzer needs send SNMP GET requests at a regular basis
• IP Flow Information eXport (IPFIX)
• Push protocol  monitoring device exports data without being queried
• Usage not limited to flow-level measurement data
• Traffic analyzer operates as IPFIX Collector

26

IPFIX Protocol

 Template based data export

• Measurement data encoded in

Data Records with several fields

• Data Record structure specified

by Template Record

• Data type and semantic of a field

defined by Information Element

• Field length specified in Template

(or in the Data Record itself)

 Protocol description

• IPFIX Exporter sends Template Record prior to associated Data Records

 IPFIX Collector has necessary information to decode Data Records

• Unidirectional communication (no return traffic from Collector to Exporter)

27

IPFIX Protocol: Example



Template Record



Associated Data Set with two Data Records

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 256 | Field Count = 3 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| destinationIPv4Address (12) | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0|destinationTransportPort (11)| Field Length = 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| observationTimeSeconds (332)| Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 256 | Length = 24 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 192.0.2.12 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 80 | 1276696036 ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | 192.0.2.27 ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | 38893 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1276696037 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

28

Outline

 Why do we need to measure traffic in the Internet?  Active measurement vs. passive measurement  Passive measurement: Packet-level vs. flow-level  Protocols for transporting measurement data  What are you going to do in the Lab?

29

Traffic Measurement Lab

 Scenario  Outline

• Implement pcap-based sniffer starting from some skeleton code
• programming language is C
• Extract interesting information from TCP packets
• Export resulting data to Vermont using the IPFIX protocol
• IPFIX exporter library is available
• Use Vermont to receive and analyze the measurement data

Sniffer (IPFIX Exporter) Vermont (IPFIX Collector) Measurement data transmitted with IPFIX protocol

passive traffic monitoring using pcap library display measurement data and search for signatures

30

Traffic Measurement Lab: Learning Targets

 Learn how passive traffic capturing works

• How applications like tcpdump, Wireshark, and Snort work?
• How does flow generation work?

 Get to know the IPFIX protocol  Get familiar with protocol specification in RFCs

• Get a better understanding of IP, TCP, and higher layer protocol
• By extracting protocol header fields and writing signatures

31

Questions?