Packet Capture in 10-Gigabit Ethernet Environments Using - - PowerPoint PPT Presentation

▶
packet capture in 10 gigabit ethernet environments using
SMART_READER_LITE
LIVE PREVIEW

Packet Capture in 10-Gigabit Ethernet Environments Using - - PowerPoint PPT Presentation

Mar 05, 2024 155 likes •419 views

Packet Capture in 10-Gigabit Ethernet Environments Using Contemporary Commodity Hardware Fabian Schneider J org Wallerich Anja Feldmann { fabian,joerg,anja } @net.t-labs.tu-berlin.de Technische Universtit at Berlin Deutsche Telekom

slide-1
SLIDE 1

Packet Capture in 10-Gigabit Ethernet Environments Using Contemporary Commodity Hardware

Fabian Schneider J¨

  • rg Wallerich

Anja Feldmann {fabian,joerg,anja}@net.t-labs.tu-berlin.de

Technische Universtit¨ at Berlin Deutsche Telekom Laboratories

Passive and Active Measurement Conference 5th April 2007

Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 1 / 20

slide-2
SLIDE 2

Introduction Motivation

Motivation

Example Scenario: Network security tool at the edge of your network

  • need access to packet level data for application layer analysis
  • High-speed networks ⇒ high data and packet rate

Challenge: capture full packets without missing any packet

  • One approach: specialized hardware
  • e.g. Monitoring cards from Endace
  • Drawbacks: high costs, single purpose

Question: Is it feasible to capture traffic with commodity hardware?

Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 2 / 20

slide-3
SLIDE 3

Monitoring 10-Gigabit

Outline

1 Monitoring 10-Gigabit

Approach Link Bundling

2 Comparing 1-Gigabit Monitoring Systems 3 Results 4 Summary

Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 3 / 20

slide-4
SLIDE 4

Monitoring 10-Gigabit Approach

Approach for 10-Gigabit Monitoring

  • Problem: No recent host bus or disk

system can handle the bandwidth needs

  • f 10-Gigabit environments
  • Solution: split up traffic and distribute

the load (e.g. 10-Gigabit on multiple 1-Gigabit links)

  • Use a switch: e.g. link bundling

feature

  • Use specialized hardware
  • Keep corresponding data together!

10 x 1GigE 10GigE Monitor

Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 4 / 20

slide-5
SLIDE 5

Monitoring 10-Gigabit Link Bundling

Link Bundling

Feasibility

Etherchannel (Cisco) feature enables link-bundling for:

  • higher bandwidth, redundancy, . . .
  • or load-balancing e. g. for Webservers

Feasibility test:

  • Tested on a Cisco 3750
  • 1-Gigabit Ethernet link split on eight FastEthernet (100 Mbit/s) links.
  • Assign packets to links based on both IP addresses.

⇒ It works with real traffic!

Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 5 / 20

slide-6
SLIDE 6

Monitoring 10-Gigabit Link Bundling

Link Bundling

Load-Balancing

  • Simple switches use only MAC addresses

⇒ Not useful for a router-to-router link

  • On a Cisco 3750: any combination of IP and/or MAC addresses

⇒ is sufficient for our example scenario

  • On a Cisco 65xx: MAC’s, IP’s, and/or Port Numbers

Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 6 / 20

slide-7
SLIDE 7

Comparing 1-Gigabit Monitoring Systems

Comparing 1-Gigabit Monitoring Systems

1 Monitoring 10-Gigabit 2 Comparing 1-Gigabit Monitoring Systems

Methodology System under Test Measurement Setup

3 Results 4 Summary

Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 7 / 20

slide-8
SLIDE 8

Comparing 1-Gigabit Monitoring Systems Methodology

Methodology

  • Comparable priced systems with
  • Different processor architectures
  • Different operating systems
  • Task of those systems:
  • Capture full packets
  • Do not analyze them (Out-of-Scope)
  • Workload:
  • All system are subject to identical input
  • Increase bandwidth up to a fully loaded Gigabit link
  • Realistic packet size distribution
  • Measurement Categories:
  • Capturing Rate: number of captured packets (simple libpcap app)
  • System Load: CPU usage while capturing (simple top like app)

Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 8 / 20

slide-9
SLIDE 9

Comparing 1-Gigabit Monitoring Systems System under Test

Systems under Test

Two examples of any of the systems:

  • One installed with Linux
  • The other with FreeBSD

First set of systems purchased in 2005:

  • 2x AMD Opteron 244 (1 MB Cache, 1.8 GHz),
  • 2x Intel Xeon (Netburst, 512 kB Cache, 3.06 GHz),

Second set purchased in 2006:

  • 2x Dual Core AMD Opteron 270 (1 MB Cache, 2.0 GHz)

All: 2 Gbytes of RAM, optical Intel Gigabit Ethernet card, RAID array

Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 9 / 20

slide-10
SLIDE 10

Comparing 1-Gigabit Monitoring Systems Measurement Setup

Measurement Setup

Generator (LKPG) Cisco C3500XL

  • ptical Splitter (mulitiplies every Signal)

Linux/ AMD Opteron FreeBSD/ Intel Xeon (Netburst) FreeBSD/ AMD Opteron Linux/ Intel Xeon (Netburst) SNMP Interface Counter Queries Workload -> Control Network eth0 eth1 eth2

Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 10 / 20

slide-11
SLIDE 11

Results

1 Monitoring 10-Gigabit 2 Comparing 1-Gigabit Monitoring Systems 3 Results

Using multiple processors? Increasing buffer sizes Additional Insights (I) Write to disk Additional Insights (II)

4 Summary

Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 11 / 20

  • second set of systems

measurements

  • first set of systems

measurements

slide-12
SLIDE 12

Results Using multiple processors? Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 12 / 20

Single processor, 1st Set

Capturing Rate [%] CPU usage [%] Datarate [Mbit/s] Linux/AMD Linux/Intel FreeBSD/AMD FreeBSD/Intel Capturing Rate [%] CPU usage [%] 10 20 30 40 50 60 70 80 90 100 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 10 20 30 40 50 60 70 80 90 100 (32) no SMP, no HT, std. buffers, 1 app, no filter, no load

X-Axis: Generated Bandwidth Lower Part: CPU Usage SP: 100% corresponds to one fully utilised processor MP: 50% corresponds to one fully utilised processor Upper Part: Capturing Rate

slide-13
SLIDE 13

Results Using multiple processors? Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 12 / 20

Single processor, 1st Set

Capturing Rate [%] CPU usage [%] Datarate [Mbit/s] Linux/AMD Linux/Intel FreeBSD/AMD FreeBSD/Intel Capturing Rate [%] CPU usage [%] 10 20 30 40 50 60 70 80 90 100 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 10 20 30 40 50 60 70 80 90 100 (32) no SMP, no HT, std. buffers, 1 app, no filter, no load

slide-14
SLIDE 14

Results Using multiple processors? Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 12 / 20

Single processor, 1st Set

Capturing Rate [%] CPU usage [%] Datarate [Mbit/s] Linux/AMD Linux/Intel FreeBSD/AMD FreeBSD/Intel Capturing Rate [%] CPU usage [%] 10 20 30 40 50 60 70 80 90 100 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 10 20 30 40 50 60 70 80 90 100 (32) no SMP, no HT, std. buffers, 1 app, no filter, no load

Opteron/FreeBSD system performs best Sharp decline at high data rates

slide-15
SLIDE 15

Results Using multiple processors? Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 13 / 20

Multiprocessor (SMP), 1st Set

Capturing Rate [%] CPU usage [%] Datarate [Mbit/s] Linux/AMD Linux/Intel FreeBSD/AMD FreeBSD/Intel Capturing Rate [%] CPU usage [%] 10 20 30 40 50 60 70 80 90 100 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 10 20 30 40 50 60 70 80 90 100 (31) SMP, no HT, std. buffers, 1 app, no filter, no load

slide-16
SLIDE 16

Results Using multiple processors? Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 13 / 20

Multiprocessor (SMP), 1st Set

Capturing Rate [%] CPU usage [%] Datarate [Mbit/s] Linux/AMD Linux/Intel FreeBSD/AMD FreeBSD/Intel Capturing Rate [%] CPU usage [%] 10 20 30 40 50 60 70 80 90 100 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 10 20 30 40 50 60 70 80 90 100 (31) SMP, no HT, std. buffers, 1 app, no filter, no load

. . . even though the second processor is not used extensively All systems are benefitting . . .

slide-17
SLIDE 17

Results Increasing buffer sizes

Increasing Buffer Sizes ?

Setup:

  • First Set of systems
  • Dual processor
  • Increased buffer sizes

Operating system buffers: FreeBSD 6.x: sysctl’s net.bpf.bufsize and net.bpf.maxbufsize FreeBSD 5.x: sysctl’s debug.bpf bufsize and debug.maxbpf bufsize Linux: /proc/sys/net/core/rmem default, /proc/sys/net/core/rmem max, and /proc/sys/net/core/netdev max backlog

Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 14 / 20

slide-18
SLIDE 18

Results Increasing buffer sizes Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 15 / 20

increased buffers, 1st Set

Capturing Rate [%] CPU usage [%] Datarate [Mbit/s] Linux/AMD Linux/Intel FreeBSD/AMD FreeBSD/Intel Capturing Rate [%] CPU usage [%] 10 20 30 40 50 60 70 80 90 100 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 10 20 30 40 50 60 70 80 90 100 (17) SMP, no HT, inc. buffers, 1 app, no filter, no load

slide-19
SLIDE 19

Results Increasing buffer sizes Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 15 / 20

increased buffers, 1st Set

Capturing Rate [%] CPU usage [%] Datarate [Mbit/s] Linux/AMD Linux/Intel FreeBSD/AMD FreeBSD/Intel Capturing Rate [%] CPU usage [%] 10 20 30 40 50 60 70 80 90 100 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 10 20 30 40 50 60 70 80 90 100 (17) SMP, no HT, inc. buffers, 1 app, no filter, no load

The capturing rate could be increased again

slide-20
SLIDE 20

Results Additional Insights (I)

Additional Insights

First set of measurements

  • Filtering is cheap with respect to its benefit (reduced packet

processing)

  • Running multiple capturing applications concurrently leads to bad

performance.

  • Measurement with additional compression show some advantage for

Intel Systems

  • Intel Hyperthreading does not change the performance
  • using the memory-map patch from Phil Woods (Linux only) does help

Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 16 / 20

slide-21
SLIDE 21

Results Write to disk

Writing packets to disk?

preliminary measurements have shown that

  • newer system do not lose any packet: with buffers, SMP, etc.
  • disk writing speed is not the bottleneck

Setup:

  • Newer systems: 2x dual core AMD systems

⇒ CPU usage: 25% correspond to one fully utilized processor

  • Increased buffer sizes
  • No filter
  • Linux vs. FreeBSD
  • 32bit vs. 64bit OS’es

Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 17 / 20

slide-22
SLIDE 22

Results Write to disk Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 18 / 20

Writing to disk, 2nd Set

Capturing Rate [%] CPU usage [%] Datarate [Mbit/s] 32bit FreeBSD/Opteron 64bit FreeBSD/Opteron 32bit Linux/Opteron 64bit Linux/Opteron Capturing Rate [%] CPU usage [%] 10 20 30 40 50 60 70 80 90 100 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 10 20 30 40 50 60 70 80 90 100 (2-8) SMP, no HT, inc. buffers, 1 app, no filter, writing to disk

slide-23
SLIDE 23

Results Write to disk Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 18 / 20

Writing to disk, 2nd Set

Capturing Rate [%] CPU usage [%] Datarate [Mbit/s] 32bit FreeBSD/Opteron 64bit FreeBSD/Opteron 32bit Linux/Opteron 64bit Linux/Opteron Capturing Rate [%] CPU usage [%] 10 20 30 40 50 60 70 80 90 100 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 10 20 30 40 50 60 70 80 90 100 (2-8) SMP, no HT, inc. buffers, 1 app, no filter, writing to disk

Feasible up to 600-700 Mbit/s!

slide-24
SLIDE 24

Results Additional Insights (II)

Additional Insights

Second set of measurements

  • additional load (copying the packets in memory) shows significantly

better performance for FreeBSD

  • 64bit systems drop more packets
  • Using 4 cores (2x Dual Core) is slightly better than 2 cores (1x Dual

Core)

Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 19 / 20

slide-25
SLIDE 25

Summary

Summary

  • Split up 10-Gigabit on multiple 1-Gigabit monitoring systems
  • FreeBSD/AMD Opteron combination in general performs best
  • Utilizing multiple processors proves to be benefitting
  • Choosing large enough buffer size is important
  • Capturing full traces to disk is feasible up to about 600-700 Mbit/s

For further information see: High Performance Packet Capture http://www.net.t-labs.tu-berlin.de/research/hppc/

Schneider, Wallerich, Feldmann (TU Berlin/DT Labs) Packet Capturing on 10-Gigabit Ethernet Links PAM 2007 20 / 20