SLIDE 1
2nd European Zeek (Bro) Workshop Patented technologies
- Selective Packet Capture: Problem definition
- Optimizations
- Long queue emulation
- Lockless bimodal queues
- Tail early dropping
- LFN tables
- Multiresolution priority queues
- Zeek script
Selective Packet Capture at High Speed Rates Reservoir Labs Peter - - PowerPoint PPT Presentation
Selective Packet Capture at High Speed Rates Reservoir Labs Peter - - PowerPoint PPT Presentation
Selective Packet Capture at High Speed Rates Reservoir Labs Peter Cullen, James Ezick, Kelly Fox, Troy Hanson, Richard Lethin, Erik Mogus, Jordi Ros-Giralt, Alison Ryan, {cullen, ezick, fox, hanson, lethin, mogus, giralt, ryan}@reservoir.com
SLIDE 2
SLIDE 3
2nd European Zeek (Bro) Workshop Patented technologies
Packet Capturing at Very High Speed Rates
3
- Scalability issue: performing packet capture is either intractable or
requires highly expensive hardware both in processing and storage.
- Liability issue: indiscriminate packet capture poses a liability issue.
- Selective Packet Capture (SPC) provides a sweet-spot solution to both
- f these problems.
- SPC gets a "free lunch" by leveraging all the heavy lifting work done
by Zeek
SLIDE 4
2nd European Zeek (Bro) Workshop Patented technologies
Capturing packets at very high speed rate is an HPC problem… So let's talk first about performance optimization...
Problem Definition
4
SLIDE 5
2nd European Zeek (Bro) Workshop Patented technologies
- System wide performance optimization of network components like
routers, firewalls, or network analyzers such as a Zeek sensor is complex.
- Hundreds of different SW algorithms and data structures interrelated in
subtle ways.
- Two interdependent problems:
- Shifting micro-bottlenecks
- Nonlinear performance collapse
- Special focus on the problem of packet capturing at very high speed
rates Problem Definition
5
SLIDE 6
2nd European Zeek (Bro) Workshop Patented technologies
Problem Definition
6
Shifting Micro-Bottlenecks
It’s difficult...
SLIDE 7
2nd European Zeek (Bro) Workshop Patented technologies
Problem Definition
7
Shifting Micro-Bottlenecks
...to optimize...
SLIDE 8
2nd European Zeek (Bro) Workshop Patented technologies
Problem Definition
8
Shifting Micro-Bottlenecks
...bottlenecks...
SLIDE 9
2nd European Zeek (Bro) Workshop Patented technologies
Problem Definition
9
Shifting Micro-Bottlenecks
...that keep moving...
SLIDE 10
2nd European Zeek (Bro) Workshop Patented technologies
Problem Definition
10
Shifting Micro-Bottlenecks
...every microsecond...
SLIDE 11
2nd European Zeek (Bro) Workshop Patented technologies
Problem Definition
11
Shifting Micro-Bottlenecks
...or so.
SLIDE 12
2nd European Zeek (Bro) Workshop Patented technologies
Non-linear Performance Collapse
12
Net I/O
PCIE CPU Disk I/O Cache Memory
40Gbps 64 Gbps 1092 Gbps 56 GHz 10.4 Gbps L1-I cache: 896 kB L1-D cache: 896 kB L2 cache: 7168 kB L3 cache: 71680 kB
SLIDE 13
2nd European Zeek (Bro) Workshop Patented technologies
13
Net I/O
PCIE CPU Disk I/O Cache Memory
40Gbps 64 Gbps 1092 Gbps 56 GHz 10.4 Gbps L1-I cache: 896 kB L1-D cache: 896 kB L2 cache: 7168 kB L3 cache: 71680 kB
State 1: network is the bottleneck Healthy cache regime:
- CPU operates out of cache
- High cache hit ratios
Non-linear Performance Collapse
SLIDE 14
2nd European Zeek (Bro) Workshop Patented technologies
14
Net I/O
PCIE CPU Disk I/O Cache Memory
40Gbps 64 Gbps 1092 Gbps 56 GHz 10.4 Gbps L1-I cache: 896 kB L1-D cache: 896 kB L2 cache: 7168 kB L3 cache: 71680 kB
State 2: network is no longer the bottleneck Highly inefficient memory regime:
- CPU operates out of RAM
- High cache miss ratios
10x penalty
Non-linear Performance Collapse
SLIDE 15
2nd European Zeek (Bro) Workshop Patented technologies
15
Net I/O
PCIE CPU Disk I/O Cache Memory
40Gbps 64 Gbps 1092 Gbps 56 GHz 10.4 Gbps L1-I cache: 896 kB L1-D cache: 896 kB L2 cache: 7168 kB L3 cache: 71680 kB
State 2: network is no longer the bottleneck 10x penalty By removing the network bottleneck, system spends more time processing packets that will need to be dropped anyway → net performance degradation (performance collapse)
Non-linear Performance Collapse
Highly inefficient memory regime:
- CPU operates out of RAM
- High cache miss ratios
input
- utput
SLIDE 16
2nd European Zeek (Bro) Workshop Patented technologies
Performance Optimization: Algorithms and Data Structures
16
Long queue emulation Reduces packet drops due to fixed-size hardware rings Lockless bimodal queues Improves packet capturing performance Tail early dropping (TED) Increases information entropy and extracted metadata LFN tables Reduces state sharing overhead Multiresolution priority queues Reduces cost of processing timers
SLIDE 17
2nd European Zeek (Bro) Workshop Patented technologies
Performance Optimization: Algorithms and Data Structures
17
Long queue emulation Reduces packet drops due to fixed-size hardware rings Lockless bimodal queues Improves packet capturing performance Tail early dropping (TED) Increases information entropy and extracted metadata LFN tables Reduces state sharing overhead Multiresolution priority queues Reduces cost of processing timers
SLIDE 18
2nd European Zeek (Bro) Workshop Patented technologies
18
Dispatcher Model: Long queue emulation Model:
- Packet read cache penalty.
- Descriptor read cache penalty
- Packet drop penalty under certain
conditions
Long Queue Emulation
SLIDE 19
2nd European Zeek (Bro) Workshop Patented technologies
19
Long Queue Emulation: Operational Lemma
SLIDE 20
2nd European Zeek (Bro) Workshop Patented technologies
20
Use LQE
Long Queue Emulation in Practice
SLIDE 21
2nd European Zeek (Bro) Workshop Patented technologies
21
Long Queue Emulation
- Optimal LQE size
SLIDE 22
2nd European Zeek (Bro) Workshop Patented technologies
Performance Optimization: Algorithms and Data Structures
22
Long queue emulation Reduces packet drops due to fixed-size hardware rings Lockless bimodal queues Improves packet capturing performance Tail early dropping Increases information entropy and extracted metadata LFN tables Reduces state sharing overhead Multiresolution priority queues Reduces cost of processing timers
SLIDE 23
2nd European Zeek (Bro) Workshop Patented technologies
23
Lockless Bimodal Queues
- Goal: move packets from the memory ring to disk without using locks
(trigger capture)
SLIDE 24
2nd European Zeek (Bro) Workshop Patented technologies
24
- Goal: move packets from the memory ring to disk without using locks
Lockless Bimodal Queues
SLIDE 25
2nd European Zeek (Bro) Workshop Patented technologies
25
Lockless Bimodal Queues
SLIDE 26
2nd European Zeek (Bro) Workshop Patented technologies
26
Lockless Bimodal Queues
SLIDE 27
2nd European Zeek (Bro) Workshop Patented technologies
27
Lockless Bimodal Queues
SLIDE 28
2nd European Zeek (Bro) Workshop Patented technologies
28
SPC Workflow
SLIDE 29
2nd European Zeek (Bro) Workshop Patented technologies
29
Selective Packet Capture API
- The function spc_capture() takes two arguments as shown by its
function prototype:
## API for capturing a Pcap function spc_capture(prefix: string, filter: string);
- The prefix argument allows users to specify a prefix for the generated
Pcap file name. The filter argument can be used to specify a BPF filter applied to the captured packets as they are written to the pcap
- file. See https://www.tcpdump.org/manpages/pcap-filter.7.html for the
expression syntax of the BPF filter. If set to the empty string “”, all packets (without any filtering) are written to the Pcap file.
SLIDE 30
2nd European Zeek (Bro) Workshop Patented technologies
30
Selective Packet Capture by Example
SLIDE 31