goProbe: A Scalable Distributed Network Monitoring Solution - PowerPoint PPT Presentation

goProbe: A Scalable Distributed Network Monitoring Solution Christian Decker Lennart Elsen Fabian Kohn Roger Wattenhofer

Goal Enable quick and efficient retrieval of key pieces of information about traffic patterns in global networks

Goal Enable quick and efficient retrieval of key pieces of information about traffic patterns in global networks Scalability

? ? ? Debugging/Operations Reporting

Acquisition of Traffic Data Packet Capture Storage

Acquisition of Traffic Data Grouping Information Reduction Packet Capture Storage

NetFlow NetFlow Packet Field N Length Source IP Network Packet aggregation by set of Destination IP Field N Type Next Layer Protocol … shared attributes IPv4/6 Next Hop … … Field 2 Length Transport Source Port Field 2 Type Network packet headers & Destination Port Field 1 Length … packet counters Field 1 Type Count Packet Size Number of Packets Meta Info … Sampling Interval Expiry time System Uptime TTL Sequence # Interface Name … NetFlow Version

NetFlow NetFlow Packet Field N Length Source IP Network Destination IP NetFlow Field N Type Exporter Next Layer Protocol … IPv4/6 Next Hop … … Field 2 Length Transport Source Port Field 2 Type NetFlow Destination Port Exporter Field 1 Length … Field 1 Type Count Network A Packet Size Number of Packets Meta Info … Sampling Interval System Uptime TTL Sequence # Interface Name … NetFlow Version Network B NetFlow Collector

NetFlow NetFlow Packet Field N Length Source IP Network Destination IP NetFlow Field N Type Exporter Next Layer Protocol … IPv4/6 Next Hop … … Field 2 Length Transport Source Port Field 2 Type NetFlow Destination Port Exporter Field 1 Length … Field 1 Type Count Network A Packet Size Number of Packets Meta Info … Sampling Interval System Uptime TTL Sequence # Interface Name … NetFlow Version Network B NetFlow Collector

Current Network Monitoring System Analysts Request Traffic d Metadata Aggregated Formatted Results Results Query Flow Exporter Tool Data DB nProbe FastBit Queries Single Host

Challenges Capturing Process Query nProbe Tool FastBit

Challenges Capturing Process Immense memory footprint Query nProbe Tool FastBit

Challenges Capturing Process One process per capture interface Query nProbe nProbe nProbe Tool FastBit

Challenges Storage Backend Query nProbe nProbe nProbe Tool FastBit

Challenges Storage Backend Inefficient memory management Query nProbe nProbe nProbe Tool FastBit

Challenges Storage Backend No data compression Query nProbe nProbe nProbe Tool FastBit

Challenges Storage Backend Long query execution times Query nProbe nProbe nProbe Tool FastBit

Challenges y t i l i b a l a c S r o o P Query nProbe nProbe nProbe Tool FastBit

Reduced Flow Format IP Packets Packets Bytes Bytes Src IP Dst IP Src Port Dst Port Protocol Rcvd Sent Rcvd Sent Shared Attributes Counters

Reduced Flow Format Deep Packet Inspection Appl. Src Port Dst Port Layer Protocol Shared Attributes Counters

Reduced Flow Format Deep Packet Inspection Appl. Src Port Dst Port Layer Protocol Flow in goProbe Source Port Aggregation ✗ Appl. Layer Dst Port Protocol Stored Flow

Collection of Flow Information — goProbe Written in Google Go One capture routine per interface goProbe Packet capture using modified libpcap Database flush in regular intervals

goProbe – Concept (Multiple Interfaces) Flow Table Interface Data Channel Timer Data Aggregation Prepare … DB Local Database