A 10GE Monitoring System Arin Vijn arien@ams-ix.net Agenda - - PDF document

A 10GE Monitoring System Ariën Vijn arien@ams-ix.net

Agenda - Introduction The role of an internet exchange (IX). - The problem to be solved. Real life examples - The chosen solution for that problem * The Force10's P10 IDS/IPS card * Photonic cross connects

ISP A ISP B ISP D ISP C ISP E ISP F ISP G ISP H ISP I

ISP A ISP B ISP D ISP C Tier 1 ISP E ISP F Tier 2 ISP G ISP H ISP I Tier 3

AS2 AS1 AS3 AS4 switch park AS6 AS5

AS2 AS1 BGP BGP Router Router BGP BGP AS3 AS4 Router Router switch park BGP BGP Router Router AS6 AS5

AS2 AS1 BGP BGP Router Router BGP BGP AS3 AS4 Router Router switch park BGP BGP Router Router AS6 AS5

AS2 AS1 BGP BGP Router Router BGP BGP AS3 AS4 Router Router switch park BGP BGP Router Router AS6 AS5

AS2 AS1 BGP BGP Router Router BGP BGP AS3 AS4 Router Router switch park BGP BGP Router Router AS6 AS5

monitor AS2 AS1 BGP BGP Router Router BGP BGP AS3 AS4 Router Router switch park BGP BGP Router Router AS6 AS5

Example #1

UNI.. 1 M 1 AS1 M 1 UNI.. 2 AS2 switch park M 1 UNI.. 3 AS3

UNI.. 1 M 1 AS1 M 1 UNI.. 2 AS2 switch park M 1 UNI.. 3 AS3

F 2 ARP 1 AS1 Ethernet Destination: ff:ff:ff:ff:ff:ff Source: 00:00:00:00:00:01 Type: ARP (0x0806) Address Resolution Protocol (request) ARP 2 F Hardware type: Ethernet (0x0001) 2 AS2 Protocol type: IP (0x0800) Hardware size: 6 switch park Protocol size: 4 Opcode: request (0x0001) Sender MAC address: 0 1 :00:00:00:00:02 Sender IP address: 195.69.144.2 Target MAC address: 00:00:00:00:00:00 Target IP address: 195.69.144.3 F 2 ARP 3 AS3

F 2 ARP 1 AS1 Ethernet Destination: ff:ff:ff:ff:ff:ff Source: 00:00:00:00:00:01 Type: ARP (0x0806) Address Resolution Protocol (request) ARP 2 F Hardware type: Ethernet (0x0001) 2 AS2 Protocol type: IP (0x0800) Hardware size: 6 switch park Protocol size: 4 Opcode: request (0x0001) Sender MAC address: 0 1 :00:00:00:00:02 Sender IP address: 195.69.144.2 Target MAC address: 00:00:00:00:00:00 F 2 ARP Target IP address: 195.69.144.3 3 AS3

F 2 ARP 1 AS1 ARP 2 F 2 AS2 switch park F 2 ARP 3 AS3

Tuttle AS2 AS1 BGP BGP Router Router BGP BGP AS3 AS4 Router Router switch park BGP BGP Router Router AS6 AS5 Buttle

monitor AS2 AS1 BGP BGP Router Router BGP BGP AS3 AS4 Router Router switch park BGP BGP Router Router AS6 AS5

AS2 AS1 BGP BGP Router Router BGP BGP AS3 AS4 Router Router switch park BGP BGP Router Router AS6 monitor AS5

monitor AS2 AS1 BGP BGP Router Router BGP BGP AS3 AS4 Router Router switch park BGP BGP Router Router AS6 AS5

Example #2

transit AS 1 transit own netwerk BGP shared link router AS 1 switch AS 1 ixp IXP

BGP BGP router router AS 1 AS 2 shared shared link link private switch AS 1 switch AS 2 interconnect AS1 - AS2 ixp ixp AS1 AS2 IXP

1 D BGP 1 ROUTER AS1 switch park

S != 1 D BGP 1 ROUTER AS1 switch park syslog source mac

PAYL. S D BGP 1 ROUTER AS1 Ethernet Destination: 00:00:00:00:00:02 Source: 00:00:5e:00:01:11 (IETF-VRRP-virtual-router-VRID_11) IP (0x0800) Internet Protocol Protocol: ICMP (0x01) Header checksum Source: 10.0.0.1 Destination: 10.0.0.2 switch park Internet Control Message Protocol Type: 0 (Echo (ping) reply) Code: 0 [...]

Monitoring

S == 1 D BGP 1 ROUTER AS1 mirror port switch park NIC PCI-X General purpose PC (libpcap)

≤ 5Gbps optic MAC PP TM ≤ 5Gbps monitored port switch fabric optic MAC PP TM ≤ 5Gbps mirror port

Ethernet Router switch

Ethernet Router switch MEMS based Fiber Switch AKA Photonic Cross Connect (PXC)

Fiber Lens Array Fixed Mirror Dry Air Movable Mirror Array

Fiber Lens Array Fixed Mirror Dry Air Movable Mirror Array

core switch core switch science park Telecity metrofiber BGP router

core switch core switch science park Telecity metrofiber BGP router

AMS-IX Router switch XENPAK XENPAK traffic generator / analyser Anritsu MD1230A

frame n frame n+1 64 byte, 51.2 ns 9.6 ns 64 byte, 51.2 ns 9.6 ns

Solution

AMS-IX Router switch XPAK XPAK interface card PCI-X General Purpose Computer (Linux)

max. 14.8Mfps full duplex 10GE 10GE max. 10 Gbps full duplex Layer 1 repeater and tap Hardware max. 2 x 14.8 Mfps one way max. 2 x 10 Gbps one way Programmable Filter max. ~ 600kfps PCI-X interface PCI-X bus software driver eth2 'normal' ethernet interface

XPAK XPAK front-end flash FPGA JTAG back-end SRAM FPGA CPLD JTAG PCI

XPAK Transceivers ch1 ch2 XAUI XAUI AND AND 64 64 128 128 frame data frame data block ch1 block ch2 ch 2 ch1 back-end FPGA

127 111 95 79 63 47 31 15 0 preamble / SFD dest. MAC addr. src 0

preamble / SFD dest. MAC addr. src 0 127 111 95 79 63 47 31 15 0 4 H TTL pro MAC addr. type L TOS total len. ID F/F 1 to. HCS src. IP addr. dst. IP addr IP payload 2 IP payload n

front-end FPGA valid frame data offset clk frame data register R0 R1 Rn C0 C1 Cn reduction network match memory filtered data CPLD

127 111 95 79 63 47 31 15 0 preamble / SFD dest. MAC addr. src 0 MAC addr. type payload 1 payload n

127 111 95 79 63 47 31 15 0 preamble / SFD dest. MAC addr. src 0 Rx Cx reduction network

127 111 95 79 63 47 31 15 0 MAC addr. type payload 1 Ry Rz Cy Cz reduction network

front-end FPGA frame block offset valid data clk memory packet processor management user defined functionality addr. match data R/W memory capture filtered host interface data CPLD

front-end FPGA valid frame data offset clk frame data register R0 R1 Rn C0 C1 Cn reduction network counter host interface CPLD

front-end FPGA valid frame data offset clk frame data register R0 R1 Rn R0 R1 Rn C0 C1 Cn C0 C1 Cn reduction network reduction network counter host interface CPLD

random number frame counter generator register Cr reduction network

127 111 95 79 63 47 31 15 0 preamble / SFD dest. MAC addr. src 0 4 H TTL pro MAC addr. type L TOS total len. ID F/F 1 to. HCS src. IP addr. dst. IP addr IP payload 2 IP payload n

XPAK XPAK front-end flash FPGA JTAG back-end SRAM FPGA CPLD JTAG PCI

Thank you for your attention