Chair for Network Architectures and Services Technical University of Munich Component Model for Linux-based Network Systems Guided Research Dominik Scholz March 16, 2016 Chair for Network Architectures and Services Department of Informatics Technical University of Munich Dominik Scholz – Component Model for Linux-based Network Systems 1
Chair for Network Architectures and Services Technical University of Munich Motivation Profiling of Linux Network Stack Research Questions Linux Network Stack Routing Subsystem Netfilter Component Model Methodology Evaluation IPv4 & IPv6 Forwarding Performance NAT Conclusion iptables Dominik Scholz – Component Model for Linux-based Network Systems 2
Chair for Network Architectures and Services Technical University of Munich Motivation Software routers are faced with new challenges ◮ Standards beyond 10 GbE: 40 GbE and 100 GbE [1] ◮ Growing routing tables: > 550.000 rules for backbone routers [2] ◮ Deployment of IPv6 ◮ Multifaceted tasks (packet filter, NAT, ...) How can performance be analysed? Dominik Scholz – Component Model for Linux-based Network Systems 3
Chair for Network Architectures and Services Technical University of Munich Profiling of Linux Network Stack Network Stack ◮ Full data plane functionality ◮ Customisable Profiling using perf ◮ SW/HW counters (e.g. CPU cycles, cache misses) ◮ Use to understand processing and find potential bottlenecks ◮ Problem: hundreds of functions Dominik Scholz – Component Model for Linux-based Network Systems 4
Chair for Network Architectures and Services Technical University of Munich Visualised Call-Stack Dominik Scholz – Component Model for Linux-based Network Systems 5
Chair for Network Architectures and Services Technical University of Munich Research Questions ◮ How can profiling be used to analyse the performance of routers? ◮ How is the performance of IPv4 and IPv6 software routers? ◮ Can optional/additional tasks (packet filter, NAT, ...) be analysed with this method? Dominik Scholz – Component Model for Linux-based Network Systems 6
Chair for Network Architectures and Services Technical University of Munich Linux Network Stack – Forwarding conntrack iptable filter nat ipv 4 out NF Hooks PRE FWD POST ip rcv ip forward ip output Routing Subsystem NIC NIC IPv6: similar but separate path Dominik Scholz – Component Model for Linux-based Network Systems 7
Chair for Network Architectures and Services Technical University of Munich Components – Routing Subsystem Where to forward packet to? → Longest Prefix Matching Data Structure: FIB trie [3] ◮ Ordered prefix tree, IP used as key ◮ Lookup has O ( n ) complexity, n is length of key [4] → n is known → Independent of number of entries! Dominik Scholz – Component Model for Linux-based Network Systems 8
Chair for Network Architectures and Services Technical University of Munich Components – Netfilter Several hooks during processing path of packet → Forwarding: prerouting, forwarding, postrouting hooks Source NAT ◮ Connection tracking (prerouting) ◮ Packet modifications (postrouting) Dominik Scholz – Component Model for Linux-based Network Systems 9
Chair for Network Architectures and Services Technical University of Munich Component Model C ct C ipt C nat out C prer C r C postr C net Dominik Scholz – Component Model for Linux-based Network Systems 10
Chair for Network Architectures and Services Technical University of Munich Methodology Data generation ◮ Generate load (MoonGen) ◮ Measure RX and TX packet rate of router ◮ Profiling with perf ◮ Percentage distribution of CPU cycles across functions ◮ Total CPU cycles ◮ L1, L2, LLC cache misses Data processing ◮ Calculate cycles per function per packet ◮ Sort into groups (based on call-stack, source code) Problem: functions reused during different steps of processing Dominik Scholz – Component Model for Linux-based Network Systems 11
Chair for Network Architectures and Services Technical University of Munich Forwarding Performance – Parameterized Model 900 2500 1 flow 1 flow 800 2 flows 2 flows 64 flows 64 flows 2000 700 256 flows 256 flows CPU-cycles per packet CPU-cycles per packet 1k flows 1k flows 600 16k flows 16k flows 1500 500 100k flows 500k flows 500k flows 1mil flows 400 1mil flows 1000 300 200 500 100 0 0 C prer C ct C r C ipt C postr C nat _ out C prer C ct C r C ipt C postr C nat _ out (a) IPv4 (b) IPv6 → Only routing subsystem influences performance Dominik Scholz – Component Model for Linux-based Network Systems 12
Chair for Network Architectures and Services Technical University of Munich Forwarding Performance 10 0 10 1 10 2 10 3 10 4 10 5 10 6 10 0 10 1 10 2 10 3 10 4 10 5 10 6 20 20 Cycles Cycles 3 , 500 3 , 500 L1 cache misses L1 cache misses Cache Misses per Packet [-] Cache Misses per Packet [-] L2 cache misses L2 cache misses Cycles per Packet [-] Cycles per Packet [-] 15 15 L3 cache misses L3 cache misses 3 , 000 3 , 000 10 10 2 , 500 2 , 500 5 5 2 , 000 2 , 000 0 0 10 0 10 1 10 2 10 3 10 4 10 5 10 6 10 0 10 1 10 2 10 3 10 4 10 5 10 6 Flows [log] Flows [log] (a) IPv4 (b) IPv6 → Areas of approx. constant cost because of FIB trie ◮ Constant lookup cost ◮ Linear in regard to memory → cache sizes! Dominik Scholz – Component Model for Linux-based Network Systems 13
Chair for Network Architectures and Services Technical University of Munich NAT – IPv4 10 0 10 1 10 2 10 3 10 4 30 2000 7 , 000 Cycles Cache Misses per Packet [-] 1 flow 1k flows L1 cache misses 25 Cycles per Packet [-] 6 , 500 16k flows L2 cache misses 1500 32k flows CPU-cycles per packet L3 cache misses 20 6 , 000 5 , 500 15 1000 5 , 000 10 4 , 500 500 5 4 , 000 0 10 0 10 1 10 2 10 3 10 4 0 C prer C ct C r C ipt C postr C nat _ out Flows [log] (a) Profiling (b) Performance → approx. constant cost of 1900 cpp for up to 10000 rules Dominik Scholz – Component Model for Linux-based Network Systems 14
Chair for Network Architectures and Services Technical University of Munich Conclusion Methodology ◮ Difficult for complex scenarios, functions reused! → alternative approach: once with frame-pointers (performance loss!) to calibrate model ◮ Can be applied to other software routers → e.g. analyse different forwarding algorithms Forwarding with Linux Network Stack ◮ IPv6 significantly slower: longer addresses → caching ◮ ... but for instance no need for NAT Paper submitted to ITC Dominik Scholz – Component Model for Linux-based Network Systems 15
Chair for Network Architectures and Services Technical University of Munich Bibliography I [1] D. J. Law, A. Healey, P . Anslow, S. B. Carlson, and V. Maguire: IEEE 802.3bm-2015 (2015). [2] Huston, G.: BGP in 2015. In: labs.apnic.net (2016) [3] LC-trie implementation notes. Available: www.kernel.org/doc/Documentation/networking/fib_trie.txt [4] Bellini, A.C.: The Trie: A Neglected Data Structure. Available: www.toptal.com/java/the-trie-a-neglected-data-structure Dominik Scholz – Component Model for Linux-based Network Systems 16
Chair for Network Architectures and Services Technical University of Munich Thank you for your attention! Dominik Scholz – Component Model for Linux-based Network Systems 17
Chair for Network Architectures and Services Technical University of Munich iptables – IPv4, stateless 10 0 10 1 10 2 10 3 10 4 2000 10 3 Cache Misses per Packet [log] 2 rules Cycles 32 rules Cycles per Packet [log] 10 5 L1 cache misses 64 rules 1500 128 rules L2 cache misses CPU-cycles per packet 10 2 10 4 1000 10 3 10 1 10 2 500 10 0 10 1 10 0 10 1 10 2 10 3 10 4 0 C prer C ct C r C ipt C postr C nat _ out Stateless Rules [log] (a) Profiling (b) Performance Dominik Scholz – Component Model for Linux-based Network Systems 18
Recommend
More recommend
