Testing applications with traffic control In containers Alban - - PowerPoint PPT Presentation

In containers

Alban Crequy

Testing applications with traffic control

ContainerCon North America - August 2016 https://goo.gl/ZLLlv3

Alban Crequy

∘ Worked on rkt the last 1.5 years ∘ Currently tech lead on rkt ∘ In 2014, worked on traffic control for multimedia applications in cars (tcmmd)

https://github.com/alban

Berlin-based software company building foundational Linux technologies

Some examples of what we work on...

OSTree

git for operating system binaries

Find out more about us… Blog: http://kinvolk.io/blog Github: https://github.com/kinvolk Twitter: https://twitter.com/kinvolkio Email: hello@kinvolk.io

∘ What is traffic control and how does it work on Linux ∘ How it can be used for testing a microservices application ∘ Demo ∘ With CoreOS Linux, Kubernetes, Weave Scope

Plan

What is traffic control? How does it work on Linux?

Traffic control, why?

web server client client client

THE INTERNET

∘ fair distribution

• f bandwidth

∘ reserve bandwidth to specific applications ∘ avoid bufferbloat

∘ Network scheduling algorithm ∘ which packet to emit next? ∘ when? ∘ Configurable at run-time: ∘ /sbin/tc ∘ Netlink ∘ Default on new network interfaces: sysctl net.core.default_qdisc

Queuing disciplines (qdisc)

eth0

THE INTERNET qdisc

Stochastic Fairness Queueing (sfq)

eth0

THE INTERNET FIFO n FIFO 1 FIFO 0

...

round robin

Traffic control for testing?

Network emulator (netem)

eth0

THE INTERNET netem

bandwidth latency packet loss corrupt ...

Testing with containers

container 1 container 2

eth0 eth0 Testing framework configure “netem” qdiscs: bandwidth, latency, packet drop...

The demo application

microservices-demo

https://github.com/microservices-demo/microservices-demo

Some micro-services

front-end Firefox catalogue

• rders
• rders-db

payment

Kubernetes

Kubernetes objects

• Kubernetes “Pods”
• Group of container(s) running together
• Kubernetes “Replication Sets”
• Control the number of “Pods” replicas running
• Kubernetes “Service”
• Directing the traffic to “Pods”

pod pod pod service

Testing with traffic control in Kubernetes

Kubernetes node 1 pod pod Kubernetes node 2 pod pod tc tc

controls ∘ Latency ∘ Bandwidth ∘ Packet drop

∘ configure network simulator ∘ play scenarios

Weave Scope

Weave Scope

Testing with Weave Scope

Kubernetes node 1 tc Scope Probe pod pod pod pod Kubernetes Node 2 tc Scope Probe pod pod pod pod Scope App

Demo

Reproduce the demo yourself: https://github.com/kinvolk/demo

Plugins in Scope

tc plugin Scope Probe Scope App report report control control

• Unix socket in /var/run/scope/plugins/
• Protocols: report and control
• Write your own plugins for your testing needs

Testing framework for web apps

Selenium

Testing more complex scenarios (my “wishlist”)

Add latency on a specific connection

front-end Firefox catalogue

• rders
• rders-db

payment

latency=100ms

How to define classes of traffic

eth0 netem

interface latency=100ms

dest_ip=10.0.4.* dest_ip=10.0.5.*

• ther

u32: filter on content

eth0 HTB HTB HTB HTB HTB netem netem netem

interface root qdisc (type = HTB) root class (type = HTB) leaf qdiscs (type = netem) leaf classes (type = HTB) filters (type=u32)

• ther

ip=10.0.5.* ip=10.0.4.* latency=10ms

Filtering with cBPF/eBPF

eth0 BPF netem netem

kernel userspace BPF_JMP... BPF_LD... BPF_RET... if (skb->protocol…) return TC_H_MAKE(TC_H_ROOT, mark); compilation clang... -march=bpf upload in the kernel:

• bpf()
• Netlink

x86_64 code JIT compilation

eBPF maps

eth0 BPF netem netem

kernel userspace x86_64 code eBPF map

tc ∘ Build statistics ∘ Make them available to the testing framework

The End

Try the demos yourself: https://github.com/kinvolk/demo Read more: https://kinvolk.io/blog/ The slides: https://goo.gl/ZLLlv3

Questions?