Can We Containerize Internet Measurements? Chris Misa (University - PowerPoint PPT Presentation

Can We Containerize Internet Measurements? Chris Misa (University of Oregon) Sudarsun Kannan (Rutgers University) Ramakrishnan Durairajan (University of Oregon) cmisa@cs.uoregon.edu 1

Outline • Containerized measurement issues • Proposed solutjon: MACE • Evaluatjon of MACE cmisa@cs.uoregon.edu 2

Containers • Lightweight virtualizatjon mechanism – Package, deploy, isolate cmisa@cs.uoregon.edu 3

Containers • Lightweight virtualizatjon mechanism – Package, deploy, isolate • Based on recent developments in Linux – Namespaces, cgroups cmisa@cs.uoregon.edu 4

Containers • Lightweight virtualizatjon mechanism – Package, deploy, isolate • Based on recent developments in Linux – Namespaces, cgroups • Rapidly replacing VMs – Smaller, faster cmisa@cs.uoregon.edu 5

Motivation • Streamline experiments – Package scripts, tools, libraries – Consistent interface cmisa@cs.uoregon.edu 6

Motivation • Streamline experiments – Package scripts, tools, libraries – Consistent interface • Expose new, cloud-natjve vantage points – Azure – AWS – GCP – etc. cmisa@cs.uoregon.edu 7

Motivation • Streamline experiments – Package scripts, tools, libraries – Consistent interface • Expose new, cloud-natjve vantage points – Azure – AWS – GCP – etc. • Less CPU and memory overheads than VMs [1] PlanetLab since 2012 [0] cmisa@cs.uoregon.edu 8

Sure we can! cmisa@cs.uoregon.edu 9

Sure we can! Why not? cmisa@cs.uoregon.edu 10

Network Isolation ● Extra latency [2] – ~50μs in restjng system cmisa@cs.uoregon.edu 11

Network Isolation ● Extra latency [2] – ~50μs in restjng system ● Co-located containers – Up to 300μs depending on traffjc cmisa@cs.uoregon.edu 12

Network Isolation ● Extra latency [2] – ~50μs in restjng system ● Co-located containers – Up to 300μs depending on traffjc ● Biased measurement results – Non-constant latency overheads cmisa@cs.uoregon.edu 13

Network Isolation ● Extra latency [2] – ~50μs in restjng system ● Co-located containers – Up to 300μs depending on traffjc ● Biased measurement results – Non-constant latency overheads ● Slim [3], FreeFlow [4] don’t help – Flow-based, RDMA cmisa@cs.uoregon.edu 14

Importance of Latency • An error of 300μs translates to – 90km at the speed of light [6, 7] – $1.2 million for online trading [5] Source: Google cmisa@cs.uoregon.edu 15

Importance of Latency • An error of 300μs translates to – 90km at the speed of light [6, 7] – $1.2 million for online trading [5] • Hard to isolate latencies – OS, virtualizatjon, physical Source: Google cmisa@cs.uoregon.edu 16

How to account for latency in a running container system? cmisa@cs.uoregon.edu 17

How to account for latency in a running container system? MACE: Measure the Added Container Expense cmisa@cs.uoregon.edu 18

Outline • Containerized measurement issues • Proposed solutjon: MACE • Evaluatjon of MACE cmisa@cs.uoregon.edu 19

MACE: Goals Host Container • Packet-level latencies – Ingress and egress eth0: 172.17.0.2 – High accuracy veth ? docker0 eth0: 198.168.0.2 cmisa@cs.uoregon.edu 20

MACE: Goals Host Container • Packet-level latencies – Ingress and egress eth0: 172.17.0.2 – High accuracy veth • Minimal impact on network performance ? docker0 eth0: 198.168.0.2 cmisa@cs.uoregon.edu 21

MACE: Goals Host Container • Packet-level latencies – Ingress and egress eth0: 172.17.0.2 – High accuracy veth • Minimal impact on network performance ? • Consistent, container-friendly interface docker0 eth0: 198.168.0.2 cmisa@cs.uoregon.edu 22

MACE: How? • Linux Kernel Tracepoints [9] – Hooks into kernel – Net device and system call subsytems Source: http://www.brendangregg.com cmisa@cs.uoregon.edu 23

MACE: How? • Linux Kernel Tracepoints [9] – Hooks into kernel – Net device and system call subsytems • Existjng tracers – Large perturbatjon Source: http://www.brendangregg.com cmisa@cs.uoregon.edu 24

MACE: How? • Linux Kernel Tracepoints [9] – Hooks into kernel – Net device and system call subsytems • Existjng tracers – Large perturbatjon • Kernel module – For container hosts Source: http://www.brendangregg.com – Report to containers cmisa@cs.uoregon.edu 25

MACE: Design ● Filter trace events – Interface – Namespace cmisa@cs.uoregon.edu 26

MACE: Design ● Filter trace events – Interface – Namespace ● Correlate events in hash tables – Ingress – Egress cmisa@cs.uoregon.edu 27

MACE: Design ● Filter trace events – Interface – Namespace ● Correlate events in hash tables – Ingress – Egress ● Maintain list of latencies – Report via device fjle cmisa@cs.uoregon.edu 28

MACE: Implementation ● High accuracy Open source at: github.com/chris-misa/mace – Read tsc for tjming cmisa@cs.uoregon.edu 29

MACE: Implementation ● High accuracy Open source at: github.com/chris-misa/mace – Read tsc for tjming ● Low perturbatjon – Only lock hash buckets – Atomic types for ring bufger cmisa@cs.uoregon.edu 30

MACE: Implementation ● High accuracy Open source at: github.com/chris-misa/mace – Read tsc for tjming ● Low perturbatjon – Only lock hash buckets – Atomic types for ring bufger ● Consistent API – Interface is namespace-aware – Allow and enable per container cmisa@cs.uoregon.edu 31

MACE: Interface ● Select the container’s namespace: # echo 1 > sys/class/mace/on cmisa@cs.uoregon.edu 32

MACE: Interface ● Select the container’s namespace: # echo 1 > sys/class/mace/on ● Execute measurement: # ping -c 10 google.com cmisa@cs.uoregon.edu 33

MACE: Interface ● Select the container’s namespace: # echo 1 > sys/class/mace/on ● Execute measurement: # ping -c 10 google.com ● Collect latencies: # cat dev / mace [1552589043.315681] (1) egress: 80932 [1552589043.315937] (1) ingress: 46208 [1552589043.316012] (2) egress: 13699 . . . cmisa@cs.uoregon.edu 34

How do we know those numbers are correct? cmisa@cs.uoregon.edu 35

Outline • Containerized measurement issues • Proposed solutjon: MACE • Evaluatjon of MACE cmisa@cs.uoregon.edu 36

Evaluation: Methodology • No direct method cmisa@cs.uoregon.edu 37

Evaluation: Methodology • No direct method Host Target • Use difgerence in RTT Container cmisa@cs.uoregon.edu 38

Evaluation: Methodology • No direct method Host Target (1) • Use difgerence in RTT Container (1) RTT from container cmisa@cs.uoregon.edu 39

Evaluation: Methodology • No direct method Host Target (1) • Use difgerence in RTT Container (2) (1) RTT from container (2) Latency overheads from MACE cmisa@cs.uoregon.edu 40

Evaluation: Methodology • No direct method Host Target (1) • Use difgerence in RTT Container (2) (3) (1) RTT from container (2) Latency overheads from MACE (3) ‘corrected’ RTT = (1) minus (2) cmisa@cs.uoregon.edu 41

Evaluation: Methodology • No direct method Host Target (1) • Use difgerence in RTT Container (2) (3) (1) RTT from container (2) Latency overheads from MACE (4) (3) ‘corrected’ RTT = (1) minus (2) Ground-truth hardware RTT (4) Compare with RTT measured from hardware cmisa@cs.uoregon.edu 42

Evaluation: Setting • Ping across single physical link – Minimize network latency cmisa@cs.uoregon.edu 43

Evaluation: Setting • Ping across single physical link – Minimize network latency • Add co-located containers – Flood ping – Worst-case traffjc settjng cmisa@cs.uoregon.edu 44

Evaluation: Setting • Ping across single physical link – Minimize network latency • Add co-located containers – Flood ping – Worst-case traffjc settjng • Run on Cloudlab [10] – Some RTT noise from experiment network cmisa@cs.uoregon.edu 45

Results: RTT Bias ● Reported RTT - actual RTT – ‘raw’ container (blue) – ‘corrected’ container (black) cmisa@cs.uoregon.edu 46

Results: RTT Bias ● Reported RTT - actual RTT – ‘raw’ container (blue) – ‘corrected’ container (black) ● MACE-corrected RTT is within 20μs in worst case cmisa@cs.uoregon.edu 47

Results: RTT Bias ● Reported RTT - actual RTT – ‘raw’ container (blue) – ‘corrected’ container (black) ● MACE-corrected RTT is within 20μs in worst case ● Traffjc impacts all sofuware RTTs – Up to 100 μs cmisa@cs.uoregon.edu 48

Results: Coverage ● Latency reports / packets (%) cmisa@cs.uoregon.edu 49

Results: Coverage ● Latency reports / packets (%) ● Decrease due to collisions in hash tables cmisa@cs.uoregon.edu 50

Results: Coverage ● Latency reports / packets (%) ● Decrease due to collisions in hash tables ● Increased table size can improve coverage to 100% cmisa@cs.uoregon.edu 51

Results: Perturbation ● Instrumented RTT minus non-instrumented RTT – MACE (black) – Ftrace (blue) cmisa@cs.uoregon.edu 52

Results: Perturbation ● Instrumented RTT minus non-instrumented RTT – MACE (black) – Ftrace (blue) ● MACE scales well as traffjc increases cmisa@cs.uoregon.edu 53