Can We Containerize Internet Measurements? Chris Misa (University - - PowerPoint PPT Presentation

▶
can we containerize internet measurements
SMART_READER_LITE
LIVE PREVIEW

Can We Containerize Internet Measurements? Chris Misa (University - - PowerPoint PPT Presentation

Apr 26, 2023 393 likes •1.02k views

Can We Containerize Internet Measurements? Chris Misa (University of Oregon) Sudarsun Kannan (Rutgers University) Ramakrishnan Durairajan (University of Oregon) cmisa@cs.uoregon.edu 1 Outline Containerized measurement issues Proposed

slide-1
SLIDE 1

cmisa@cs.uoregon.edu

1

Can We Containerize Internet Measurements?

Chris Misa (University of Oregon) Sudarsun Kannan (Rutgers University) Ramakrishnan Durairajan (University of Oregon)

slide-2
SLIDE 2

cmisa@cs.uoregon.edu

2

Outline

  • Containerized measurement issues
  • Proposed solutjon: MACE
  • Evaluatjon of MACE
slide-3
SLIDE 3

cmisa@cs.uoregon.edu

3

Containers

  • Lightweight virtualizatjon mechanism

– Package, deploy, isolate

slide-4
SLIDE 4

cmisa@cs.uoregon.edu

4

Containers

  • Lightweight virtualizatjon mechanism

– Package, deploy, isolate

  • Based on recent developments in Linux

– Namespaces, cgroups

slide-5
SLIDE 5

cmisa@cs.uoregon.edu

5

Containers

  • Lightweight virtualizatjon mechanism

– Package, deploy, isolate

  • Based on recent developments in Linux

– Namespaces, cgroups

  • Rapidly replacing VMs

– Smaller, faster

slide-6
SLIDE 6

cmisa@cs.uoregon.edu

6

Motivation

  • Streamline experiments

– Package scripts, tools, libraries – Consistent interface

slide-7
SLIDE 7

cmisa@cs.uoregon.edu

7

Motivation

  • Streamline experiments

– Package scripts, tools, libraries – Consistent interface

  • Expose new, cloud-natjve vantage points

– Azure – AWS – GCP – etc.

slide-8
SLIDE 8

cmisa@cs.uoregon.edu

8

Motivation

  • Streamline experiments

– Package scripts, tools, libraries – Consistent interface

  • Expose new, cloud-natjve vantage points

– Azure – AWS – GCP – etc.

  • Less CPU and memory overheads than VMs [1]

PlanetLab since 2012 [0]

slide-9
SLIDE 9

cmisa@cs.uoregon.edu

9

Sure we can!

slide-10
SLIDE 10

cmisa@cs.uoregon.edu

10

Sure we can! Why not?

slide-11
SLIDE 11

cmisa@cs.uoregon.edu

11

Network Isolation

  • Extra latency [2]

– ~50μs in restjng system

slide-12
SLIDE 12

cmisa@cs.uoregon.edu

12

Network Isolation

  • Extra latency [2]

– ~50μs in restjng system

  • Co-located containers

– Up to 300μs depending on traffjc

slide-13
SLIDE 13

cmisa@cs.uoregon.edu

13

Network Isolation

  • Extra latency [2]

– ~50μs in restjng system

  • Co-located containers

– Up to 300μs depending on traffjc

  • Biased measurement results

– Non-constant latency overheads

slide-14
SLIDE 14

cmisa@cs.uoregon.edu

14

Network Isolation

  • Extra latency [2]

– ~50μs in restjng system

  • Co-located containers

– Up to 300μs depending on traffjc

  • Biased measurement results

– Non-constant latency overheads

  • Slim [3], FreeFlow [4] don’t help

– Flow-based, RDMA

slide-15
SLIDE 15

cmisa@cs.uoregon.edu

15

Importance of Latency

  • An error of 300μs translates to

– 90km at the speed of light [6, 7] – $1.2 million for online trading [5]

Source: Google

slide-16
SLIDE 16

cmisa@cs.uoregon.edu

16

Importance of Latency

  • An error of 300μs translates to

– 90km at the speed of light [6, 7] – $1.2 million for online trading [5]

  • Hard to isolate latencies

– OS, virtualizatjon, physical

Source: Google

slide-17
SLIDE 17

cmisa@cs.uoregon.edu

17

How to account for latency in a running container system?

slide-18
SLIDE 18

cmisa@cs.uoregon.edu

18

How to account for latency in a running container system? MACE: Measure the Added Container Expense

slide-19
SLIDE 19

cmisa@cs.uoregon.edu

19

Outline

  • Containerized measurement issues
  • Proposed solutjon: MACE
  • Evaluatjon of MACE
slide-20
SLIDE 20

cmisa@cs.uoregon.edu

20

MACE: Goals

  • Packet-level latencies

– Ingress and egress – High accuracy

Host Container eth0: 172.17.0.2 veth docker0 eth0: 198.168.0.2

?

slide-21
SLIDE 21

cmisa@cs.uoregon.edu

21

MACE: Goals

  • Packet-level latencies

– Ingress and egress – High accuracy

  • Minimal impact on network performance

Host Container eth0: 172.17.0.2 veth docker0 eth0: 198.168.0.2

?

slide-22
SLIDE 22

cmisa@cs.uoregon.edu

22

MACE: Goals

  • Packet-level latencies

– Ingress and egress – High accuracy

  • Minimal impact on network performance
  • Consistent, container-friendly interface

Host Container eth0: 172.17.0.2 veth docker0 eth0: 198.168.0.2

?

slide-23
SLIDE 23

cmisa@cs.uoregon.edu

23

MACE: How?

  • Linux Kernel Tracepoints [9]

– Hooks into kernel – Net device and system

call subsytems

Source: http://www.brendangregg.com

slide-24
SLIDE 24

cmisa@cs.uoregon.edu

24

MACE: How?

  • Linux Kernel Tracepoints [9]

– Hooks into kernel – Net device and system

call subsytems

  • Existjng tracers

– Large perturbatjon

Source: http://www.brendangregg.com

slide-25
SLIDE 25

cmisa@cs.uoregon.edu

25

MACE: How?

  • Linux Kernel Tracepoints [9]

– Hooks into kernel – Net device and system

call subsytems

  • Existjng tracers

– Large perturbatjon

  • Kernel module

– For container hosts – Report to containers

Source: http://www.brendangregg.com

slide-26
SLIDE 26

cmisa@cs.uoregon.edu

26

MACE: Design

  • Filter trace events

– Interface – Namespace

slide-27
SLIDE 27

cmisa@cs.uoregon.edu

27

MACE: Design

  • Filter trace events

– Interface – Namespace

  • Correlate events in hash tables

– Ingress – Egress

slide-28
SLIDE 28

cmisa@cs.uoregon.edu

28

MACE: Design

  • Filter trace events

– Interface – Namespace

  • Correlate events in hash tables

– Ingress – Egress

  • Maintain list of latencies

– Report via device fjle

slide-29
SLIDE 29

cmisa@cs.uoregon.edu

29

MACE: Implementation

  • High accuracy

– Read tsc for tjming

Open source at: github.com/chris-misa/mace

slide-30
SLIDE 30

cmisa@cs.uoregon.edu

30

MACE: Implementation

  • High accuracy

– Read tsc for tjming

  • Low perturbatjon

– Only lock hash buckets – Atomic types for ring bufger

Open source at: github.com/chris-misa/mace

slide-31
SLIDE 31

cmisa@cs.uoregon.edu

31

MACE: Implementation

  • High accuracy

– Read tsc for tjming

  • Low perturbatjon

– Only lock hash buckets – Atomic types for ring bufger

  • Consistent API

– Interface is namespace-aware – Allow and enable per container

Open source at: github.com/chris-misa/mace

slide-32
SLIDE 32

cmisa@cs.uoregon.edu

32

MACE: Interface

  • Select the container’s namespace:

# echo 1 > sys/class/mace/on

slide-33
SLIDE 33

cmisa@cs.uoregon.edu

33

MACE: Interface

  • Select the container’s namespace:

# echo 1 > sys/class/mace/on

  • Execute measurement:

# ping -c 10 google.com

slide-34
SLIDE 34

cmisa@cs.uoregon.edu

34

MACE: Interface

  • Select the container’s namespace:

# echo 1 > sys/class/mace/on

  • Execute measurement:

# ping -c 10 google.com

  • Collect latencies:

# cat dev/mace [1552589043.315681] (1) egress: 80932 [1552589043.315937] (1) ingress: 46208 [1552589043.316012] (2) egress: 13699

. . .

slide-35
SLIDE 35

cmisa@cs.uoregon.edu

35

How do we know those numbers are correct?

slide-36
SLIDE 36

cmisa@cs.uoregon.edu

36

Outline

  • Containerized measurement issues
  • Proposed solutjon: MACE
  • Evaluatjon of MACE
slide-37
SLIDE 37

cmisa@cs.uoregon.edu

37

Evaluation: Methodology

  • No direct method
slide-38
SLIDE 38

cmisa@cs.uoregon.edu

38

Evaluation: Methodology

  • No direct method
  • Use difgerence in RTT

Container Target Host

slide-39
SLIDE 39

cmisa@cs.uoregon.edu

39

Evaluation: Methodology

  • No direct method
  • Use difgerence in RTT

(1) RTT from container

Container Target Host

(1)

slide-40
SLIDE 40

cmisa@cs.uoregon.edu

40

Evaluation: Methodology

  • No direct method
  • Use difgerence in RTT

(1) RTT from container (2) Latency overheads from MACE

Container Target Host

(1) (2)

slide-41
SLIDE 41

cmisa@cs.uoregon.edu

41

Evaluation: Methodology

  • No direct method
  • Use difgerence in RTT

(1) RTT from container (2) Latency overheads from MACE (3) ‘corrected’ RTT = (1) minus (2)

Container Target Host

(1) (2) (3)

slide-42
SLIDE 42

cmisa@cs.uoregon.edu

42

Evaluation: Methodology

  • No direct method
  • Use difgerence in RTT

(1) RTT from container (2) Latency overheads from MACE (3) ‘corrected’ RTT = (1) minus (2) (4) Compare with RTT measured from hardware

Container Target Host Ground-truth hardware RTT

(1) (2) (4) (3)

slide-43
SLIDE 43

cmisa@cs.uoregon.edu

43

Evaluation: Setting

  • Ping across single physical link

– Minimize network latency

slide-44
SLIDE 44

cmisa@cs.uoregon.edu

44

Evaluation: Setting

  • Ping across single physical link

– Minimize network latency

  • Add co-located containers

– Flood ping – Worst-case traffjc settjng

slide-45
SLIDE 45

cmisa@cs.uoregon.edu

45

Evaluation: Setting

  • Ping across single physical link

– Minimize network latency

  • Add co-located containers

– Flood ping – Worst-case traffjc settjng

  • Run on Cloudlab [10]

– Some RTT noise from

experiment network

slide-46
SLIDE 46

46

Results: RTT Bias

  • Reported RTT - actual RTT

– ‘raw’ container (blue) – ‘corrected’ container (black)

cmisa@cs.uoregon.edu

slide-47
SLIDE 47

47

Results: RTT Bias

  • Reported RTT - actual RTT

– ‘raw’ container (blue) – ‘corrected’ container (black)

  • MACE-corrected RTT is within

20μs in worst case

cmisa@cs.uoregon.edu

slide-48
SLIDE 48

48

Results: RTT Bias

  • Reported RTT - actual RTT

– ‘raw’ container (blue) – ‘corrected’ container (black)

  • MACE-corrected RTT is within

20μs in worst case

  • Traffjc impacts all sofuware RTTs

– Up to 100 μs

cmisa@cs.uoregon.edu

slide-49
SLIDE 49

49

Results: Coverage

  • Latency reports / packets (%)

cmisa@cs.uoregon.edu

slide-50
SLIDE 50

50

Results: Coverage

  • Latency reports / packets (%)
  • Decrease due to collisions in

hash tables

cmisa@cs.uoregon.edu

slide-51
SLIDE 51

51

Results: Coverage

  • Latency reports / packets (%)
  • Decrease due to collisions in

hash tables

  • Increased table size can improve

coverage to 100%

cmisa@cs.uoregon.edu

slide-52
SLIDE 52

52

Results: Perturbation

  • Instrumented RTT

minus non-instrumented RTT

– MACE (black) – Ftrace (blue)

cmisa@cs.uoregon.edu

slide-53
SLIDE 53

53

Results: Perturbation

  • Instrumented RTT

minus non-instrumented RTT

– MACE (black) – Ftrace (blue)

  • MACE scales well as traffjc

increases

cmisa@cs.uoregon.edu

slide-54
SLIDE 54

cmisa@cs.uoregon.edu

54

Results: MACE Functions

  • Executjon tjme of MACE

functjons

– Tracepoint probes – Hash table management – Latency list management

slide-55
SLIDE 55

cmisa@cs.uoregon.edu

55

Results: MACE Functions

  • Executjon tjme of MACE

functjons

– Tracepoint probes – Hash table management – Latency list management

  • System call tracepoints are slow

– Accessing data in userspace – Needed for correlatjon

slide-56
SLIDE 56

cmisa@cs.uoregon.edu

56

Future Goals

  • Improving MACE

– Add TCP, UDP support – Hardware tjmestamps – Betuer in-fmight correlatjon – Ease of applicatjon-level correlatjon

slide-57
SLIDE 57

cmisa@cs.uoregon.edu

57

Future Goals

  • Improving MACE

– Add TCP, UDP support – Hardware tjmestamps – Betuer in-fmight correlatjon – Ease of applicatjon-level correlatjon

  • Applying MACE

– Improving measurement accuracy (e.g. geolocatjon) – Virtual network telemetry

slide-58
SLIDE 58

cmisa@cs.uoregon.edu

58

Summary

  • Containerized measurement issues
  • Proposed solutjon: MACE
  • Evaluatjon of MACE
slide-59
SLIDE 59

cmisa@cs.uoregon.edu

59

Thank You!

  • UO VPRI* and NSF
  • Anonymous reviewers
  • CloudLab team

* This work is supported by a fellowship from the University of Oregon Office of the Vice President for Research and Innovation.

slide-60
SLIDE 60

cmisa@cs.uoregon.edu

60

Questions?

slide-61
SLIDE 61

cmisa@cs.uoregon.edu

61

Citations

  • [0] htups://planet-lab.org/node/263, Sept. 2012 (accessed Feb. 2019)
  • [1] W. Felter et al., “An updated performance comparison of virtual machines and linux containers.” Proceedings of the IEEE Internatjonal

Symposium on Performance Analysis of Systems and Sofuware, 2015.

  • [2] Y. Zhao et al., “Performance of container networking technologies.” Proceedings of HotConNet 2017.
  • [3] D. Zhuo et al., “Slim: OS Kernel Support for a Low-Overhead Container Overlay Network.” NSDI 2019.
  • [4] D. Kim et al., “Freefmow: Sofuware-based Virtual RDMA Networking for Containerized Clouds.” NSDI 2019.
  • [5] N. Shalom and Y. Einav, “Amazon Found Every 100ms of Latency Cost Them 1% in Sales.” htup://goo.gl/BUJgV (accessed Feb. 2019).
  • [6] htups://www.britannica.com/science/speed-of-light (accessed Feb. 2019).
  • [7] A. Singla et al., “The Internet at the speed of light.” Proceedings of HotNets, October 2014.
  • [8] M. Mathis and M. Allman, “A Framework for Defjning Empirical Bulk Transfer Capacity Metrics.” RFC 3148, July 2001.
  • [9] M. Desnoyers, “Using the Linux Kernel Tracepoints.” htups://www.kernel.org/doc/Documentatjon/trace/tracepoints.txt (accessed Feb.

2019)

  • [10] R. Ricci et al.,“Introducing CloudLab: Scientjfjc infrastructure for advancing cloud architectures and applicatjons.” ;login:, 2014.