Large-scale Virtualization in the Emulab Network Testbed Mike - - PowerPoint PPT Presentation

▶
large scale virtualization in the emulab network testbed
SMART_READER_LITE
LIVE PREVIEW

Large-scale Virtualization in the Emulab Network Testbed Mike - - PowerPoint PPT Presentation

Nov 04, 2023 166 likes •1.06k views

Large-scale Virtualization in the Emulab Network Testbed Mike Hibler, Robert Ricci, Leigh Stoller, Jonathon Duerig, Shashi Guruprasad, Tim Stack, Kirk Webb, Jay Lepreau Emulab: Network Testbed 2 Emulab: Network Testbed 2 Emulab: Network

slide-1
SLIDE 1

Large-scale Virtualization in the Emulab Network Testbed

Mike Hibler, Robert Ricci, Leigh Stoller, Jonathon Duerig, Shashi Guruprasad, Tim Stack, Kirk Webb, Jay Lepreau

slide-2
SLIDE 2

Emulab: Network Testbed

2

slide-3
SLIDE 3

Emulab: Network Testbed

2

slide-4
SLIDE 4

Emulab: Network Testbed

2

slide-5
SLIDE 5

Emulab: Network Testbed

2

slide-6
SLIDE 6

What’s Wrong?

3

slide-7
SLIDE 7

What’s Wrong?

Too small

3

slide-8
SLIDE 8

What’s Wrong?

Too small Inefficient

3

slide-9
SLIDE 9

What’s Wrong?

Too small Inefficient

3

Solution?

slide-10
SLIDE 10

What’s Wrong?

Too small Inefficient

3

Solution?

Virtualize

slide-11
SLIDE 11

What’s Wrong?

Too small Inefficient

3

Solution?

Virtualize Multiplex

slide-12
SLIDE 12

The Basic Idea:

slide-13
SLIDE 13

The Basic Idea:

Use virtualization to perform network experiments using fewer physical resources.

slide-14
SLIDE 14

The Basic Idea: ... and do this in a way that:

Use virtualization to perform network experiments using fewer physical resources.

slide-15
SLIDE 15

The Basic Idea: ... and do this in a way that:

is transparent to applications and preserves experiment fidelity Use virtualization to perform network experiments using fewer physical resources.

slide-16
SLIDE 16

5

slide-17
SLIDE 17

Challenges

Opportunities

5

slide-18
SLIDE 18

Challenges

  • Fidelity

Opportunities

5

slide-19
SLIDE 19

Challenges

  • Fidelity
  • Preserve network

topology

Opportunities

5

slide-20
SLIDE 20

Challenges

  • Fidelity
  • Preserve network

topology

Opportunities

  • Closed world

5

slide-21
SLIDE 21

Challenges

  • Fidelity
  • Preserve network

topology

Opportunities

  • Closed world
  • Can re-run

experiments

5

slide-22
SLIDE 22

Not Just Picking a VM Technology

6

slide-23
SLIDE 23

Not Just Picking a VM Technology Complete Virtual Network Experimentation System

6

slide-24
SLIDE 24

Full System

  • Virtualization technology
  • Host and network
  • Resource mapping
  • Feedback-directed emulation
  • IP address assignment
  • Scalable control system
  • Routing table calculation

7

slide-25
SLIDE 25

<virtualization>

slide-26
SLIDE 26

Start: FreeBSD jail

  • Namespace isolation
  • Virtual disks
  • We added network virtualization:
  • Ability to bind to multiple interfaces
  • New virtual network device (veth)
  • Separate routing tables

9

slide-27
SLIDE 27

10

slide-28
SLIDE 28

10

slide-29
SLIDE 29

10

slide-30
SLIDE 30

10

slide-31
SLIDE 31

10

slide-32
SLIDE 32

10

slide-33
SLIDE 33

10

slide-34
SLIDE 34

10

slide-35
SLIDE 35

<mapping>

slide-36
SLIDE 36

What does it mean to make a good mapping?

slide-37
SLIDE 37

Good Mapping

  • Pack well
  • Use resources efficiently
  • Specifying packing criteria
  • Do it quickly
  • Critical path for creating an

experiment

13

slide-38
SLIDE 38

assign

  • Solves an NP-hard problem
  • Pack both nodes and links
  • Avoid scarce resources
  • Paper: [Ricci+:CCR03]
  • Based on simulated annealing
  • We extended for virtual nodes

14

slide-39
SLIDE 39

Resource-Based Packing

  • Use quantities we can directly

measure

  • Resource-based system

“This virtual node uses 100 MHz of CPU” “This physical node has 3 GHz of CPU”

  • Works well for heterogenous virtual

and physical nodes

15

slide-40
SLIDE 40

Assigning Quickly

slide-41
SLIDE 41

Small Topologies

17

slide-42
SLIDE 42

Small Topologies

17

slide-43
SLIDE 43

Small Topologies

17

slide-44
SLIDE 44

Virtual Topologies

18

slide-45
SLIDE 45

Virtual Topologies

18

slide-46
SLIDE 46

Virtual Topologies

18

slide-47
SLIDE 47

Prepass

19

slide-48
SLIDE 48

Prepass

19

slide-49
SLIDE 49

Prepass

19

slide-50
SLIDE 50

Scaling With Prepass

20

slide-51
SLIDE 51

Scaling With Prepass

20

200

slide-52
SLIDE 52

Mapping Quality

21

slide-53
SLIDE 53

Mapping Quality

21

slide-54
SLIDE 54

Mapping Quality

21

slide-55
SLIDE 55

Mapping Quality

21

slide-56
SLIDE 56

<feedback>

slide-57
SLIDE 57

How do I know how tightly I can pack my virtual nodes?

slide-58
SLIDE 58

How do I know how tightly I can pack my virtual nodes? I don’t!

slide-59
SLIDE 59

Closed, repeatable world

slide-60
SLIDE 60

The Plan

25

slide-61
SLIDE 61

The Plan

  • Pick a packing

25

slide-62
SLIDE 62

The Plan

  • Pick a packing
  • Run experiment

25

slide-63
SLIDE 63

The Plan

  • Pick a packing
  • Run experiment
  • Monitor for artifacts

25

slide-64
SLIDE 64

The Plan

  • Pick a packing
  • Run experiment
  • Monitor for artifacts
  • If artifacts found:

25

slide-65
SLIDE 65

The Plan

  • Pick a packing
  • Run experiment
  • Monitor for artifacts
  • If artifacts found:
  • Re-pack

25

slide-66
SLIDE 66

The Plan

  • Pick a packing
  • Run experiment
  • Monitor for artifacts
  • If artifacts found:
  • Re-pack
  • Repeat

25

slide-67
SLIDE 67

The Plan

  • Pick a packing
  • Run experiment
  • Monitor for artifacts
  • If artifacts found:
  • Re-pack
  • Repeat

25

slide-68
SLIDE 68

The Plan

25

  • Pick a packing
  • Run experiment
  • Monitor for artifacts
  • If artifacts found:
  • Re-pack
  • Repeat
slide-69
SLIDE 69

Picking Initial Packing

  • Start one-to-one
  • Possibly with a subset of topology
  • Start tightly packed
  • Optimistically assume low usage

26

slide-70
SLIDE 70

Monitoring for Artifacts

  • CPU near 100%
  • Significant paging activity
  • Disk utilization

27

slide-71
SLIDE 71

Re-Packing

  • Measure resource use
  • Feed into resource-based packing

28

slide-72
SLIDE 72

Feedback in a Nutshell

  • Rely on packing, not isolation
  • Discover packing factors empirically
  • Re-use between experiments

29

slide-73
SLIDE 73

<numbers>

slide-74
SLIDE 74

kindex: Packing Factors

slide-75
SLIDE 75

Feedback Case Study

slide-76
SLIDE 76

Feedback Case Study

Round Transactions Per Second Response Time (s)

slide-77
SLIDE 77

Feedback Case Study

Round Transactions Per Second Response Time (s)

Bootstrap: 74 physical 2.29 0.43

slide-78
SLIDE 78

Feedback Case Study

Round Transactions Per Second Response Time (s)

Bootstrap: 74 physical 2.29 0.43 Round 1: 7 physical 1.85 0.53

slide-79
SLIDE 79

Feedback Case Study

Round Transactions Per Second Response Time (s)

Bootstrap: 74 physical 2.29 0.43 Round 1: 7 physical 1.85 0.53 Round 2: 7 physical 2.29 0.43

slide-80
SLIDE 80

Deployed Use

  • Creation time: 7 minutes for 100 nodes
  • 5,125 experiments
  • 296,621 virtual nodes
  • 32% of Emulab nodes virtual
  • 5.75 average packing factor
  • Average: 58 nodes, max: 1,000 nodes

33

slide-81
SLIDE 81

Conclusion

  • Virtualization increases Emulab’s capacity
  • Transparently
  • Preserves fidelity
  • Requires solving several challenging

problems

  • Proven useful in production

34

www.emulab.net

slide-82
SLIDE 82

<end/>

slide-83
SLIDE 83

Related Work

  • Virtual Network Emulation
  • ModelNet, DieCast
  • Virtual Machines
  • Xen, VMWare, vservers, OpenVZ, NET
  • Network Virtualization
  • NetNS, OpenVZ, Trellis, IMUNES
  • Feedback-based Mapping
  • Hippodrome

36

slide-84
SLIDE 84

ModelNet

  • Applications can only run on edge nodes:

single-homed only

  • More basic virtualization
  • No artifact detection
  • No feedback system
  • Emulab has richer control framework
  • Scales to much larger interior networks

37

slide-85
SLIDE 85

Minimal Effective Virtualization

  • Application transparency
  • Application fidelity
  • System capacity

38

slide-86
SLIDE 86

Application Transparency

  • Real applications
  • Simulation
  • Virtual machines
  • Keep most semantics of unshared machines
  • Simple processes
  • Preserve experimenter’s topology
  • Full network virtualization

39

slide-87
SLIDE 87

Application Fidelity

  • Physical results ≈ Virtual

results

  • Virtual node interference
  • Perfect resource isolation
  • Detect artifacts and re-run

40

slide-88
SLIDE 88

System Capacity

  • Low overhead
  • In-kernel (vservers, jails)
  • Hypervisor (VMWare, Xen)
  • Don’t prolong experiments
  • DieCast

41