DieCast: Testing Distributed Systems with an Accurate Scale Model - - PowerPoint PPT Presentation

▶
diecast testing distributed systems with an accurate
SMART_READER_LITE
LIVE PREVIEW

DieCast: Testing Distributed Systems with an Accurate Scale Model - - PowerPoint PPT Presentation

Aug 22, 2022 285 likes •595 views

DieCast: Testing Distributed Systems with an Accurate Scale Model Diwaker Gupta Diwaker Gupta Kashi V. Vishwanath Amin Vahdat University of California, San Diego High performance Alice filesystem Limited testing infrastructure Diverse

slide-1
SLIDE 1

DieCast: Testing Distributed Systems with an Accurate Scale Model

Diwaker Gupta Diwaker Gupta Kashi V. Vishwanath Amin Vahdat University of California, San Diego

slide-2
SLIDE 2

High performance filesystem Alice

June 7, 2008 NSDI 2008 | DieCast 2

Limited testing infrastructure Diverse deployment environments

Use smaller infrastructure to test a much larger system

slide-3
SLIDE 3

Goals

  • Fidelity

– How closely can we replicate the target system?

  • Reproducibility
  • Reproducibility

– Can we do controlled experiments?

  • Efficiency

– Use fewer resources

June 7, 2008 NSDI 2008 | DieCast 3

DieCast can scale up a test infrastructure by an order of magnitude

slide-4
SLIDE 4

DieCast Overview

Replicate target system using fewer machines Resource equivalence: perceived CPU capacity, disk and network characteristics capacity, disk and network characteristics Preserve application performance

× Not scaled

× Physical memory: mitigating solutions × Secondary storage: cheap

June 7, 2008 NSDI 2008 | DieCast 4

slide-5
SLIDE 5

Original System

Application servers

June 7, 2008 NSDI 2008 | DieCast 5

Load balancer Web servers Database servers Switches

  • Fidelity
  • Reproducibility
  • Efficiency
slide-6
SLIDE 6

Server Consolidation (VMs)

June 7, 2008 NSDI 2008 | DieCast 6

Network emulation

  • Fidelity
  • Reproducibility
  • Efficiency
slide-7
SLIDE 7

Multiplexing Leads to Resource Partitioning

3 GHz CPU, 1 Gbps N/W, 15 Mbps disk I/O, 2 GB RAM

June 7, 2008 NSDI 2008 | DieCast 7

Split equally among 5 VMs ~ 600 MHz CPU, 200 Mbps N/W, 3 Mbps disk I/O, 400 MB RAM each

slide-8
SLIDE 8

Time Dilation [NSDI 2006]

Real time (No dilation) Events 1 sec 10 Mb

  • Slow down passage of time

within the OS

  • CPU, network, disk – all appear

faster

Key idea: time is also a resource!

June 7, 2008 NSDI 2008 | DieCast 8

Perceived bandwidth = 10 Mb/s Dilated time Events 100 msec 10 Mb Perceived bandwidth = 100 Mb/s

faster

  • Experiments take longer

Time Dilation Factor (TDF) = Real time/Virtual time In this example, TDF = 1sec/100ms = 10

slide-9
SLIDE 9

Multiplexing Under Time Dilation

3 GHz CPU, 1 Gbps N/W, 15 Mbps disk I/O, 2 GB RAM

June 7, 2008 NSDI 2008 | DieCast 9

~ 600 MHz CPU, 200 Mbps N/W, 3 Mbps disk I/O, 400-MB RAM, each ~ 3 GHz CPU, 1 Gbps N/W, 15 Mbps disk I/O?, 400 MB RAM each TDF 5

slide-10
SLIDE 10

Time Dilation: External Interactions

Dilated Time Frame

June 7, 2008 NSDI 2008 | DieCast 10

Network External systems running in the real time frame

slide-11
SLIDE 11

Disk I/O Scaling

  • Invariant: perceived disk characteristics

are preserved

– Seek time – Read/write throughput – Read/write throughput

  • Issues

– Low level functionality in firmware – Different I/O models – Per request scaling is difficult

June 7, 2008 NSDI 2008 | DieCast 11

slide-12
SLIDE 12

Implementation Details

  • Supported platforms

– Xen 2.0.7, 3.0.4, 3.1 – Can be ported to non-virtualized systems

  • Support for unmodified guest OSes
  • Support for unmodified guest OSes
  • Disk I/O scaling for different I/O models

– Fully virtualized: integration with DiskSim – Paravirtualized: scaling in device driver

June 7, 2008 NSDI 2008 | DieCast 12

slide-13
SLIDE 13

Disk I/O Scaling: Fully Virtualized VMs

ioemu disksim

VM

(Unmodified OS)

VM

(Unmodified OS)

Domain-0 Domain-0

Guest OS unaware that no real disk Request completion time in simulated disk

June 7, 2008 NSDI 2008 | DieCast 13

VM disk image ioemu Disk device driver

Xen Xen

exists Guest OS filesystem I/O emulation

slide-14
SLIDE 14

Disk I/O Scaling: Fully Virtualized VMs

ioemu disksim

VM

(Unmodified OS)

VM

(Unmodified OS)

Domain-0 Domain-0

Required perceived time: Tsim ⇒Total real time Service time in simulated disk: Tsim DiskSim running time: Tdisksim

June 7, 2008 NSDI 2008 | DieCast 14

VM disk image ioemu Disk device driver

Xen Xen

Treal = TDF*Tsim Actual time to service: Tioemu Delay: Delay

Treal = Tioemu + Delay + Tdisksim ⇒Delay = (TDF*Tsim) – Tdisksim – Tioemu

slide-15
SLIDE 15

Network I/O Scaling

Real Configuration Perceived Configuration

Invariant: Perceived network characteristics (bandwidths and latencies) must be preserved 10 Mb/s, 20ms RTT

Real Configuration Perceived Configuration

Original system (TDF 1) 10 Mb/s, 20 ms 10 Mb/s, 20 ms Time Dilation (TDF 5) 10 Mb/s, 20 ms 50 Mb/s, 4 ms DieCast (TDF 5) 2 Mb/s, 100 ms 10 Mb/s, 20 ms

June 7, 2008 NSDI 2008 | DieCast 15

Network emulation: ModelNet, Dummynet

slide-16
SLIDE 16

Recap

  • Multiplex VMs for efficiency
  • Time dilation to scale resources
  • Disk I/O scaling
  • Network I/O scaling
  • Network I/O scaling

At this point, the scaled system almost looks like original system!

June 7, 2008 NSDI 2008 | DieCast 16

  • Fidelity
  • Reproducibility
  • Efficiency
slide-17
SLIDE 17

Validation

  • How well does DieCast scaled performance

match the original system?

– Application specific metrics

  • Can a smaller system be configured to match

the resources of a larger system? the resources of a larger system?

– Resource utilization profiles

  • Applications: RUBiS, BitTorrent, Isaac
  • RUBiS

– eBay like e-Commerce service – Ships with workload generator

June 7, 2008 NSDI 2008 | DieCast 17

slide-18
SLIDE 18

RUBiS: Topology

4 DB 8 Web Servers 4 DB 8 Web Servers

Wide Area Link

June 7, 2008 NSDI 2008 | DieCast 18

16 Workload Generators Wide Area Link

slide-19
SLIDE 19

Experimental Setup

Baseline configuration: 40 physical machines DieCast scaled Configuration: 4 physical machines, 10 VMs each

  • Xen 3.1, fully virtualized VMs
  • Debian Etch, Linux 2.6.17, 256 MB RAM
  • DiskSim emulating Seagate ST3217
  • Network emulation using ModelNet

June 7, 2008 NSDI 2008 | DieCast 19

slide-20
SLIDE 20

RUBiS: Throughput

June 7, 2008 NSDI 2008 | DieCast 20

slide-21
SLIDE 21

RUBiS: Response Time

June 7, 2008 NSDI 2008 | DieCast 21

slide-22
SLIDE 22

RUBiS: Resource Usage

CPU

June 7, 2008 NSDI 2008 | DieCast 22

Memory Network

slide-23
SLIDE 23

Validation Recap

  • Evaluated

– RUBiS – BitTorrent – Isaac Many more details in the paper – Isaac

  • Demonstrated

– Match application specific metrics – Preserve resource utilization profile

June 7, 2008 NSDI 2008 | DieCast 23

slide-24
SLIDE 24

Case study: Panasas

  • Panasas builds scalable storage systems

for high performance computing

– http://www.panasas.com

  • Caters to variety of clients
  • Caters to variety of clients
  • Difficult or even impossible to replicate

deployment environment of all clients

  • Limited resources for testing

June 7, 2008 NSDI 2008 | DieCast 24

slide-25
SLIDE 25

DieCast in Panasas

  • Custom OS
  • Integrated hw/sw offering
  • Not runnable on Xen
  • Porting DieCast to non-

virtualized environments Clients

June 7, 2008 NSDI 2008 | DieCast 25

Clients run Linux, can be virtualized Dummynet for network scaling Storage cluster

slide-26
SLIDE 26

Panasas: Evaluation Summary

Baseline DieCast scaled: 1 PM, 10 VMs

  • Validation

– Two benchmarks from standard test suite: IOZone, MPI-IO; varying block sizes – Match performance metrics

June 7, 2008 NSDI 2008 | DieCast 26

Scaling: Used 100 machines to scale to 1000 clients

slide-27
SLIDE 27

Limitations

  • Memory scaling
  • Long running workloads
  • Specialized hardware appliances
  • Fine grained timing

June 7, 2008 NSDI 2008 | DieCast 27

slide-28
SLIDE 28

Summary

  • DieCast: scalable testing

– Fidelity, Reproducibility, Efficiency

  • Contributions

– Support for unmodified operating systems – Support for unmodified operating systems – Implement disk I/O scaling (DiskSim integration) – CPU scheduler enhancements for time dilation – Comprehensive evaluation, including a commercial storage system

June 7, 2008 NSDI 2008 | DieCast 28

slide-29
SLIDE 29

Thanks!

Questions? dgupta@cs.ucsd.edu