Introduction Workloads for Experiments Introduction to workloads - PDF document

Introduction Workloads for Experiments • Introduction to workloads CS 239 • Workload selection Experimental Methodologies for • Types of workloads System Software • Characterizing a workload Peter Reiher April 19, 2007 Lecture 5 Lecture 5 Page 1 Page 2 CS 239, Spring 2007 CS 239, Spring 2007 Introduction to Workloads What is a Workload? • A workload is anything a computer is • What is a workload? asked to do • Real workloads • Test workload: any workload used to • Synthetic workloads analyze performance • Real workload: any observed during normal operations • Synthetic workload : workload created for controlled testing Lecture 5 Lecture 5 Page 3 Page 4 CS 239, Spring 2007 CS 239, Spring 2007 Workloads and Systems Desirable Properties in Test Experiments Workloads • Systems do something • Realistic • Point of experiments is to find out how • Representative of whole range of real well workloads • The workload is what the system does • Controllable • To determine true systems • Reproducible performance, must apply good • Tractable workload Lecture 5 Lecture 5 Page 5 Page 6 CS 239, Spring 2007 CS 239, Spring 2007 1

Some Problems in Why Not Use the Real Workload? Experimental Workloads • Why not just test with reality? • What is the real workload you’d like to match? • Not always possible • How can you accurately mirror that • Generally not reproducible workload? • Definitely not controllable • How do you handle wide ranges of • Sometimes legal issues workload variations? • How do you handle predicted workloads? • Still, occasionally possible • How do you scale workloads to –Usually after other methods show experimental conditions? general feasibility and properties Lecture 5 Lecture 5 Page 7 Page 8 CS 239, Spring 2007 CS 239, Spring 2007 Real Workloads Synthetic Workloads • Advantage is they represent reality • Advantages: • Disadvantage is they’re uncontrolled – Controllable – Can’t be repeated – Repeatable – Can’t be described simply – Sometimes standardizable – Difficult to analyze – Portable to other systems • Nevertheless, often useful for “final – Easily modified analysis ” papers • Disadvantage: can never be sure real world – E.g., “We ran Ficus and it works well” will match them Lecture 5 Lecture 5 Page 9 Page 10 CS 239, Spring 2007 CS 239, Spring 2007 Workload Selection Services Exercised • What services does system actually use? • Services Exercised – Faster CPU won’t speed up big “cp” • Level of Detail – Network performance useless for matrix • Representivity work • What metrics measure these services? • Timeliness – MIPS for CPU speed • Other Considerations – Bandwidth for network, I/O – TPS for transaction processing Lecture 5 Lecture 5 Page 11 Page 12 CS 239, Spring 2007 CS 239, Spring 2007 2

Completeness Component Testing • Sometimes only individual components are • Computer systems are complex compared –Effect of interactions hard to predict – Would a new CPU speed up our system? – Would IPV6 affect Web server –So must be sure to test entire system performance? • Important to understand balance • But component may not be directly related between components to performance –I.e., don’t use CPU workload to – Analysis of Variation (ANOVA) test can evaluate I/O-bound application help here Lecture 5 Lecture 5 Page 13 Page 14 CS 239, Spring 2007 CS 239, Spring 2007 Service Testing Characterizing a Service • May be possible to isolate interfaces to • Identify service provided by major just one component subsystem –E.g., instruction mix for CPU • List factors affecting performance • Consider services provided and used • List metrics that quantify demands and by that component performance • System often has layers of services • Identify workload provided to that –Can cut at any point and insert service workload Lecture 5 Lecture 5 Page 15 Page 16 CS 239, Spring 2007 CS 239, Spring 2007 Example: Web Server Web Client Analysis Web Page Visits • Services: visit page, follow hyperlink, Web Client display information TCP/IP Connections Network • Factors: page size, number of links, fonts HTTP Requests required, embedded graphics, sound Web Server Web Page Accesses • Metrics: response time (both definitions) File System Disk Transfers • Workload: a list of pages to be visited and Disk Drive links to be followed Lecture 5 Lecture 5 Page 17 Page 18 CS 239, Spring 2007 CS 239, Spring 2007 3

Network Analysis Web Server Analysis • Services: connect to server, transmit • Services: accept and validate connection, request, transfer data fetch HTTP data • Factors: bandwidth, latency, protocol • Factors: Network performance, CPU speed, used system load, disk subsystem performance • Metrics: response time, connections served • Metrics: connection setup time, response latency, achieved bandwidth • Workload: a stream of incoming HTTP connections and requests • Workload: a series of connections to one or more servers, with data transfer Lecture 5 Lecture 5 Page 19 Page 20 CS 239, Spring 2007 CS 239, Spring 2007 File System Analysis Disk Drive Analysis • Services: open file, read file (writing • Services: read sector, write sector doesn’t matter for Web server) • Factors: seek time, transfer rate • Factors: disk drive characteristics, file • Metrics: response time system software, cache size, partition size • Workload: a stream of read/write requests • Metrics: response time, transfer rate • Workload: a series of file-transfer requests Lecture 5 Lecture 5 Page 21 Page 22 CS 239, Spring 2007 CS 239, Spring 2007 Level of Detail Representivity • Detail trades off accuracy vs. cost • Obviously, workload should represent • Highest detail is complete trace desired application – Arrival rate of requests • Lowest is one request, usually most – Resource demands of each request common – Resource usage profile of workload over • Intermediate approach: weight by time frequency • Again, accuracy and cost trade off • We will return to this when we discuss • Need to understand whether detail matters workload characterization Lecture 5 Lecture 5 Page 23 Page 24 CS 239, Spring 2007 CS 239, Spring 2007 4

Timeliness Types of Workloads • Use patterns change over time • Microbenchmarks – File size grows to match disk size – Web pages grow to match network • Benchmarks bandwidth • Traces • If using “old” workloads, must be sure user • Generators and exercisers behavior hasn’t changed • Live workloads • Even worse, behavior may change after test, as result of installing new system – “Latent demand” phenomenon Lecture 5 Lecture 5 Page 25 Page 26 CS 239, Spring 2007 CS 239, Spring 2007 Microbenchmarks Purpose of Microbenchmark • Sometimes that’s precisely what you want • A test of the performance of a very low to measure level operation – E.g., measuring an improvement in –CPU arithmetic operation memory allocator –Sending one message • Sometimes it describes key property of overall system –Allocating one buffer – Message send cost is crucial in distributed system Lecture 5 Lecture 5 Page 27 Page 28 CS 239, Spring 2007 CS 239, Spring 2007 Disadvantages of Microbenchmarks Advantages of Microbenchmarks + Generally simple to test - Doesn’t show interactions + Pretty easy to understand - Often not relevant to the real issue + Limited amount of work to test - Tend not to be considered in varying circumstances + Can reveal most important elements of system behavior - Usually “best case” + Sometimes exactly what you are - May offer little insight on how to looking for improve system Lecture 5 Lecture 5 Page 29 Page 30 CS 239, Spring 2007 CS 239, Spring 2007 5

Using Microbenchmarks Benchmarks • Usually suitable for simple situations • A standardized artificial workload – Or when minimum cost is of interest • Generally designed to test specific type of • Generally don’t fully describe real systems system • Microbenchmarks are almost never enough – File system, database, web server, etc. – So do them only when they provide • Usually intended for wide use insight – Which allows system comparisons – Be suspicious of whole systems studies • In principle . . . that only report microbenchmarks Lecture 5 Lecture 5 Page 31 Page 32 CS 239, Spring 2007 CS 239, Spring 2007 Where Do Benchmarks Come From? Some Types of Benchmarks • Sometimes from standards bodies • File system benchmarks –Or industry consortia • Processor performance benchmarks –Occasionally government fiat • Database benchmarks • Sometimes proposed by leading researchers –Either picked up by others or not Lecture 5 Lecture 5 Page 33 Page 34 CS 239, Spring 2007 CS 239, Spring 2007 How Do You Build a Benchmark? Popular Benchmarks • Sieve • Pick a representative real-world application • Whetstone • Pick sample data • Debit/credit • Run it on system to be tested • SPEC • Modified Andrew Benchmark, MAB, is a • Modified Andrew Benchmark real-world benchmark • Easy to do, accurate for that sample data • Fails to consider other applications, data – So just how representative was your choice? Lecture 5 Lecture 5 Page 35 Page 36 CS 239, Spring 2007 CS 239, Spring 2007 6