Latency and Throughput Latency (of task): Time elapsed between - PowerPoint PPT Presentation

Latency and Throughput • Latency (of task): – Time elapsed between start of the task and and its finish Example : Travel from Houston to NY Takeoff to Landing Arrival at Airport to Baggage pickup Leave for Airport to Arrival at Destination hotel Need to define start and end times by specifying the “task” carefully Different subsystems may contribute to end-to-end latency Each subsystem employs different latency reduction mechanisms 8

Latency and Throughput • Throughput (of system or set of tasks): Rate of task completion: Number of tasks completed per unit time Instantaneous throughput (at time t): Rate of task completion in a “small” interval around time t Average throughput (in interval [0, T]): Number of tasks completed in interval / T Burst: 0.9 per sec Average = 0.5/sec

Latency and Throughput Moving Large Data Sets to Data Center Ship 1TB of data from Point of Creation to Point of Storage FedEx: Latency : 12 Hours Throughput: 10 6 MB / (12 x 3600) sec ~ 23MB/sec Not the usual understanding of throughput -- implicitly assumes small time granularity Internet: 10Mb/sec link Throughput = 10Mbps (assumes no errors/retransmissions, ignoring metadata overhead, link utilization) Latency = 1TB/(10Mbps) = 800,000 sec ~ 224 hours ~ 10 days With OC3 (155Mbits/sec) ~ 16 hours Streaming Video 5 GB File 10Mbps link Latency (for file download) = 5GB/10Mbps = 4000s ~ 1.1 hours (ideal) Throughput: Smaller of 10MBps or Playback Rate Latency (till start of transmission) : ~ 1s Latency (per packet) ~ ms

Latency and Throughput Houston NY • Travelers concern: Latency: Time for flight from Houston to NY --- 3 hours • Airlines concern: Throughput: Rate at which it can move people • Assume 1 airplane of capacity 150 used on route • Maximum throughput: 150 persons / 6 hours = 25 pph H NY H NY H NY H 13

Latency and Throughput Improve Latency? Better Technology: Faster plane (materials, fuels, engine, …) • Better Implementation: Faster route (wind pattern, distance, ….) • These will also usually improve the throughput • What if faster but smaller plane? • What if not enough passengers? • Improve Throughput? Bigger Plane: Capacity of 300 --- Throughput = 50 pph • Multiple Planes: • Fly multiple independent routes simultaneously • Houston Throughput: 100 pph (with 4 planes) Theoretically continue to increase throughput by adding more planes/routes Practical limits: Share segments and synchronize at intermediate stops (increase latency) 14 Popular and unpopular routes/aircraft raises load balancing issues

Latency and Throughput • Pipeline along same route Houston NY 6 Houston 0 1 2 3 4 5 Assuming 1 hour mandatory separation between successive flights on the same route 1 Can have 6 flights simultaneously in 2 progress Throughput: 150 persons / 1hr = 150 pph 3 Note: 4 • Cannot use more planes (increase throughput) with given separation constraint 5 • Fundamental limits on minimum separation ( e.g. one plane length!!!) 6 15

Latency and Throughput • Latency (of task): – Time elapsed between start of the task and and its finish Example : Travel from Houston to NY following storm Takeoff to Landing Wait Time due to built up passenger traffic Can increasing throughput reduce latency? Latency dependence on Load No load (lightly Loaded): No effect Heavy Load: Reduces Queuing (wait) time 8

Latency and Throughput Improve Latency? 1. Better Technology: Faster plane Increase Clock Rate 2. Better Implementation: Faster route Multi-cycle implementation These will also improve the throughput • 1. T Instruction latency halved T/2 Instruction throughput doubled 2. Instructions have different latencies ---- average latency reduces depending on instruction mix Average Latency = ∑ t i w i t i : Latency for class i instructions w i : execution frequency of class i instructions Average Throughput:1 / ∑ w i t i 16 Best and worst case measures use pathological instruction mix

Latency and Throughput Improve Throughput? Bigger Plane Multiple Issue Processors • (VLIW, Superscalar, SMT) 1 2 3 4 5 6 7 8 9 10 11 1 4 7 10 T: 1 2 3 4 5 6 7 8 9 10 11 Each instruction bundle holds several instructions of the single-issue processor Throughput (potentially) increases by factor of bundle size 17

Latency and Throughput Improve Throughput? Multiple Planes: • • Fly multiple independent routes simultaneously Multithreaded processors • Multi-core processors • Multiprocessors • Single-threaded loop execution Thread 1 Thread 2 Thread 3 Multi-threaded loop execution with independent iterations 18

Latency and Throughput Improve Throughput? Multiple Planes: • • Pipeline along same route Begin next instruction as soon as the previous instructions clears the first stage (path segment) Overlapped pipelined execution 19

Latency and Throughput Improve Throughput? Multiple Planes: • • Pipeline along same route Begin next instruction as soon as the previous instructions clears the first stage (path segment) Overlapped pipelined execution 20