SimBench A Portable Benchmarking Methodology for Full-System - PowerPoint PPT Presentation

SimBench A Portable Benchmarking Methodology for Full-System Simulators Harry Wagstaff Bruno Bodin Tom Spink Björn Franke Institute for Computing Systems Architecture University of Edinburgh ISPASS 2017 1

Motivation Instruction Set Simulation Evaluating Simulation Tools The SimBench Methodology Overview Implementation Porting SimBench Evaluation 2

Instruction Set Simulation Instruction Set Simulation is used in a wide variety of contexts: 3 Motivation Instruction Set Simulation

Instruction Set Simulation Instruction Set Simulation is used in a wide variety of contexts: • Design Space Exploration • Gem5 • Multi2Sim 3 Motivation Instruction Set Simulation

Instruction Set Simulation Instruction Set Simulation is used in a wide variety of contexts: • Design Space Exploration • Software Development • QEMU • Android Emulator 3 Motivation Instruction Set Simulation

Instruction Set Simulation Instruction Set Simulation is used in a wide variety of contexts: • Design Space Exploration • Software Development • Backwards Compatibility • Apple Rosetta • Nintendo NES Classic 3 Motivation Instruction Set Simulation

Instruction Set Simulation Instruction Set Simulation is used in a wide variety of contexts: • Design Space Exploration • Software Development • Backwards Compatibility Simulators can be broken up into several categories: • Functional Only • Profiling/Performance Modelling • User-Mode • Full System 3 Motivation Instruction Set Simulation

User-Mode Simulation Simulated Flat Memory CPU • Execute a single binary Syscall Emulation Layer • Emulate System Calls • Simplified Memory System Host Host Host Console Timers File System 4 Motivation Instruction Set Simulation

Full-System Simulation Simulated CPU • Boot out of reset Simulated Physical MMU Memory • System-mode Instructions • Memory Translation Simulated Simulated Simulated Storage • Asynchronous Interrupts Serial Port Timers Device • External Devices Host Host Host Console Timers File System 5 Motivation Instruction Set Simulation

Fast Simulation We want to make our simulators go fast! To be clear, we want the simulator itself to be efficient. 6 Motivation Instruction Set Simulation

Fast Simulation We want to make our simulators go fast! To be clear, we want the simulator itself to be efficient. How? • Dynamic Binary Translation • Memory-Related Techniques • Efficient Interrupt Modelling • Control Flow Handling 6 Motivation Instruction Set Simulation

Fast Simulation We want to make our simulators go fast! To be clear, we want the simulator itself to be efficient. How? • Dynamic Binary Translation • Memory-Related Techniques • Efficient Interrupt Modelling • Control Flow Handling How do we evaluate the effectiveness of these techniques in spite of complex interactions? 6 Motivation Instruction Set Simulation

Typical Benchmarking Approaches Most common approaches fall into two categories: • Large, complex macrobenchmark suites • Small, targeted but ad-hoc microbenchmarks Both approaches are problematic for several reasons. 7 Motivation Evaluating Simulation Tools

Macrobenchmarking The most common approach is to use large macrobenchmarks However, this presents several problems: • Benchmark runtimes are long • Difficult to perform detailed analysis • Aggregated runtimes may hide interesting results 1.4 sjeng 1.3 SPEC (overall) 1.2 mcf Speedup 1.1 1 0.9 0.8 0.7 1.7.0 1.7.1 1.7.2 2.0.0 2.0.1 2.0.2 2.1.0 2.1.1 2.1.2 2.1.3 2.2.0 2.2.1 2.3.0 2.3.1 2.4.0.1 2.4.0 2.4.1 2.5.0-rc0 2.5.0-rc1 2.5.0-rc2 QEMU Version 8 Motivation Evaluating Simulation Tools

Ad-Hoc Microbenchmarking Many papers also attempt to use ad-hoc microbenchmarks to assess performance. However, these also have problems: • Source code may be unavailable, damaging reproducibility • Secondary effects may not be taken into account 9 Motivation Evaluating Simulation Tools

Sum Two Arrays - Source Code float out[ARRAY_SIZE]; float a[ARRAY_SIZE]; float b[ARRAY_SIZE]; ... void foo() { for(int i = 0; i < ARRAY_SIZE; ++i) { out[i] = a[i] + b[i]; } } 10 Motivation Evaluating Simulation Tools

Sum Two Arrays - Assembly Code foo: 0: add r3, r0, #512 ; 0x200 4: vldmia r0!, {s15} 8: vldmia r1!, {s14} c: cmp r0, r3 10: vadd.f32 s15, s15, s14 14: vstmia r2!, {s15} 18: bne 4 <foo+0x4> 1c: bx lr 11 Motivation Evaluating Simulation Tools

Sum Two Arrays - Assembly Code ✘ Instruction Page Fault foo: ✘ Code Generation 0: add r3, r0, #512 ; 0x200 4: vldmia r0!, {s15} 8: vldmia r1!, {s14} c: cmp r0, r3 10: vadd.f32 s15, s15, s14 14: vstmia r2!, {s15} 18: bne 4 <foo+0x4> 1c: bx lr Instruction Page fault, Code generation 11 Motivation Evaluating Simulation Tools

Sum Two Arrays - Assembly Code ✘ Instruction Page Fault foo: ✘ Code Generation 0: add r3, r0, #512 ; 0x200 ✘ Cold Memory Access 4: vldmia r0!, {s15} ✘ Data Page Fault 8: vldmia r1!, {s14} c: cmp r0, r3 10: vadd.f32 s15, s15, s14 14: vstmia r2!, {s15} 18: bne 4 <foo+0x4> 1c: bx lr Cold memory access, Data Page fault 11 Motivation Evaluating Simulation Tools

Sum Two Arrays - Assembly Code ✘ Instruction Page Fault foo: ✘ Code Generation 0: add r3, r0, #512 ; 0x200 ✘ Cold Memory Access 4: vldmia r0!, {s15} ✘ Data Page Fault 8: vldmia r1!, {s14} ✘ Hot Memory Access c: cmp r0, r3 10: vadd.f32 s15, s15, s14 14: vstmia r2!, {s15} 18: bne 4 <foo+0x4> 1c: bx lr Hot memory access 11 Motivation Evaluating Simulation Tools

Sum Two Arrays - Assembly Code ✘ Instruction Page Fault foo: ✘ Code Generation 0: add r3, r0, #512 ; 0x200 ✘ Cold Memory Access 4: vldmia r0!, {s15} ✘ Data Page Fault 8: vldmia r1!, {s14} ✘ Hot Memory Access c: cmp r0, r3 ✓ FP Operation 10: vadd.f32 s15, s15, s14 14: vstmia r2!, {s15} 18: bne 4 <foo+0x4> 1c: bx lr FP Operation 11 Motivation Evaluating Simulation Tools

Sum Two Arrays - Assembly Code ✘ Instruction Page Fault foo: ✘ Code Generation 0: add r3, r0, #512 ; 0x200 ✘ Cold Memory Access 4: vldmia r0!, {s15} ✘ Data Page Fault 8: vldmia r1!, {s14} ✘ ✘ Hot Memory Access c: cmp r0, r3 ✓ FP Operation 10: vadd.f32 s15, s15, s14 14: vstmia r2!, {s15} 18: bne 4 <foo+0x4> 1c: bx lr Hot memory access 11 Motivation Evaluating Simulation Tools

Sum Two Arrays - Assembly Code ✘ Instruction Page Fault foo: ✘ Code Generation 0: add r3, r0, #512 ; 0x200 ✘ Cold Memory Access 4: vldmia r0!, {s15} ✘ Data Page Fault 8: vldmia r1!, {s14} ✘ ✘ Hot Memory Access c: cmp r0, r3 ✓ FP Operation 10: vadd.f32 s15, s15, s14 ✘ Direct Control Flow 14: vstmia r2!, {s15} ✘ Interrupt 18: bne 4 <foo+0x4> 1c: bx lr Direct control flow, Interrupt 11 Motivation Evaluating Simulation Tools

Sum Two Arrays - Assembly Code ✘ Instruction Page Fault foo: ✘ ✘ Code Generation 0: add r3, r0, #512 ; 0x200 ✘ Cold Memory Access 4: vldmia r0!, {s15} ✘ Data Page Fault 8: vldmia r1!, {s14} ✘ ✘ Hot Memory Access c: cmp r0, r3 ✓ FP Operation 10: vadd.f32 s15, s15, s14 ✘ Direct Control Flow 14: vstmia r2!, {s15} ✘ Interrupt 18: bne 4 <foo+0x4> 1c: bx lr Code generation 11 Motivation Evaluating Simulation Tools

Sum Two Arrays - Assembly Code ✘ Instruction Page Fault foo: ✘ ✘ ✘ Code Generation 0: add r3, r0, #512 ; 0x200 ✘ Cold Memory Access 4: vldmia r0!, {s15} ✘ Data Page Fault 8: vldmia r1!, {s14} ✘ ✘ Hot Memory Access c: cmp r0, r3 ✓ FP Operation 10: vadd.f32 s15, s15, s14 ✘ Direct Control Flow 14: vstmia r2!, {s15} ✘ Interrupt 18: bne 4 <foo+0x4> 1c: bx lr Code generation 11 Motivation Evaluating Simulation Tools

Sum Two Arrays - Assembly Code ✘ Instruction Page Fault foo: ✘ ✘ ✘ Code Generation 0: add r3, r0, #512 ; 0x200 ✘ Cold Memory Access 4: vldmia r0!, {s15} ✘ Data Page Fault 8: vldmia r1!, {s14} ✘ ✘ Hot Memory Access c: cmp r0, r3 ✓ FP Operation 10: vadd.f32 s15, s15, s14 ✘ Direct Control Flow 14: vstmia r2!, {s15} ✘ ✘ Interrupt 18: bne 4 <foo+0x4> ✘ Indirect Control Flow 1c: bx lr Indirect control flow, Interrupt 11 Motivation Evaluating Simulation Tools

SimBench We present SimBench, which is designed to address some of these problems. 12 The SimBench Methodology Overview

SimBench We present SimBench, which is designed to address some of these problems. SimBench. . . • Contains a range of targeted microbenchmarks 12 The SimBench Methodology Overview

SimBench We present SimBench, which is designed to address some of these problems. SimBench. . . • Contains a range of targeted microbenchmarks • Is aimed at full-system simulation techniques 12 The SimBench Methodology Overview

SimBench We present SimBench, which is designed to address some of these problems. SimBench. . . • Contains a range of targeted microbenchmarks • Is aimed at full-system simulation techniques • Is easily portable to new platforms and architectures 12 The SimBench Methodology Overview