The Open Source ProtoFlex Simulator Eric S. Chung, Michael K. - - PowerPoint PPT Presentation

▶
the open source
SMART_READER_LITE
LIVE PREVIEW

The Open Source ProtoFlex Simulator Eric S. Chung, Michael K. - - PowerPoint PPT Presentation

Oct 12, 2023 333 likes •694 views

The Open Source ProtoFlex Simulator Eric S. Chung, Michael K. Papamichael, James C. Hoe, Babak Falsafi, Ken Mai Computer Architecture Lab at RAMP Retreat, June 2009 The ProtoFlex Simulator History Project started (circa 2007) to build

slide-1
SLIDE 1

Computer Architecture Lab at

RAMP Retreat, June 2009

The Open Source ProtoFlex Simulator

Eric S. Chung, Michael K. Papamichael, James C. Hoe, Babak Falsafi, Ken Mai

slide-2
SLIDE 2

The ProtoFlex Simulator

  • History

– Project started (circa 2007) to build scalable, full-system multiprocessor simulators using FPGAs

  • Key Features

– Functional simulator for N-way UltraSPARC III server (~50-90 MIPS) – Using hybrid simulation, runs real server apps + Solaris OS – Employs multithreading to virtualize # CPUs per FPGA core

2

Hybrid Simulation Virtualization

slide-3
SLIDE 3

Open Sourcing ProtoFlex

  • Why open source?

– Demonstration of FPGAs as viable architecture research vehicle – Facilitate adoption of hybrid simulation & host multithreading – Encourage building on top of our work

  • What are we releasing?

– Bluespec source HDL, Verilog and pre-generated netlists for SPARCV9 CPU model + interfaces – XUPV5 Reference Design for EDK 10.1 – Virtutech Simics plug-ins for hybrid simulation – Top-level SW controller, user command-line interface – Documentation through online wiki

3

slide-4
SLIDE 4

Outline

  • Motivation
  • The ProtoFlex Simulator

– High level components – UltraSPARC core model

  • Using ProtoFlex
  • XUPV5 Reference Design
  • Distribution Details

4

slide-5
SLIDE 5

The ProtoFlex Simulator

  • User perceives familiar SW-like UltraSPARC III simulator

– Software user interface similar to Simics – Applications load directly from Simics checkpoints – Standard simulation features: state viewing, scripting, single-stepping, checkpointing, terminal, profiling/monitoring

5

slide-6
SLIDE 6

The ProtoFlex Simulator

  • User perceives familiar SW-like UltraSPARC III simulator

– Software user interface similar to Simics – Applications load directly from Simics checkpoints – Standard simulation features: state viewing, scripting, single-stepping, checkpointing, terminal, profiling/monitoring

6

FPGA Linux PC

PFMON SIMICS (I/O) FPGA Core Main Memory PowerPC (or uBlaze)

Ethernet

User Interfac e

slide-7
SLIDE 7

Our UltraSPARC III Core Model

  • ISA Specifications

– 64-bit SPARCV9 ISA + US III extensions – 8 register windows, 4 global register files – 512-entry D-TLB, 128 I-TLB

  • Implementation

– 14-stage, multi-threaded pipeline, switch context on each cycle – On Virtex-5, XST~148MHz, Placed & routed @ 100MHz – Parameterized non-blocking caches – FP + rare MMU instructions are SW-emulated by nearby uBlaze – 100% mirrors Virtutech Simics model

7

I-TLB Stage 1 I-TLB Stage 2 64-bit ALU Stage 1 Context Scheduler I-Fetch Address Generate I-Fetch Tag Check US III Decoder 64-bit ALU Stage 2 D-TLB Stage 1 D-TLB Stage 2 D-TLB Stage 3 D-Cache Address Generate D-Cache Tag Check Multi-Cycle Instruction Unit Nonblocking I-cache (BRAM) Writeback Arbiter to DDR Memory Integer RF (BRAM) Nonblocking D-cache (BRAM)

slide-8
SLIDE 8

Core Design Statistics

  • Runs 100MHz on V5

– Synthesizes up to 148MHz using standard tools (ISE XST)

  • Logic usage

– 23.5 KLUTs (11.3% LX330T)

  • BRAM usage

– 120 BRAMs for 16-context configuration (37% LX330T)

  • Future optimizations

– Paging structures to SRAM or DRAM can reduce BRAM by significant amount – Will release in future updates

8

slide-9
SLIDE 9

Outline

  • Motivation
  • The ProtoFlex Simulator
  • Using ProtoFlex
  • XUPv5 Reference Design
  • Distribution

9

slide-10
SLIDE 10

Using ProtoFlex

  • Add passive monitors

– Counters, histograms – Roll your own

10

I-TLB Stage 1 I-TLB Stage 2 64-bit ALU Stage 1 Context Scheduler I-Fetch Address Generate I-Fetch Tag Check US III Decoder 64-bit ALU Stage 2 D-TLB Stage 1 D-TLB Stage 2 D-TLB Stage 3 D-Cache Address Generate D-Cache Tag Check Multi-Cycle Instruction Unit Nonblocking I-cache (BRAM) Writeback Arbiter to DDR Memory Integer RF (BRAM) Nonblocking D-cache (BRAM)

  • Trace-based simulation

– Collect dynamic traces – Feed traces to functional-first timing model

  • Sampled Program Monitoring

– Use micro-blaze (or PPC) to monitor core/memory state – Unintrusive profiling w/o changes to target SW

Counters Counters Counters

Histogram Tracker

Histogram

Tracker Histogram Trackers

Timing Model

FPGA Hard/Soft Core (PowerPC or MicroBlaze)

slide-11
SLIDE 11

Applications of ProtoFlex

  • Examples

– Functional-first CMP cache coherency model for first-order timing models and functional warming *TRETS’09+ – Real-time stack trace profiling – CMP interconnect model (in progress) – Realistic CPU traffic generators (in progress)

  • … running real 16-CPU server workloads

– Oracle TPC-C, IBM DB/2 TPC-C, TPC-H, SPEC2K

11

Piranha CMP Cache (First-Order Timing Model) Statistics + Warmed Coherency & Tag States

slide-12
SLIDE 12

What does the RTL look like?

  • We use Bluespec System Verilog (high-level, synthesizable HDL)

– 4-8 weeks learning curve for normal HDL users – Once learned, easier to read/modify than conventional RTL – Requires BSV compiler (free for academics) – Paper in MEMOCODE’09 describes BSV coding/validation of core

  • Sample code:

12

rule split_ALU_pipeline (True); … p1 = piperegs[DECODE]; piperegs[ALU1] <= doALUStage1(p1, alu_ifc); p2 = piperegs[ALU1]; piperegs[ALU2] <= doALUStage2(p2, alu_ifc); … endrule rule merged_ALU_pipeline (True); … p1 = piperegs[DECODE]; p_tmp1 = doALUStage1(p1, alu_ifc); p_tmp2 = doALUStage2(p_tmp1, alu_ifc); piperegs[ALU] <= p_tmp2; … endrule

2-stage ALU 1-stage ALU

slide-13
SLIDE 13

Other Simulator Features

  • Changing Core Parameters

– Number of CPU contexts – Cache sizes – Merge/split pipeline stages – Enable/disable modules for profiling & debugging – Clock frequency (tested @ 10 MHz – 100 MHz) – Set optimal LUTRAM size (16 = V2P, 64 = V5) – Choose LUTRAMs or BRAMs for any CPU state

  • System Parameters

– UDP or TCP/IP (for PFMON-to-FPGA communication) – XUPv5, BEE2

13

slide-14
SLIDE 14

Outline

  • Motivation
  • The ProtoFlex Simulator
  • Using ProtoFlex
  • XUPv5 Reference Design
  • Distribution

14

slide-15
SLIDE 15

Platform Release: XUPV5

  • Why XUPv5?

– Inexpensive (~$750), easily accessible – Standard tool flows (EDK, ISE) – Reference design portable to other platforms – just drop in our ‘pcores’

  • Supporting other platforms

– Future ports to BEE3 & Xilinx Accelerated Computing Platform (ACP) – Plan to release with future updates

15

slide-16
SLIDE 16

Required Equipment

16

FPGA Board Linux PC Ethernet BlueSPARC PFMON + Simics

slide-17
SLIDE 17

Required Equipment

17

FPGA Board Linux PC Ethernet BlueSPARC PFMON + Simics

slide-18
SLIDE 18

18

XUPv5 Overview

  • Virtex-5 LX110T
  • DDR2 Memory

– up to 2GB

  • 1ΜΒ SRAM
  • 1Gbps Ethernet
  • 3Gbps SATA
  • Serial Port
slide-19
SLIDE 19

SRAM Controller Multi-Port Memory Controller

Reference Design Block Diagram

19

PLB

BlueSPARC MicroBlaze Ethernet BRAM Serial Port BRAM BRAM

XUPv5

LX110T DRAM SRAM

slide-20
SLIDE 20

SRAM Controller Multi-Port Memory Controller

BlueSPARC

20

PLB

MicroBlaze Ethernet BRAM Serial Port BRAM BRAM

XUPv5

LX110T DRAM SRAM

  • EDK IP core

– connects to PLB & NPI

  • Runs @ 100MHz
  • 4 CPU contexts
  • 64KB I&D L1 caches

BlueSPARC

slide-21
SLIDE 21

SRAM Controller Multi-Port Memory Controller

Reference Design Block Diagram

21

PLB

BlueSPARC MicroBlaze Ethernet BRAM Serial Port BRAM BRAM

XUPv5

LX110T DRAM SRAM

slide-22
SLIDE 22

SRAM Controller Multi-Port Memory Controller

BlueSPARC

22

PLB

BlueSPARC MicroBlaze Ethernet Serial Port

XUPv5

LX110T DRAM SRAM

  • 81% utilization

– Core 51% (76 out of 148) – Rest 30% (45 out of 148)

BRAM BRAM BRAM

slide-23
SLIDE 23

SRAM Controller Multi-Port Memory Controller

Reference Design Block Diagram

23

PLB

BlueSPARC MicroBlaze Ethernet BRAM Serial Port BRAM BRAM

XUPv5

LX110T DRAM SRAM

slide-24
SLIDE 24

SRAM Controller Multi-Port Memory Controller

Ethernet

24

PLB

BlueSPARC MicroBlaze BRAM Serial Port BRAM BRAM

XUPv5

LX110T DRAM SRAM

  • 4 MB/sec bandwidth
  • 350 usec RTT latency
  • Socket Abstraction

– using LWIP RAW interface Ethernet

slide-25
SLIDE 25

SRAM Controller Multi-Port Memory Controller

Reference Design Block Diagram

25

PLB

BlueSPARC MicroBlaze Ethernet BRAM Serial Port BRAM BRAM

XUPv5

LX110T DRAM SRAM

slide-26
SLIDE 26

SRAM Controller

DDR2 Memory Controller

26

PLB

BlueSPARC MicroBlaze Ethernet BRAM Serial Port BRAM BRAM

XUPv5

LX110T DRAM SRAM

Multi-Port Memory Controller

  • 1.5GB/s peak BW
  • 115ns latency
  • Multiple ports/interfaces
slide-27
SLIDE 27

Required Equipment

27

FPGA Board Linux PC Ethernet BlueSPARC PFMON + Simics

slide-28
SLIDE 28

Required Equipment

28

FPGA Board Linux PC Ethernet BlueSPARC PFMON + Simics

slide-29
SLIDE 29

29

Linux PC

  • Software requirements

– SuSE Linux 10.1 – CAD tools + licenses (Bluespec compiler, Xilinx ISE/EDK) – Simics 3.0.22 – Hybrid simulation plug-in modules – ProtoFlex MONitor tool (PFMON)

slide-30
SLIDE 30

30

Linux PC

  • Runs PFMON (ProtoFlex MONitor)

– Orchestrates communication between Simics & BlueSPARC – Provides CLI interface to simulator (like Simics Console)

  • Runs Simics

– Handles I/O, FPGA Core and memory initialization

BlueSPARC PFMON

Linux PC

Simics

slide-31
SLIDE 31

31

From RTL to Running System

  • Bluespec  Verilog

– Bluespec compiler – ~30 minutes

  • Verilog  Bitstream

– Xilinx EDK – ~ 3 hours

  • Bitstream  Working System

– Stream mem. image over ethernet – ~ 5 minutes (for 512MB image)

Bluespec code Verilog code Bitstream

1 2 3 1 2

Working System

3

slide-32
SLIDE 32

XUPv5 Caveats

  • Limitations

– Due to BRAM limits, only 4 CPU contexts (compared to 16 on BEE2’s V2P70) – Slow PC-to-FPGA latency via LwIP/Ethernet (1 key/sec in terminal) – Limited DDR2 RAM capacity (up to 2GB)

  • However…

– Our XUPv5 design still has much room for improvement – Useful for familiarizing with ProtoFlex tools – Can still run multithreaded workloads + perform monitoring – Easily portable to more powerful platforms

32

slide-33
SLIDE 33

Outline

  • Motivation
  • The ProtoFlex Simulator
  • Using ProtoFlex
  • XUPv5 Reference Design
  • Distribution

33

slide-34
SLIDE 34

Distribution Details

  • Release dates

– All source code will be available in 1 week (June) – Send email to echung@ece.cmu.edu for more info

  • Licensing

– GNU GPL

  • Support

– Documentation: www.ece.cmu.edu/~protoflex – Support mailing list/forum will be available soon

34

slide-35
SLIDE 35

Thank you!

  • Acknowledgments

– Funding for this work provided by: NSF CCF-0811702, NSF CNS- 0509356, C2S2 Marco Center, and Sun Microsystems – Paul Hartke & Xilinx for FPGA donations & email support

  • Topics & References

– Hybrid Simulation and FPGA Core Multithreading

A Complexity-Effective Architecture for Accelerating Full-System Multiprocessor Simulations Using FPGAs [FPGA’08]

– Functional-First, First-Order Timing Model for a CMP cache

ProtoFlex: Towards Scalable, Full-System Multiprocessor Simulations Using FPGAs [TRETS’09]

– FPGA Core Design & Validation Using Flight Data Recorder

Implementing a High-performance Multithreaded Microprocessor: A Case Study in High- level Design and Validation [MEMOCODE’09]

35