scaling the cascades
play

Scaling the Cascades Interconnect-aware FPGA implementation of - PowerPoint PPT Presentation

Sep 05, 2023 •162 likes •993 views

BRAM DSP URAM Scaling the Cascades Interconnect-aware FPGA implementation of Machine Learning problems Anand Samajdar, Tushar Garg, Tushar Krishna, Nachiket Kapre nachiket@uwaterloo.ca Claim Hard FPGA interconnect (cascades) e ffi ciently

  1. BRAM DSP URAM Scaling the Cascades Interconnect-aware FPGA implementation of Machine Learning problems Anand Samajdar, Tushar Garg, Tushar Krishna, Nachiket Kapre nachiket@uwaterloo.ca

  4. Claim • Hard FPGA interconnect (cascades) e ffi ciently supports nearest neighbour communication + reuse in ML workloads • Three kinds of UltraScale+ cascades [DSP , BRAM, URAM] • Combination of (1) pixel, (2) row, (3) map reuse • Deliverables: • 650 MHz full-chip operation • 7x better latency, 30% lower throughput than the formidable Xilinx SuperTile design for GoogLeNet v1 � 4

  5. Landscape of FPGA+ML accelerators � 5

  6. Communication Requirements of 3x3 Convolution Output Map J x x x Input Row k Input Row k+1 x x x + Output Row k Input Row k+2 x x x Weights Input Map I � 6

  7. Communication Requirements of 3x3 Convolution 2 1 Output 3 Map J row streaming pixel streaming x x x Input Row k channel streaming Input Row k+1 x x x + Output Row k Input Row k+2 x x x Weights Input Map I � 7

  8. Reuse Patterns Output Map J x x x Input Row k + Input Row k+1 x x x Output Row k Input Row k+2 x x x Input Map I Weight Row 0 Weight Row 1 Weight Row 2 Input Row k+1 Input Row k+2 Input Row k x+ x+ x+ x+ x+ x+ x+ x+ x+ P cascade for summation pixel streaming � 8

  9. Reuse Patterns Output Map J x x x Input Row k + Input Row k+1 x x x Output Row k Input Row k+2 x x x Input Map I Weight Row 0 Weight Row 1 Weight Row 2 Input Row k+1 Input Row k+2 Input Row k B cascade for weight streaming A cascade for x+ x+ x+ x+ x+ x+ x+ x+ x+ pixel streaming P cascade for summation 1 pixel streaming � 9

  10. Reuse Patterns Output Map J x x x Input Row k + Input Row k+1 x x x Output Row k Input Row k+2 x x x row streaming 2 Input Map I Weight Row 0 Weight Row 1 Weight Row 2 Exploit Input Row k+1 Input Row k+2 Input Row k Data Reuse B cascade for weight streaming A cascade for x+ x+ x+ x+ x+ x+ x+ x+ x+ pixel streaming P cascade for summation 1 pixel streaming � 10

  11. Reuse Patterns Output Map J x x x Input Row k + Input Row k+1 x x x Output Row k Input Row k+2 x x x row streaming 2 Input Map I Weight Row 0 Weight Row 1 Weight Row 2 Exploit Input Row k+1 Input Row k+2 Input Row k Data Reuse Input Map I B cascade for weight streaming A cascade for x+ x+ x+ x+ x+ x+ x+ x+ x+ pixel streaming P cascade for summation 1 pixel streaming Output Map J � 11

  12. Reuse Patterns Output Map J x x x Input Row k + Input Row k+1 x x x Output Row k Input Row k+2 x x x Input Map I Input Map I Weights 3x3 Convolution Tile Output Map J � 12

  13. Reuse Patterns Weights Input Output 3x3 Convolution Tile Map I Map J Weights Input 3x3 Convolution Tile Map I+1 Weights Input 3x3 Convolution Tile Map I+.. � 13

  14. Reuse Patterns Weights Input Output 3x3 Convolution Tile Map I Map J channel streaming Weights Input 3x3 Convolution Tile Map I+1 3 Weights Input 3x3 Convolution Tile Map I+.. � 14

  15. Reuse Patterns Weights Input Output 3x3 Convolution Tile Map I Map J channel streaming Weights Input Map I+1 3 Weights Input Map I+.. � 15

  16. Xilinx UltraScale+ FPGA Cascades • BRAM18 support A/B row streaming pixel streaming cascades 2x72b-wide links • DSP48 supports A, B, P channel streaming 2 1 cascades (systolic input and summation) BRAM 3 • URAM288 supports A, B DSP cascades URAM � 16

  17. Outline • Understanding Cascades • Assembling the FPGA accelerator + FPGA Layout • MLPerf Evaluation • Conclusions + Discussion � 17

  18. Promise of Cascades • Absorb data movement onto dedicated interconnect vs. General-purpose wiring • Higher clock frequency operation, layout-friendly architecture � 18

  19. Our approach • Exploit cascades aggressively! • DSP48 • For 3x3 convolution, length-9 cascades • P cascade for summation (like INT8 paper) • A cascade for systolic retiming (like DSP48E2 user guide) • B cascades for weights (our contribution) 19

  20. Our approach • Exploit cascades aggressively! • RAMB18E2 (our contribution) • For 3x3 convolution, only need 3 BRAM-long chains • A/B cascade for shift operation • Swap between A and B to keep one read port available. 20

  21. Our approach • Exploit cascades aggressively! • URAM288 (our contribution) • Alternating A/B cascades of length-2 • Both data + addresses cascades • Shift operation tricky! • Due to 72b width, and resource ratios, idle cycles available for realizing shifts 21

  22. Putting it together x x x x x x x x x + + + + + + + + + DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 22

  23. Putting it together x x x x x x x x x + + + + + + + + + DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 23

  24. Putting it together Weights (initial shift) x x x x x x x x x + + + + + + + + + DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 24

  25. Putting it together Weights Pixel streaming (initial shift) x x x x x x x x x + + + + + + + + + DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 25

  26. Putting it together Row i+2 Row i+1 Row i RAMB RAMB RAMB 18 (C) 18 (B) 18 (A) x x x x x x x x x + + + + + + + + + DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 26

  27. Putting it together RAMB RAMB RAMB Row streaming 18 (C) 18 (B) 18 (A) x x x x x x x x x + + + + + + + + + DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 27

  28. Putting it together to next URAM RAMB RAMB RAMB 18 (C) 18 (B) 18 (A) URAM 288 (Input) (Weights) x x x x x x x x x from previous URAM + + + + + + + + + RAMB 18 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 (Kern) 28

  29. Putting it together to next URAM RAMB RAMB RAMB 18 (C) 18 (B) 18 (A) URAM 288 (Input) (Weights) x x x x x x x x x from previous URAM + + + + + + + + + RAMB 18 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 (Kern) Map streaming 29

  30. Putting it together to next URAM RAMB RAMB RAMB 18 (C) 18 (B) 18 (A) URAM URAM 288 288 (Input) (Output) (Weights) + x x x x x x x x x from previous URAM + + + + + + + + + RAMB 18 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 (Kern) 30

  31. Putting it together row streaming 2 to next URAM channel streaming RAMB RAMB RAMB 18 (C) 18 (B) 18 (A) URAM URAM 288 288 (Input) (Output) (Weights) + x x x x x x x x x from previous URAM 3 1 pixel streaming + + + + + + + + + RAMB 18 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 DSP48 (Kern) 31

  32. A 3x3 tile layout � 32

  33. A 3x3 tile layout � 33

  34. A 3x3 tile layout Places and routes at 1.2ns � 34

  35. Tiling the design • VU37P device has specific resource mix • For each URAM, you get 4.2 BRAMs and 9.4 DSP48s • Repeating pattern must conform to this ratio • One tile: 2 URAMs, 8 BRAMs, 18 DSPs • Physical layout XDC constraints must account for irregular column arrangement of hard resources � 35

  36. Tiling the design • VU37P device has specific resource mix • For each URAM, you get 4.2 BRAMs and 9.4 DSP48s • Repeating pattern must conform to this ratio • One tile: 2 URAMs, 8 BRAMs, 18 DSPs • Physical layout XDC constraints must ONE TILE account for irregular column arrangement of hard resources � 36

  37. Matrix-Matrix Multiplication • Limited Reuse opportunities • Split large matrix across URAMs • Each URAM stores a set of complete rows —> allows result vector to be independently processed. • Partial vector results then circulated across the chip in a ring-like fashion —> using BRAM cascades • URAM cascades only used for loading matrix at start � 37

  38. VU37P Layout � 38

  39. VU37P FPGA Mapping CONVOLUTION MATRIX-MULTIPLY � 39

  40. VU37P FPGA Mapping 80% 20% CONVOLUTION MATRIX-MULTIPLY � 40

  41. VU37P FPGA Mapping 80% 20% � 41

  42. Effect of using cascades • Registers in hard interconnect Cascade No Cascade save us fabric registers for other Route Count 75K pipelining needs 50K • Clock period marginally better 25K • Obvious reduction in interconnect 0 20 40 60 80 utilization Congestion (%) � 42

  43. Effect of using cascades • Registers in hard interconnect Cascade No Cascade save us fabric registers for other Route Count 75K pipelining needs 50K • Clock period marginally better 25K • Obvious reduction in interconnect 0 20 40 60 80 utilization Congestion (%) � 43

  44. Evaluation Methodology • We use the SCALE-Sim cycle- accurate simulator • https://github.com/ARM- software/SCALE-Sim • Map URAMs -> IFMAP/ OFMAP SRAMs • BRAMs and DSP cascades => systolic array links • VU37P can fit systolic array of size 960x9 (conv), 480x9 (mm) 44

  45. Xilinx SuperTile • GoogLeNet v1 mapped to VU9P FPGA (Amazon F1) • 3046 images/s + 3.3ms latency • Scorching 720 MHz operation! • Mind-numbing 88% overall e ffi ciency http://isfpga.org/slides/Compute-E ffi cient_Neural-Network_Acceleration.pdf 45

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Optimizing cascades & submodular optimization Rik Sarkar Today Maximizing cascades

Optimizing cascades & submodular optimization Rik Sarkar Today Maximizing cascades

Optimizing cascades & submodular optimization Rik Sarkar Today Maximizing cascades Other applications of submodularity Network flows NP completeness Recap: Selecting nodes to activate We have a network of n nodes And

565 views • 33 slides
Information Cascades in Human Networks Milo Trujillo Professor Gao Information Cascades

Information Cascades in Human Networks Milo Trujillo Professor Gao Information Cascades

Information Cascades in Human Networks Milo Trujillo Professor Gao Information Cascades Generalization of Infection Modeling Infection based on % threshold of neighbors Agent-Based model to examine heterogeneous networks Discrete

784 views • 20 slides
Analysis of Scaling Algorithms for Matrix & Operator Scaling Contents Scaling Algorithms

Analysis of Scaling Algorithms for Matrix & Operator Scaling Contents Scaling Algorithms

Rafael Oliveira University of Toronto Analysis of Scaling Algorithms for Matrix & Operator Scaling Contents Scaling Algorithms Three Step Analysis Generalization One More Application of Scaling Non-Negative Matrices &

604 views • 15 slides
Cascades Social and Technological Networks Rik Sarkar University of Edinburgh, 2019. Network

Cascades Social and Technological Networks Rik Sarkar University of Edinburgh, 2019. Network

Cascades Social and Technological Networks Rik Sarkar University of Edinburgh, 2019. Network cascades Things that spread (diffuse) along network edges Epidemics Ideas Innovation: We use technology our friends/colleagues use

462 views • 30 slides
Scaling From simple models to rich strategies PPPLab Day, November 30th Scaling: recent

Scaling From simple models to rich strategies PPPLab Day, November 30th Scaling: recent

Scaling From simple models to rich strategies PPPLab Day, November 30th Scaling: recent publications Insight Series Tool Animation The rationale for scaling As the SDGs require transformational change, scaling can provide: Reaching more

609 views • 37 slides
ETI se Scaling for many organizations Hierarchy Recursive vs T.to eRaokon traverses the

ETI se Scaling for many organizations Hierarchy Recursive vs T.to eRaokon traverses the

Discovery Naming or IN 5 Why name 172.217 11.14 or google.com Indirection Security Isafety concerns How name iEE f google.com Et Challenge Scaling Scaling for many clients Caching D y D ETI se Scaling for many organizations Hierarchy

215 views • 4 slides
BlackBerry 10 Cascades UI FW: A Different Take Markus Landin, Product Manager, Research In Motion

BlackBerry 10 Cascades UI FW: A Different Take Markus Landin, Product Manager, Research In Motion

BlackBerry 10 Cascades UI FW: A Different Take Markus Landin, Product Manager, Research In Motion TAT November 19, 2012 November 19, 2012 Session absctract: Cascades is the new native UI Framework on BlackBerry 10 Platform. It is built

631 views • 44 slides
Outline Scaling Scalinga Plenitude of Power Laws Scaling-at-large Scaling-at-large

Outline Scaling Scalinga Plenitude of Power Laws Scaling-at-large Scaling-at-large

Scaling Outline Scaling Scalinga Plenitude of Power Laws Scaling-at-large Scaling-at-large Scaling-at-large Principles of Complex Systems Allometry Allometry Definitions Definitions Course 300, Fall, 2008 Examples Examples

568 views • 27 slides
So#ware Scaling Mo/va/on & Goals HW Configura/on & Scale Out So#ware Scaling

So#ware Scaling Mo/va/on & Goals HW Configura/on & Scale Out So#ware Scaling

So#ware Scaling Mo/va/on & Goals HW Configura/on & Scale Out So#ware Scaling Efforts System management Opera/ng system Programming environment PreAcceptance Work HW stabiliza/on & early scaling

390 views • 19 slides
UMBC A B M A L T F O U M B C I M Y O R T 1 (10/18/04) I E S R C E O V

UMBC A B M A L T F O U M B C I M Y O R T 1 (10/18/04) I E S R C E O V

Advanced VLSI Design Scaling CMPE 640 Scaling Technology scaling rate is approximately 13%/year, halving every 5 years. 100 10 m 1 0.1 1960 1970 1980 1990 2000 The size of the circuits also continues to increase. Besides increasing

362 views • 7 slides
Weighted Classification Cascades for Optimizing Discovery Significance Lester Mackey

Weighted Classification Cascades for Optimizing Discovery Significance Lester Mackey

Weighted Classification Cascades for Optimizing Discovery Significance Lester Mackey Collaborators: Jordan Bryan and Man Yue Mo Stanford University December 13, 2014 Mackey (Stanford) Weighted Classification Cascades December 13,

591 views • 15 slides
CASCADES INC. NBF Qubec Conference June 5, 2013 INTRODUCTION Where we come from

CASCADES INC. NBF Qubec Conference June 5, 2013 INTRODUCTION Where we come from

CASCADES INC. NBF Qubec Conference June 5, 2013 INTRODUCTION Where we come from Unique culture green visionaries, opportunistic turnarounds, importance of autonomy and empowerment, open book and profit sharing, decentralized

533 views • 36 slides
Information Cascades - Also following the crowd or herding - Example: Choosing a

Information Cascades - Also following the crowd or herding - Example: Choosing a

Information Cascades - Also following the crowd or herding - Example: Choosing a restaurant in an unfamiliar town you do your research, and plan to go to restaurant A When you arrive, A is empty and B next door is full if you believe

587 views • 3 slides
Outline Scalinga Plenitude of Power Laws Scaling-at-large Scaling-at-large Principles of

Outline Scalinga Plenitude of Power Laws Scaling-at-large Scaling-at-large Principles of

Scaling Scaling Outline Scalinga Plenitude of Power Laws Scaling-at-large Scaling-at-large Principles of Complex Systems Allometry Allometry CSYS/MATH 300, Fall, 2010 Definitions Definitions Examples Examples History: Metabolism

250 views • 20 slides
Cascades Recovery Inc. We care so much about paper and packaging; when youre done with it, we

Cascades Recovery Inc. We care so much about paper and packaging; when youre done with it, we

A Model EPR Program for Printed Paper and Packaging Cascades Recovery Inc. We care so much about paper and packaging; when youre done with it, we want it back A Model EPR Program for Printed Paper and Packaging ! We are brand owners Cascades

460 views • 20 slides
Colour Dipole Cascades Saturation, Correlations, and Fluctuations in pp and pA collisions Gsta

Colour Dipole Cascades Saturation, Correlations, and Fluctuations in pp and pA collisions Gsta

Content Introduction Unitarity constraints Colour Dipole Cascades Saturation, Correlations, and Fluctuations in pp and pA collisions Gsta Gustafson Department of Theoretical Physics Lund University Multiple Partonic Interactions

674 views • 48 slides
Object Detection using Haar like Features CS 395T: Visual Recognition and Search Harshdeep

Object Detection using Haar like Features CS 395T: Visual Recognition and Search Harshdeep

Object Detection using Haar like Features CS 395T: Visual Recognition and Search Harshdeep Singh The Detector Using boosted cascades of Haar like features Proposed by [Viola, Jones 2001] Implementation available in OpenCV

736 views • 32 slides
Willump: A Statistically-Aware End-to-end Optimizer for ML Inference Peter Kraft , Daniel Kang,

Willump: A Statistically-Aware End-to-end Optimizer for ML Inference Peter Kraft , Daniel Kang,

Willump: A Statistically-Aware End-to-end Optimizer for ML Inference Peter Kraft , Daniel Kang, Deepak Narayanan, Shoumik Palkar, Peter Bailis, Matei Zaharia 1 Problem: ML Inference Often performance-critical. Recent focus on tools for

959 views • 70 slides
Tree Recursion Announcements Recursive Factorial factorial (!) if n == 0 n! = 1 if n > 0

Tree Recursion Announcements Recursive Factorial factorial (!) if n == 0 n! = 1 if n > 0

Tree Recursion Announcements Recursive Factorial factorial (!) if n == 0 n! = 1 if n > 0 n! = n x (n-1) x (n-2) x ... x 1 factorial(5) def factorial(n): fact = 1 i = 1 while i <= n: 1 = 1*1 fact *= i 2 = 2*1! i += 1 6 =

498 views • 27 slides
On the well-posedness of cascades of analytic nonlinear input-output systems driven by noise

On the well-posedness of cascades of analytic nonlinear input-output systems driven by noise

On the well-posedness of cascades of analytic nonlinear input-output systems driven by noise Luis A. Duffaut Espinosa Department of Electrical and Computer Engineering The Johns Hopkins University, Baltimore, Maryland USA This research

637 views • 45 slides
Outline Contagion Contagion Basic Contagion Basic Contagion Models Models Complex Networks,

Outline Contagion Contagion Basic Contagion Basic Contagion Models Models Complex Networks,

Contagion Outline Contagion Contagion Basic Contagion Basic Contagion Models Models Complex Networks, Course 295A, Spring, 2008 Social Contagion Social Contagion Models Models Basic Contagion Models Granovetters model

298 views • 16 slides
Online Learning to Rank with Features Authors: Shuai Li, Tor Lattimore, Csaba Szepesvri The

Online Learning to Rank with Features Authors: Shuai Li, Tor Lattimore, Csaba Szepesvri The

Online Learning to Rank with Features Authors: Shuai Li, Tor Lattimore, Csaba Szepesvri The Chinese University of Hong Kong DeepMind University of Alberta Learning to Rank Amazon, YouTube, Facebook, Netflix, Taobao 1 Online Learning to

530 views • 17 slides
IN WHATSAPP GROUPS Josemar Alves Caetano, Gabriel Magno, Marcos Gonalves, Jussara Almeida,

IN WHATSAPP GROUPS Josemar Alves Caetano, Gabriel Magno, Marcos Gonalves, Jussara Almeida,

CHARACTERIZING ATTENTION CASCADES IN WHATSAPP GROUPS Josemar Alves Caetano, Gabriel Magno, Marcos Gonalves, Jussara Almeida, Humberto T. Marques-Neto, Virglio Almeida WebSci 2019 2 Messenger apps popularity App Total users WhatsApp 1.6

679 views • 20 slides
Projections of random fractals and measures and Liouville quantum gravity Kenneth Falconer

Projections of random fractals and measures and Liouville quantum gravity Kenneth Falconer

Projections of random fractals and measures and Liouville quantum gravity Kenneth Falconer University of St Andrews, Scotland, UK Joint with Xiong Jin (Manchester) Kenneth Falconer Projections of random fractals and measures and Liouville

539 views • 43 slides

More recommend