What can in-memory computing deliver, and what are the barriers? - - PowerPoint PPT Presentation

▶
what can in memory computing deliver and what are the
SMART_READER_LITE
LIVE PREVIEW

What can in-memory computing deliver, and what are the barriers? - - PowerPoint PPT Presentation

Oct 12, 2023 160 likes •366 views

What can in-memory computing deliver, and what are the barriers? Naveen Verma (nverma@princeton.edu), L.-Y. Chen, H. Jia, M. Ozatay, Y. Tang, H. Valavi, B. Zhang, J. Zhang March 20 th , 2019 The memory wall Separating memory from compute

slide-1
SLIDE 1

What can in-memory computing deliver, and what are the barriers?

Naveen Verma (nverma@princeton.edu), L.-Y. Chen, H. Jia, M. Ozatay, Y. Tang, H. Valavi, B. Zhang, J. Zhang March 20th, 2019

slide-2
SLIDE 2

The memory wall

MULT (INT8): 0.3pJ MULT (INT32): 3pJ MULT (FP32): 5pJ MULT (INT4): 0.1pJ

Memory Size (!) Energy per Access 64b Word (pJ)

  • Separating memory from compute fundamentally raises a communication cost

More data → bigger array → larger comm. distance → more comm. energy

slide-3
SLIDE 3

So, we should amortize data movement

EMEM

  • Comp. Intensity

OPS/WCOMP =

Memory Bound Compute Bound

  • Specialized (memory-compute integrated)

architectures

!" ⋮ !$ = &"," … &",) ⋮ ⋱ ⋮ &$," ⋯ &$,) ," ⋮ ,)

Processing Element (PE)

  • Reuse accessed data for compute
  • perations

⃗ ! = . × ,

slide-4
SLIDE 4

In-memory computing (IMC)

IMC Mode SRAM Mode

! "

[J. Zhang, VLSI’16][J. Zhang, JSSC’17]

  • In SRAM mode, matrix A stored in bit

cells row-by-row

  • In IMC mode, many WLs driven

simultaneously → amortize comm. cost inside array

  • Can apply to diff. mem. Technologies

→ enhanced scalability → embedded non-volatility

#$ ⋮ #& = ($,$ … ($,+ ⋮ ⋱ ⋮ (&,$ ⋯ (&,+ .$ ⋮ .+

⃗ # = 0. ⇒

slide-5
SLIDE 5

The basic tradeoffs

CONSIDER: Accessing ! bits of data associated with computation, from array with ! columns ⨉ ! rows.

Memory

(D1/2×D1/2 array)

Computation Memory & Computation

(D1/2×D1/2 array)

D1/2

Traditional IMC

Metric Traditional In-memory Bandwidth 1/ 1/D1/

1/2

1 Latency D 1 Energy D3/

3/2 2

~D ~D SNR 1 ~1 ~1/D1/

1/2

  • IMC benefits energy/delay at cost of SNR
  • SNR-focused systems design is critical

(circuits, architectures, algorithms)

slide-6
SLIDE 6

IMC as a spatial architecture

! " = $! Data Movement:

  • 1. %&,(′* broadcast min. distance

due to high-density bit cells

  • 2. (Many) +,,&′* stationary

in high-density bit cells

  • 3. High-dynamic-range analog -,,(′*

computed in distributed manner

slide-7
SLIDE 7

IMC as a spatial architecture

Operation Digital-PE Energy (fJ) Bit-cell Energy (fJ) Storage 250 50 Multiplication 100 Accumulation 200 Communication 40 5 Total 590 55

Assume:

  • 1k dimensionality
  • 4-b multiplies
  • 45nm CMOS

PRE

c11(23)

PRE PRE PRE

a11 [3] b11 [3] a11 [2] a11 [1] a11 [0] a12 [3] a12 [2] a12 [1] a12 [0] b21 [3] c11(22) c11(21) c11(20)

slide-8
SLIDE 8

Where does IMC stand today?

Energy Efficiency (TOPS/W) Normalized Throughput (GOPS/mm2)

Bankman, ISSCC’18, 28nm Yuan, VLSI’18, 65nm Moons, ISSCC’17, 28nm Ando, VLSI’17, 65nm Chen, ISSCC’16, 65nm Gonug, ISSCC’18, 65nm Biswas, ISSCC’18, 65nm Jiang, VLSI’18, 65nm

10 10e2 10e3 10e4

Valavi, VLSI’18, 65nm Khwa, ISSCC’18, 65nm Zhang, VLSI’16, 130nm Lee, ISSCC’18, 65nm Shin, ISSCC’17, 65nm Yin, VLSI’17, 65nm

10e-2 10e-1 1 10 10e2 10e3

Energy Efficiency (TOPS/W) On-chip Memory Size (kB)

Yuan, VLSI’18, 65nm

10e3 10e2 10 1 10e-2 10e-1 1 10 10e2 10e3

Valavi, VLSI’18, 65nm Bankman, ISSCC’18, 28nm Lee, ISSCC’18, 65nm Zhang, VLSI’16, 130nm Khwa, ISSCC’18, 65nm Jiang, VLSI’18, 65nm Biswas, ISSCC’18, 65nm Gonug, ISSCC’18, 65nm Chen, ISSCC’16, 65nm Yin, VLSI’17, 65nm Ando, VLSI’17, 65nm Moons, ISSCC’17, 28nm

IMC Not IMC

  • Potential for 10× higher efficiency &

throughput

  • Limited scale, robustness, configurability
slide-9
SLIDE 9

Challenge 1: analog computation

  • Use analog circuits to ‘fit’ compute in bit cells

⟶ SNR limited by analog-circuit non-idealities ⟶ Must be feasible/competitive @ 16/12/7nm

VBIAS,O VBIAS 1x 2x 16x MA,R MD,R CLASS_EN X[0] X[1] X[4] WL_RESET WL XOffset BL BLB

MA MD

Bit-cell replica I-DAC Bit-cell 0.02 0.04 0.06 WLDAC Code

ΔVBL (V)

5 10 15 20 25 30 35

Ideal transfer curve Nominal transfer curve [J. Zhang, VLSI’16][J. Zhang, JSSC’17]

slide-10
SLIDE 10

Algorithmic co-design(?)

  • •
  • WEAK classifier K

WEAK classifier 2 Weighted Voter

Classifier Trainer

WEAK classifier 1

Feature 1 Feature 2

  • • •
  • •
  • •
  • • •
  • Chip-specific weight tuning

[Z. Wang, TVLSI’15] [Z. Wang, TCAS-I’15] [S. Gonu., ISSCC’18]

  • Chip-generalized weight tuning

G gi

Training Inference

Parameters $(&, (, ℒ)

Normalized MRAM cell standard dev.

1 2 3 4 5 6 7 8 9 10 10 20 30 40 50 60 70 80 90 100

Accuracy

L = |- − /

  • (&, $)|0

L = |- − /

  • (&, $, ()|0

E.g.: BNN Model (applied to CIFAR-10)

[B. Zhang, ICASSP 2019]

slide-11
SLIDE 11

Challenge 2: programmability

[B. Fleischer, VLSI’18]

General Matrix Multiply (~256⨉2300=590k elements) Single/few-word operands (traditional, near-mem. acceleration)

  • Matrix-vector multiply is only 70-90% of operations

⟶ IMC must integrate in programmable, heterogenous architectures

slide-12
SLIDE 12

Challenge 3: efficient application mappings

  • IMC engines must be ‘virtualized’

⟶ IMC amortizes MVM costs, not weight loading. But… ⟶ Need new mapping algorithms (physical tradeoffs very diff. than digital engines)

(output activations) "#,%,&

'

(N - I⨉J⨉K filters) )*+,,,- (X⨉Y⨉Z input activations)

  • EDRAM→IMC/4-bit: 40pJ
  • Reuse: .×0×1 (10-20 lyrs)
  • EMAC,4-b: 50fJ

Activation Accessing Weight Accessing

  • EDRAM→IMC/4-bit: 40pJ
  • Reuse: 2×3
  • EMAC,4-b:50fJ

Reuse ≈ 1k

Memory Bound Compute Bound

slide-13
SLIDE 13

Path forward: charge-domain analog computing

1. Digital multiplication 2. Analog accumulation

[H. Valavi, VLSI’18]

~1.2fF metal capacitor (on top of bit cell)

slide-14
SLIDE 14

2.4Mb, 64-tile IMC

Moons, ISSCC’17 Bang, ISSCC’17 Ando, VLSI’17 Bankman, ISSCC’18 Valavi, VLSI’18

Technology

28nm 40nm 65nm 28nm 65nm

Area (!!")

1.87 7.1 12 6 17.6

Operating VDD

1 0.63-0.9 0.55-1 0.8/0.8 (0.6/0.5) 0.94/0.68/1.2

Bit precision

4-16b 6-32b 1b 1b 1b

  • n-chip Mem.

128kB 270kB 100kB 328kB 295kB

Throughput (GOPS)

400 108 1264 400 (60) 18,876

TOPS/W

10 0.384 6 532 (772) 866

  • 10-layer CNN demos for MNIST/CIFAR-10/SVHN at energies of

0.8/3.55/3.55 μJ/image

  • Equivalent performance to software implementation

[H. Valavi, VLSI’18]

slide-15
SLIDE 15

Programmable IMC

CPU (RISC-V) AXI Bus DMA Timers GPIO UART 32 Program Memory (128 kB) Boot- loader Data Memory (128 kB) Compute-In-Memory Unit (CIMU)

  • 590 kb
  • 16 bank

Ext.

  • Mem. I/F

Config. Regs. To E2PROM To DRAM Controller

Config

APB Bus 32 32 Tx Rx 8 13 (data) (addr.) 32 (data/addr.)

[H. Jia, arXiv:1811.04047]

w2b Reshaping Buffer Sparsity/AND-logic Controller

x

Data Mask Row Decoder/ WL Drivers Memory Read/Write I/F

<0> <767>

x0

32b <0> <255> 8b Near-Mem. Data Path <0> Near-Mem. Data Path <31> 32b ADC & ABN <63> <192>

A

32b

Bit Cell

ADC & ABN

f(y = A x)

Compute-In- Memory Array (CIMA)

xb0 x2303 xb2303

slide-16
SLIDE 16

Bit-scalable mixed-signal compute

10 20 30 40 6 10 14 18 2 3 4 5 6 7 8 2 4 6

BA SQNR (dB)

Bx=2 Bx=4 Bx=8

N=2304, 2000, 1500, 1000, 500, 255 N=2304, 2000, 1500, 1000, 500, 255 N=2304, 2000, 1500, 1000, 500, 255

  • SQNR different that standard integer

compute

[H. Jia, arXiv:1811.04047]

slide-17
SLIDE 17

Development board

To Host Processor

slide-18
SLIDE 18

Design flow

  • 1. Deep-learning Training Libraries

(Keras)

  • 2. Deep-learning Inference Libraries

(Python, MATLAB, C)

Dense(units, ...) Conv2D(filters, kernel_size, ...) ...

Standard Keras libs:

QuantizedDense(units, nb_input=4, nb_weight=4, chip_quant=True, ...) QuantizedConv2D(filters, kernel_size, nb_input=4, nb_weight=4, chip_quant=True, ...) ... QuantizedDense(units, nb_input=4, nb_weight=4, chip_quant=False, ...) QuantizedConv2D(filters, kernel_size, nb_input=4, nb_weight=4, chip_quant=False, ...) ...

Custom libs: (INT/CHIP quant.)

chip_mode = True

  • utputs = QuantizedConv2D(inputs,

weights, biases, layer_params)

  • utputs = BatchNormalization(inputs,

layer_params) ...

High-level network build (Python): Embedded C: Function calls to chip (Python):

chip.load_config(num_tiles, nb_input=4, nb_weight=4) chip.load_weights(weights2load) chip.load_image(image2load)

  • utputs = chip.image_filter()

chip_command = get_uart_word(); chip_config(); load_weights(); load_image(); image_filter(chip_command); read_dotprod_result(image_filter_command);

slide-19
SLIDE 19

Demonstrations

2 4 6 8 2 4 6 8 2 4 6 8 5 10 15 20 SQNR (dB)

Multi-bit Matrix-Vector Multiplication

N=1152 Bit-true Sim. N=1728 Measured N=1152 N=1728

Bx=2

BA

Bx=4 20 40 60 80

  • 500

500 20 40 60 80

  • 60
  • 40
  • 20

20

Data Index

Compute Value Bx=2, BA=2 Bx=4, BA=4

Bit True Sim. Measured

BA Data Index

Neural-Network Demonstrations

Network A

(4/4-b activations/weights)

Network B

(1/1-b activations/weights)

Accuracy of chip

(vs. ideal)

92.4%

(vs. 92.7%)

89.3%

(vs. 89.8%)

Energy/10-way Class.1 105.2 μJ 5.31 μJ Throughput1 23 images/sec. 176 images/sec. Neural Network Topology

L1: 128 CONV3 – Batch norm L2: 128 CONV3 – POOL – Batch norm. L3: 256 CONV3 – Batch. norm L4: 256 CONV3 – POOL – Batch norm. L5: 256 CONV3 – Batch norm. L6: 256 CONV3 – POOL – Batch norm. L7-8: 1024 FC – Batch norm. L9: 10 FC – Batch norm. L1: 128 CONV3 – Batch Norm. L2: 128 CONV3 – POOL – Batch Norm. L3: 256 CONV3 – Batch Norm. L4: 256 CONV3 – POOL – Batch Norm. L5: 256 CONV3 – Batch Norm. L6: 256 CONV3 – POOL – Batch Norm. L7-8: 1024 FC – Batch norm. L9: 10 FC – Batch norm.

DMEM PMEM

CPU

CIMU ADC ABN DMA etc.

W2b Reshaping Buffer

4×4 CIMA Tiles 3mm 4.5mm

Near-mem. Datapath

Sparsity Controller

[H. Jia, arXiv:1811.04047]

slide-20
SLIDE 20

Conclusions

Matrix-vector multiplies (MVMs) are a little different than other computations ⟶ high-dimensionality operands lead to data movement (memory accessing) Bit cells make for dense, energy-efficient PE’s in spatial array ⟶ but require analog operation to fit compute, and impose SNR tradeoff Must focus on SNR tradeoff to enable scaling (technology/platform) and architectural integration In-memory computing greatly affects the architectural tradeoffs, requiring new strategies for mapping applications

Acknowledgements: funding provided by ADI, DARPA, NRO, SRC/STARnet