Noisy In-Memory Recursive Computation with Memristor Crossbars Elsa - - PowerPoint PPT Presentation

▶
noisy in memory recursive computation with memristor
SMART_READER_LITE
LIVE PREVIEW

Noisy In-Memory Recursive Computation with Memristor Crossbars Elsa - - PowerPoint PPT Presentation

Jan 05, 2023 145 likes •369 views

Noisy In-Memory Recursive Computation with Memristor Crossbars Elsa Dupraz , Lav Varshney elsa.dupraz@imt-atlantique.fr IMT Atlantique, Lab-STICC, UBL varshney@illinois.edu University of Illinois at Urbana-Champaign Funded by

slide-1
SLIDE 1

Noisy In-Memory Recursive Computation with Memristor Crossbars

Elsa Dupraz†, Lav Varshney‡

†elsa.dupraz@imt-atlantique.fr

IMT Atlantique, Lab-STICC, UBL

‡varshney@illinois.edu

University of Illinois at Urbana-Champaign

Funded by Thomas Jefferson fund and by ANR project EF-FECtive

slide-2
SLIDE 2

Section 1: Introduction 2

Computation in memory

Introduction Dot-product computation Iterative dot-product computation Simulation results Conclusion

◮ Data transfer bottleneck in conventional setups :

Memory Processing units bottleneck

+

Memory banks

◮ In-memory computing :

Memory

Memory banks Computationnal memory

Processing units

+ ◮ Memristors [SSSW08] :

NOISY IN-MEMORY RECURSIVE COMPUTATION WITH MEMRISTOR CROSSBARS Elsa Dupraz, Lav Varshney

slide-3
SLIDE 3

Section 1: Introduction 3

Dot-product computation from memristor crossbars [LWFV18]

Introduction Dot-product computation Iterative dot-product computation Simulation results Conclusion

Memristor crossbar :

... ... ... ... ... ...

Notation : ◮ ui : Input voltages ◮ xj : Output voltages ◮ gij : Conductance values xj =

N

  • i=1

gij N

k=0 gkj

ui Issue : ◮ Uncertainty on conductance values In this work : noisy computation from memristor crossbars

NOISY IN-MEMORY RECURSIVE COMPUTATION WITH MEMRISTOR CROSSBARS Elsa Dupraz, Lav Varshney

slide-4
SLIDE 4

Section 1: Introduction 4

Existing works

Introduction Dot-product computation Iterative dot-product computation Simulation results Conclusion

Existing works ◮ Logic-in-Memory [GTDS17, AHC+19] ◮ Dot-product computation from memristor crossbars [NKSB14] ◮ Memristor crossbars for Machine Learning [LWFV18, JAC+19] ◮ Hamming distance computation [CC15] ◮ Noisy Hamming distance computation [CSD18] In this work ◮ Noisy dot-product computation in memory : Probability distribution of final computation error ◮ Noisy iterative dot-product computation in memory : Recursive expressions of means and variances of successive outputs

NOISY IN-MEMORY RECURSIVE COMPUTATION WITH MEMRISTOR CROSSBARS Elsa Dupraz, Lav Varshney

slide-5
SLIDE 5

Section 1: Introduction 5

Table of contents

Introduction Dot-product computation Iterative dot-product computation Simulation results Conclusion

  • 1. Introduction
  • 2. Dot-product computation
  • 3. Iterative dot-product computation
  • 4. Simulation results
  • 5. Conclusion

NOISY IN-MEMORY RECURSIVE COMPUTATION WITH MEMRISTOR CROSSBARS Elsa Dupraz, Lav Varshney

slide-6
SLIDE 6

Section 2: Dot-product computation 6

Table of contents

Introduction Dot-product computation Iterative dot-product computation Simulation results Conclusion

  • 1. Introduction
  • 2. Dot-product computation
  • 3. Iterative dot-product computation
  • 4. Simulation results
  • 5. Conclusion

NOISY IN-MEMORY RECURSIVE COMPUTATION WITH MEMRISTOR CROSSBARS Elsa Dupraz, Lav Varshney

slide-7
SLIDE 7

Section 2: Dot-product computation 7

Computation model

Introduction Dot-product computation Iterative dot-product computation Simulation results Conclusion

... ... ... ... ... ...

Dot-product computation : Xj =

N

  • i=1

Gij N

k=0 Gkj

Ui Noisy computation : ◮ Gij, Ui are independent random variables ◮ 1st and 2nd order moments ◮ E[Gij] = gij (target value) ◮ E[Ui] = ui (target value)

Objective : determine the probability distributions of the Xj

NOISY IN-MEMORY RECURSIVE COMPUTATION WITH MEMRISTOR CROSSBARS Elsa Dupraz, Lav Varshney

slide-8
SLIDE 8

Section 2: Dot-product computation 8

Main result

Introduction Dot-product computation Iterative dot-product computation Simulation results Conclusion

Objective : determine the probability distribution of Xj = N

i=1 Gij N

k=0 Gkj Ui

Theorem If αj = lim

N→∞

N

i=0 gij

2 N2 = 0, then N2 √vj

  • Xj − xj

d ⇒ N

  • 0, 1

α2

j

  • ◮ xj = N

i=1 gij N

k=0 gkj ui is the true output value

◮ d is the convergence in distribution

NOISY IN-MEMORY RECURSIVE COMPUTATION WITH MEMRISTOR CROSSBARS Elsa Dupraz, Lav Varshney

slide-9
SLIDE 9

Section 2: Dot-product computation 9

Conclusions of the Theorem

Introduction Dot-product computation Iterative dot-product computation Simulation results Conclusion

◮ The Theorem is valid for a large range of distributions ◮ The distribution of Xj can be approximated by a Gaussian : Xj ∼ AN

  • xj,

vj α2

j N4

  • ◮ The mean-squared error can be approximated as :

E[(Xj − xj)2] ≈ vj α2

j N4 .

◮ Conclusion : if vj and αj tend to constants, E[(Xj − xj)2] → 0

NOISY IN-MEMORY RECURSIVE COMPUTATION WITH MEMRISTOR CROSSBARS Elsa Dupraz, Lav Varshney

slide-10
SLIDE 10

Section 3: Iterative dot-product computation 10

Table of contents

Introduction Dot-product computation Iterative dot-product computation Simulation results Conclusion

  • 1. Introduction
  • 2. Dot-product computation
  • 3. Iterative dot-product computation
  • 4. Simulation results
  • 5. Conclusion

NOISY IN-MEMORY RECURSIVE COMPUTATION WITH MEMRISTOR CROSSBARS Elsa Dupraz, Lav Varshney

slide-11
SLIDE 11

Section 3: Iterative dot-product computation 11

Computation model

Introduction Dot-product computation Iterative dot-product computation Simulation results Conclusion

... ... ... ... ... ...

Iterative dot-product computation : X(T) = G(T)G(T−1) · · · G(1)X(0) Recursion : ◮ X(t) = G(t)X(t−1).

Objective : Recursive expressions of 1st and 2nd order statistics of the X(t)

NOISY IN-MEMORY RECURSIVE COMPUTATION WITH MEMRISTOR CROSSBARS Elsa Dupraz, Lav Varshney

slide-12
SLIDE 12

Section 3: Iterative dot-product computation 12

Main results

Introduction Dot-product computation Iterative dot-product computation Simulation results Conclusion

Objective : First-order moments of X (t)

j

= N

i=1 Gij N

k=0 Gkj X (t−1)

i

Proposition 1 : Mean The second-order Taylor expansion of the mean µ(t)

j

  • f X (t)

j

is given by µ(t)

j

=

N

  • i=1

g(t)

ij

δ(t)

j

µ(t−1)

i

− Θj (δ(t)

j )2 + ΓjΛj

(δ(t)

j )3 + O

  • 1

(δ(t)

j )3

  • where δ(t)

j

= N

k=0 gkj.

Remark : we have that lim

N→∞ µ(t) j

= x(t)

j NOISY IN-MEMORY RECURSIVE COMPUTATION WITH MEMRISTOR CROSSBARS Elsa Dupraz, Lav Varshney

slide-13
SLIDE 13

Section 3: Iterative dot-product computation 13

Main results

Introduction Dot-product computation Iterative dot-product computation Simulation results Conclusion

Objective : Second-order moments of X (t)

j

= N

i=1 Gij N

k=0 Gkj X (t−1)

i

Proposition 2 : Variance The second-order Taylor expansion of the variance γ(t)

j

  • f X (t)

j

is given by γ(t)

j

=

  • Θj

δ(t)

j

2 + Ψj (δ(t)

j )2 − 2ΛjΘj

(δ(t)

j )3 + 3Θ2 j Γj

(δ(t)

j )4 − (µ(t) j )2 + O

  • 1

(δ(t)

j )3

  • Proposition 3 : Covariance

The second-order Taylor expansion of the covariance γ(t)

jj′ of X (t) j

, X (t)

j′ , with j = j′, is

given by γ(t)

jj′ = N

  • i=1

N

  • i′=1

λijλi′j′γ(t−1)

i,i′

+ O

  • 1

(δ(t)

j )3

  • NOISY IN-MEMORY RECURSIVE COMPUTATION WITH MEMRISTOR CROSSBARS

Elsa Dupraz, Lav Varshney

slide-14
SLIDE 14

Section 4: Simulation results 14

Table of contents

Introduction Dot-product computation Iterative dot-product computation Simulation results Conclusion

  • 1. Introduction
  • 2. Dot-product computation
  • 3. Iterative dot-product computation
  • 4. Simulation results
  • 5. Conclusion

NOISY IN-MEMORY RECURSIVE COMPUTATION WITH MEMRISTOR CROSSBARS Elsa Dupraz, Lav Varshney

slide-15
SLIDE 15

Section 4: Simulation results 15

Synthetic data

Introduction Dot-product computation Iterative dot-product computation Simulation results Conclusion

Parameters : ◮ Uniform random variables Gij, Ui, ◮ Dot-product computation : N = 1000, K = 10000 samples Xj ◮ Iterative computation : T = 8, N ∈ {256, 512, 1024}

Histogram for dot-product computation :

4.8 4.9 5 5.1 5.2 2 4 6 8 10 12 14 xj f

Histogram of Xj Gaussian Approximation Density Approximation of [18]

Variance approx for iterative computation :

2 4 6 8 10- 25 10- 20 10- 15 10- 10 10- 5

Iteration Number Variance

Empirical, N = 256 Gaussian approximation, N = 256 Taylor expansion, N = 256 Empirical, N = 512 Gaussian approximation, N = 512 Taylor expansion, N = 512 Empirical, N = 1024 Gaussian approximation, N = 1024 Taylor expansion, N = 1024

NOISY IN-MEMORY RECURSIVE COMPUTATION WITH MEMRISTOR CROSSBARS Elsa Dupraz, Lav Varshney

slide-16
SLIDE 16

Section 4: Simulation results 16

PCA

Introduction Dot-product computation Iterative dot-product computation Simulation results Conclusion

Parameters : ◮ Memristor-based PCA [LWFV18] ◮ 10 images of size 16 × 16 ◮ Variance σ2 ∈ {0.01, 1, 4} Results :

3 6 9 12 15 3 6 9 12 15

Original image

3 6 9 12 15 3 6 9 12 15

Noisy image

3 6 9 12 15 3 6 9 12 15

Standard PCA

3 6 9 12 15 3 6 9 12 15

  • Mem. PCA (σ=0.1)

3 6 9 12 15 3 6 9 12 15

  • Mem. PCA (σ=1)

3 6 9 12 15 3 6 9 12 15

  • Mem. PCA (σ=2)

t

NOISY IN-MEMORY RECURSIVE COMPUTATION WITH MEMRISTOR CROSSBARS Elsa Dupraz, Lav Varshney

slide-17
SLIDE 17

Section 5: Conclusion 17

Conclusion

Introduction Dot-product computation Iterative dot-product computation Simulation results Conclusion

Main results ◮ Error characterization of (iterative) dot-product computation ◮ The results show some intrinsic robustness of noisy dot-product computation Perspectives ◮ Study other computational problems such as shortest path computation

NOISY IN-MEMORY RECURSIVE COMPUTATION WITH MEMRISTOR CROSSBARS Elsa Dupraz, Lav Varshney

slide-18
SLIDE 18

ITW 2021 Kanazawa ITW 2021 Kanazawa ITW 2021 Kanazawa ITW 2021 Kanazawa

  • Oct. 17-21, 2021
  • Oct. 17-21, 2021

Website : www.itw2021.org

NOISY IN-MEMORY RECURSIVE COMPUTATION WITH MEMRISTOR CROSSBARS Elsa Dupraz, Lav Varshney

slide-19
SLIDE 19

Aayush Ankit, Izzat El Hajj, Sai Rahul Chalamalasetti, Geoffrey Ndu, Martin Foltin, R. Stanley Williams, Paolo Faraboschi, Wenmei Hwu, John Paul Strachan, Kaushik Roy, and Dejan S. Milojicic. PUMA : A programmable ultra-efficient memristor-based accelerator for machine learning inference. In Proc. 24th Int. Conf. Architectural Support for Programming Languages and Operating Systems (ASPLOS ’19), pages 715–731, April 2019. Yuval Cassuto and Koby Crammer. In-memory Hamming similarity computation in resistive arrays. In 2015 IEEE International Symposium on Information Theory (ISIT), pages 819–823, 2015. Zehui Chen, Clayton Schoeny, and Lara Dolecek. Hamming distance computation in unreliable resistive memory. IEEE Transactions on Communications, 66(11) :5013–5027, 2018.

NOISY IN-MEMORY RECURSIVE COMPUTATION WITH MEMRISTOR CROSSBARS Elsa Dupraz, Lav Varshney

slide-20
SLIDE 20

Rahul Gharpinde, Phrangboklang Lynton Thangkhiew, Kamalika Datta, and Indranil Sengupta. A scalable in-memory logic synthesis approach using memristor crossbar. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 26(2) :355–366, 2017.

  • S. Jain, A. Ankit, I. Chakraborty, T. Gokmen, M. Rasch, W. Haensch,
  • K. Roy, and A. Raghunathan.

Neural network accelerator design with resistive crossbars : Opportunities and challenges. IBM J. Res. Dev., 63(6) :10, Nov./Dec. 2019.

NOISY IN-MEMORY RECURSIVE COMPUTATION WITH MEMRISTOR CROSSBARS Elsa Dupraz, Lav Varshney

slide-21
SLIDE 21

Sijia Liu, Yanzhi Wang, Makan Fardad, and Pramod K Varshney. A memristor-based optimization framework for artificial intelligence applications. IEEE Circuits and Systems Magazine, 18(1) :29–44, 2018. Ihab Nahlus, Eric P . Kim, Naresh R. Shanbhag, and David Blaauw. Energy-efficient dot product computation using a switched analog circuit architecture. In Proc. 2014 Int. Symp. Low Power Electronics and Design (ISLPED ’14), pages 315–318, August 2014. Dmitri B Strukov, Gregory S Snider, Duncan R Stewart, and R Stanley Williams. The missing memristor found. Nature, 453(7191) :80, 2008.

NOISY IN-MEMORY RECURSIVE COMPUTATION WITH MEMRISTOR CROSSBARS Elsa Dupraz, Lav Varshney