SLIDE 1
Neuromorphic Computing with Reservoir Neural Networks on Memristive Hardware
Aaron Stockdill September 2016
Neuromorphic Computing with Reservoir Neural Networks on Memristive Hardware
Aaron Stockdill September 2016
Neuromorphic Computing with Reservoir Neural Networks on Memristive - - PowerPoint PPT Presentation
Neuromorphic Computing with Reservoir Neural Networks on Memristive - - PowerPoint PPT Presentation
Neuromorphic Computing with Reservoir Neural Networks on Memristive Hardware Aaron Stockdill September 2016 Neuromorphic Computing with Reservoir Neural Networks on Memristive Hardware Aaron Stockdill September 2016 Neuromorphic Computing
SLIDE 2
SLIDE 3
Neuromorphic Computing
Trying to build an artificial brain Software: Artificial Neural Networks Hardware: this project!
Image Source: Middleton Lab, http://middleton-lab.com/
Brain Shaped
SLIDE 4
Reservoir Neural Networks
Recurrent neural network Much easier to train — single readout layer Echo State Network (ESN), Jaeger, 2001 Liquid State Machine (LSM), Maass, 2002
... ...
Inputs Outputs Win Wout W Reservior
SLIDE 5
Memristors & Atomic Switches
Memristors: Leon Chua, 1989 Resistance is based on past voltage/current Atomic switches are very similar, but instead
- f a gradual change, it’s high/low.
V I Φ Q
Resistor Memristor Capacitor Inductor
SLIDE 6
Memristors & Atomic Switches
−1.0 −0.5 0.0 0.5 1.0 Voltage V −0.4 −0.2 0.0 0.2 0.4 Current I −1.0 −0.5 0.0 0.5 1.0 Voltage V −0.10 −0.05 0.00 0.05 0.10 Current I −1.0 −0.5 0.0 0.5 1.0 Voltage V −0.0010 −0.0005 0.0000 0.0005 0.0010 Current I −1.0 −0.5 0.0 0.5 1.0 Voltage V −10 −5 5 10 Current I
Resistor Atomic Switch Memristor Echo Neuron
SLIDE 7
Motivation & Goals
Build a simulation of neuromorphic hardware See how atomic switches compare to memristors Determine what kinds of problems this hardware can solve Build the most amazing machine learning tool ever
SLIDE 8
Simulating Novel Hardware
SLIDE 9
SLIDE 10
Fostner & Brown
Focussed only on atomic switches Matlab Workflow:
- Deposit particles
- Find groups
- Calculate connections
between groups.
Image Source: Fostner & Brown, Neuromorphic behavior in percolating nanoparticle films
SLIDE 11
Statistical Depositions
Depositing individual particles is really, really slow Use distributions around known averages to “guess” groups Same trick for the gaps between them
- 0.1
0.2 0.3 0.4 0.5 0.6 0.7 2000 4000 6000 8000 Coverage Average number of groups
- Size
20 40 60 80 100 120 140 160 180 200
SLIDE 12
Statistical Depositions
Networks can be generated quickly Nodes of the graph are the group centroids Edges are the memristive connections between the groups.
SLIDE 13
Kirchhoff’s Laws
Current Law & Voltage Laws:
- Current In = Current Out
- Use all the voltage
Used for circuit simulation Build a big matrix
SLIDE 14
Constructing a Reservoir
... ...
W W W
SLIDE 15
Framework
Swap out each section as needed. Many variations tested quickly.
... ...
W W W
SLIDE 16
Readout Weights
Ridge regression to penalise high weights Simple, linear optimisation Actually very powerful in its own right
SLIDE 17
Faster in Fortran
Based on: d-length input sequence, n groups on the chip, k loops to solve the DE for memristors, Lower bounded by LU matrix decomposition. Slowest sections are now in Fortran!
SLIDE 18
Comparisons and Results
Image Source: MicroAssist, licensed under CC-BY-SA.
SLIDE 19
Handwriting Recognition
90%, Woohoo! It works! 90%, Oh no! It still works!
Memristors No reservoir at all
SLIDE 20
Mackey-Glass
Echo State Network Memristors
SLIDE 21
Memory
Echo state networks have four distinct sources of memory:
- Leaking
- Cycles
- Loops
- Discrete time steps
Memristive networks have two sources of memory:
- Leaking
- Conductive state
SLIDE 22
Cycles Loops Discrete Time ESN ESN* ESN* Feed-forward ESN+ One-hop ESN * Loops and Cycles can mimic each
- ther
Memristors Broken
+ Čerňansky
and Makula
SLIDE 23
Summary
SLIDE 24
Summary
We can speed up simulations with statistics Homogeneous neuromorphic hardware is missing key features Cycles can mimic loops, and loops can mimic cycles Discrete time is the single most important part of reservoir learners
SLIDE 25
−1.0 −0.5 0.0 0.5 1.0 Voltage V −0.4 −0.2 0.0 0.2 0.4 Current I
Future Work
Other network components: the ESN “IV” curve looks like a capacitor or inductor IV curve Heterogenous networks with “neurons,” e.g. delay mechanisms Alternative information encoding that the network may be able to handle better Specialised hardware layout: Solving mazes with memristors: A massively parallel approach
- Pershin and Di Ventra
SLIDE 26
Neuromorphic Computing with Reservoir Neural Networks on Memristive Hardware
Aaron Stockdill September 2016