GPU-Based Simulation of Spiking Neural Networks with Real-Time - - PowerPoint PPT Presentation
GPU-Based Simulation of Spiking Neural Networks with Real-Time Performance & High Accuracy Dmitri Yudanov, Muhammad Shaaban, Roy Melton, Leon Reznik Department of Computer Engineering Rochester Institute of Technology United States WCCI
Department of Computer Engineering Rochester Institute of Technology United States WCCI 2010, IJCNN, July 23
Motivation Neural network models Simulation systems of neural networks Parker-Sochacki numerical integration method CUDA GPU architecture Implementation: software architecture, computation phases Verification Results Conclusion and future work Q&A
J. Nageswaran, N. Dutt, J. Krichmar, A. Nicolau, and A. Veidenbaum, "A configurable simulation environment for the efficient simulation of large-scale spiking neural networks on graphics processors," Neural Networks, Jul. 2009. A. K. Fidjeland, E. B. Roesch, M. P. Shanahan, and W. Luk, "NeMo: A Platform for Neural Modelling of Spiking Neurons Using GPUs," Application-Specific Systems, Architectures and Processors, IEEE International Conference on, vol. 0, pp. 137-144, 2009. J.-P. Tiesel and A. S. Maida, "Using parallel GPU architecture for simulation of planar I/F networks," in , 2009, pp. 754--759. R. Stewart and W. Bair, "Spiking neural network simulation: numerical integration with the Parker-Sochacki method," Journal of Computational Neuroscience, vol. 27, no. 1, pp. 115-33,
IF: simple, but has poor spiking response
HH: has reach response, but complex
IZ: simple, has reach response, but phenomenological
IZ HH IF
N – network size F - average firing rate of a neuron p – average target neurons per spike source R. Brette, et al. Order of computation per second of simulated time
Time quantization error introduced by dt
Smaller dt more precise, but computation hangry
May result in missing events STDP unfriendly
Aligned events good for parallel computing
N – network size F - average firing rate of a neuron p – average target neurons per spike source R. Brette, et al. Order of computation per second of simulated time
Events are processed sequentially
More computation per unit-time
Spike predictor-corrector excessive re-computation
Assumes analytical solution
Small computation order
Events are unique in time no quantization error more accurate, STDP friendly
Events are processed sequentially
Largest possible dt is limited by minimum delay and highest possible transient
Refreshes every dt more structured than event-driven good for parallel computing
Events are unique in time no quantization error more accurate, STDP friendly
Doesn’t require analytical solution
N – network size, F - average firing rate of a neuron, p – average target neurons per spike source
Order of computation per second of simulated time
Euler: compute next y based on tangent to current y
Modified Euler: predict with Euler, correct with average slope
Runge-Kutta 4th Order: evaluate and average
Bulirsch–Stoer: modified midpoint method with evaluation and error tolerance check using extrapolation with rational
functions.
Parker-Sochacki: express IVP as power series. Adaptive order
Motivation: need to solve an IVP
Assume that solution function can be represented with power series. Therefore, its derivative based on Maclaurin series properties is
LLP Parallel reduction
Loop-carried circular dependence on d Only partial parallelism possible
Power series representation adaptive order error
Cauchy product reduces parallelism
Local Lipschitz constant determines the number of
Kernel: code separate, task division Thread Block (1D, 2D, 3D) Grid (1D, 2D) Divide computation based on IDs Granularity: bit level (after warp
Scheduling
Scheduling: parallel and sequential
Scalability requirement for blocks to be independent Warp
Warp = 32 threads
Warp divergence
Warp level synchronization Active blocks and threads:
Active threads / SM: maximum1024
Goal: full occupancy = 1024 threads
Adaptive order p according to required error tolerance
Can be processed in parallel for each neuron
Stewart and Bair
Translate spikes to synaptic events: global communication is required
Encoded spikes are written to the global memory: bit mask + time values
A propagation block reads and filters all spikes, decodes, fetches synaptic data and distributes into time slots
Satish et al.
GPU Device: GTX 260
24 symmetric multiprocessors
Shared memory size, 16 KB / SM
Global memory size, 938 MB
Clock rate, 1.3 GHz
Input Conditions
Random parameter allocation
Random connectivity
Zero PS error tolerance
CPU Device: AMD Opteron 285
Dual core
L2 cache size, 1 KB / core
RAM size, 4 GB
Clock rate, 2.6 GHz
Output
Membrane potential traces
Passed test for equality
Conditions: 80% excitatory / 20% inhibitory synapses, zero tolerance, 10 sec of simulation, initially excited by 0 – 200 pA current.
Results: GPU simulation 8-9 times faster, RT performance for 2-4% - connected networks with size 2048 – 4096 neurons.
Major limiting factors: shared memory, number of SM 50 100 150 200 250 2 3 4 5 6 7 8 9
Simulation Time, sec. Network size, 1000 x neurons
2% 4% 8% 16% 2% 4% 8% 16%
10 60 110 160 210 260 310 360 410 2 4 6 8 10 12
Simulation Time, sec. Mean Event Throughput, 1000 x events/(sec. x neuron)
2% 4% 8% 16% 2% 4% 8% 16%
Conditions: increasing excitatory / inhibitory ratio from 0.8/0.2 to0.98/0.02, network of 4096 neurons, zero tolerance, 10 sec of simulation, initially excited by 0 – 200 pA current.
Results: GPU simulation 6-9 times faster, up to 10,000 events per sec per neuron. RT performance for 0-2% - connected networks with size of 2048 – 4096.
Major limiting factors: shared memory, number of SM
Metric This Work Other works Reason Increase in speed 6 – 9, RT 10 – 35, RT GPU device, complexity of computation, numerical integration methods, simulation type, time scale Network Size 2K - 8K 16K - 200K Connectivity per neuron 100 - 1.3K 100 – 1K Accuracy Full single precision FP Undefined Numerical integration method Verification Direct Indirect
Add accurate STDP implementation Characterize accuracy in relation to signal processing, network
Provide an example of application Port to Open CL Further optimization Implemented high-accurate PS-based hybrid system of spiking
Directly verified implementation
R. Brette, et al., "Simulation of networks of spiking neurons: A review of tools and strategies," Journal of Computational Neurscience, vol. 23, no. 3, pp. 349-398, 2007. R. Stewart and W. Bair, "Spiking neural network simulation: numerical integration with the Parker-Sochacki method," Journal of Computational Neuroscience, vol. 27, no. 1, pp. 115-33, Aug. 2009. G. E. Parker and J. S. Sochacki, "Implementing the Picard iteration," Neural, Parallel Sci. Comput., vol. 4,
E. M. Izhikevich, "Simple model of spiking neurons," Neural Networks, IEEE Transactions on, vol. 14, pp. 1569--1572, 2003. N. Satish, M. Harris, and M. Garland, "Designing efficient sorting algorithms for manycore GPUs," in , 2009,
(2010, Apr.) CUDA Data Parallel Primitives Library. [Accessed online 04/30/2010]. http://code.google.com/p/cudpp/ (2008) NVIDIA CUDA Programming Guide 2.3. [Accessed online 04/30/2010]. http://developer.nvidia.com
J. Nageswaran, N. Dutt, J. Krichmar, A. Nicolau, and A. Veidenbaum, "A configurable simulation environment for the efficient simulation of large-scale spiking neural networks on graphics processors," Neural Networks, Jul. 2009. A. K. Fidjeland, E. B. Roesch, M. P. Shanahan, and W. Luk, "NeMo: A Platform for Neural Modelling of Spiking Neurons Using GPUs," Application-Specific Systems, Architectures and Processors, IEEE International Conference on, vol. 0, pp. 137-144, 2009. J.-P. Tiesel and A. S. Maida, "Using parallel GPU architecture for simulation of planar I/F networks," in , 2009, pp. 754--759.