CoreNeuron : Morphologically Detailed Neuron Simulations Building, - - PowerPoint PPT Presentation
CoreNeuron : Morphologically Detailed Neuron Simulations Building, Simulating and Optimizing Large Neuron Networks on GPUs Pramod Kumbhar, Michael Hines 7 th April 2016, GTC & Blue Brain HPC Team Understanding Brain.. Brain/MINDS 2014
Pramod Kumbhar, Michael Hines & Blue Brain HPC Team
7th April 2016, GTC
2013
BRAIN Initiative
2013
Human Brain Project
2014
Brain/MINDS Better understanding of brain
2007
Izhikevic: brain scale simulation on cluster
1 million cells
2009
IBM: Cat’s brain scale simulation on BG-P
1.5 billion cells
See source1
See source3 See source4
Blue Brain Project, EPFL
See source5 See source6
and labeled
neurons
recordings
Markram et al. 2015, Cell
See source8 See source7 HH, 1952
Biologist view: compartment model
Lexical Analyzer Syntax Analyzer Semantic Analyzer
~ Intermediate Representation
tokens parse tree parse tree
OpenACC C Cuda
Cyme: SIMD
DSL
backends
NMODL Portable Performance
wrap OpenACC and vectorisation hints related pragmas
auto-generated kernel OpenACC API’s to copy the complex data structure
User defined DS: Major challenge for many application AoS/SoA, Vectorisation, Memory Coalescing etc..
bksub: for(i = x; i < nodes; i++) { rhs[i] -= b[i] * rhs[ parent[i] ] rhs[i] /= d[i] }
node : 1 2 3 4 parent[i]: 0 1 2 3
step 0 Step 1 Step 2 Step 3
node : 6 7 8 9 parent[i]: 5 6 7 8 node : 156 157 158 159 parent[i]: 155 156 157 158
thread 0 thread 1 thread 31
Memory addresses
warp
1 2 3 4
1 2 3
parent_index node_index
5 6 7 8 9
5 6 7 8
2 4 6 8
1 2 3
parent_index node_index
1 3 5 7 9
5 6 7 8
1 2 3 4 5 6 7 8 9
1 2 3 4 5 6 7
all cells root Cell 0 and Cell 1 nodes interleaved Roots parent Cell 0 parent Cell 1 parent
Nodes Parents
Permutations: ions, synapses, areas, point processes..
Memory addresses
warp
A
bAC dNAC cSTUT bSTUT cIR bIR cAD cAC dSTUT cNAC bNAC m-type
C
cAC cNAC bNAC dNAC
B
bAC cIR
37% 31% 5% 25% 1.5% 1.5%
e-types me-types me-combinations
L1DAC L1NGC-DA L1NGC-SA L1HAC L1LAC L1SAC L23PC L23MC L23BTC L23DBC L23BP L23NGC L23LBC L23NBC L23SBC L23ChC L4PC L4SP L4SS L4MC L4BTC L4DBC L4BP L4NGC L4LBC L4NBC L4SBC L4ChC L5TTPC1 L5TTPC2 L5UTPC L5STPC L5MC L5BTC L5DBC L5BP L5NGC L5LBC L5NBC L5SBC L5ChC L6TPCL1 L6TPCL4 L6UTPC L6IPC L6BPC L6MC L6BTC L6DBC L6BP L6NGC L6LBC
L23 NBC
(burst Accommodating) (continuous Non-accommodating) (burst Non-accommodating) (continuous Accommodating) (delayed Non-accommodating) (continuous Stuttering) (burst Stuttering) (delayed Stuttering) (continuous Irregular) (burst Irregular) (continuous Adapting)
NetCon List
Buffering Mechanism
load
CPU GPU
current solve state
dt
copy initialize
setup
threshold
queue MPI queue mindelay
Soma
Compartment
hh pas pas pas
pas pas
larger cells 65536 3072 1024 Varying # : Rings, Cells, Branches, Compartments
Model A Model B Model C Model D Model E
vectorization
Real World Models
Ion channels are 4-8x faster in all models! Kernels with cell level parallelism, low occupancy!
393216 65536 4096 3072
180-240 threads
vectorization (XLC issue)
Varying # : Rings, Cells, Branches, Compartments
Homogenous
How much parallelism? How much imbalance?
Resource
Reconstruction and Simulation
Henry Markram,1,2,19,* Eilif Muller,1,19 Srikanth Ramaswamy,1,19 Michael W. Reimann,1,19 Marwan Abdellah,1 Carlos Aguado Sanchez,1 Anastasia Ailamaki,16 Lidia Alonso-Nanclares,6,7 Nicolas Antille,1 Selim Arsever,1 Guy Antoine Atenekeng Kahou,1 Thomas K. Berger,2 Ahmet Bilgili,1 Nenad Buncic,1 Athanassia Chalimourda,1 Giuseppe Chindemi,1 Jean-Denis Courcol,1 Fabien Delalondre,1 Vincent Delattre,2 Shaul Druckmann,4,5 Raphael Dumusc,1 James Dynes,1 Stefan Eilemann,1 Eyal Gal,4 Michael Emiel Gevaert,1 Jean-Pierre Ghobril,2 Albert Gidon,3 Joe W. Graham,1 Anirudh Gupta,2 Valentin Haenel,1 Etay Hay,3,4 Thomas Heinis,1,16,17 Juan B. Hernando,8 Michael Hines,12 Lida Kanari,1 Daniel Keller,1 John Kenyon,1 Georges Khazen,1 Yihwa Kim,1 James G. King,1 Zoltan Kisvarday,13 Pramod Kumbhar,1 Se ´ bastien Lasserre,1,15 Jean-Vincent Le Be ´ ,2 Bruno R.C. Magalha ˜ es,1 Angel Mercha ´ n-Pe ´ rez,6,7 Julie Meystre,2 Benjamin Roy Morrice,1 Jeffrey Muller,1 Alberto Mun ˜ oz-Ce ´ spedes,6,7 Shruti Muralidhar,2 Keerthan Muthurasa,1 Daniel Nachbaur,1 Taylor H. Newton,1 Max Nolte,1 Aleksandr Ovcharenko,1 Juan Palacios,1 Luis Pastor,9 Rodrigo Perin,2 Rajnish Ranjan,1,2 Imad Riachi,1 Jose ´ -Rodrigo Rodrı ´guez,6,7 Juan Luis Riquelme,1 Christian Ro ¨ ssert,1 Konstantinos Sfyrakis,1 Ying Shi,1,2 Julian C. Shillcock,1 Gilad Silberberg,18 Ricardo Silva,1 Farhan Tauheed,1,16 Martin Telefont,1 Maria Toledo-Rodriguez,14 Thomas Tra ¨ nkler,1 Werner Van Geit,1 Jafet Villafranca Dı ´az,1 Richard Walker,1 Yun Wang,10,11 Stefano M. Zaninetta,1 Javier DeFelipe,6,7,20 Sean L. Hill,1,20 Idan Segev,3,4,20 and Felix Schu ¨ rmann1,20
1Blue Brain Project, E´ cole polytechnique fe ´ de ´ rale de Lausanne (EPFL) Biotech Campus, 1202 Geneva, Switzerland
2Laboratory of Neural Microcircuitry, Brain Mind Institute, EPFL, 1015 Lausanne, Switzerland 3Department of Neurobiology, Alexander Silberman Institute of Life Sciences, The Hebrew University of Jerusalem, Jerusalem 91904, Israel 4The Edmond and Lily Safra Center for Brain Sciences, The Hebrew University of Jerusalem, Jerusalem 91904, Israel 5Janelia Farm Research Campus, Howard Hughes Medical Institute, Ashburn, VA 20147, USA 6Laboratorio Cajal de Circuitos Corticales, Centro de Tecnologı´a Biome ´ dica, Universidad Polite ´ cnica de Madrid, 28223 Madrid, Spain
7Instituto Cajal (CSIC) and CIBERNED, 28002 Madrid, Spain 8CeSViMa, Centro de Supercomputacio´ n y Visualizacio ´ n de Madrid, Universidad Polite ´ cnica de Madrid, 28223 Madrid, Spain
9Modeling and Virtual Reality Group, Universidad Rey Juan Carlos, 28933 Mo´ stoles, Madrid, Spain
10Key Laboratory of Visual Science and National Ministry of Health, School of Optometry and Opthalmology, Wenzhou Medical College,Wenzhou 325003, China
11Caritas St. Elizabeth’s Medical Center, Genesys Research Institute, Tufts University, Boston, MA 02111, USA 12Department of Neurobiology, Yale University, New Haven, CT 06510 USA 13MTA-Debreceni Egyetem, Neuroscience Research Group, 4032 Debrecen, Hungary 14School of Life Sciences, University of Nottingham, Nottingham NG7 2UH, United Kingdom 15Laboratoire d’informatique et de visualisation, EPFL, 1015 Lausanne, Switzerland 16Data-Intensive Applications and Systems Lab, EPFL, 1015 Lausanne, Switzerland 17Imperial College London, London SW7 2AZ, UK 18Department of Neuroscience, Karolinska Institutet, Stockholm 17177, Sweden 19Co-first author 20Co-senior author*Correspondence: henry.markram@epfl.ch http://dx.doi.org/10.1016/j.cell.2015.09.029
October 8, 2015 ª2015 Elsevier Inc. http://dx.doi.org/10.1016/j.cell.2015.09.029
low occupancy!
Model A Model B Model C Model D Model E larger cells 65536 2048 1024