Limits its of nume meric rical al appr proache oaches FACET - - PowerPoint PPT Presentation

NEUROSCIE

SCIENTIFI IFIC MODELIN ING WITH LARGE-SCA SCALE AND HIGHLY ACCELERATED TED NEUROMOR MORPHI HIC HARDWARE DEVICE CES

Mihai A. Petrovici, University of Heidelberg

The FACETS Project Electronic Vision(s) Group

PART

ART I

AN INTRODUCTIO

TION TO TO THE

FACET CETS S NEUROMOR

ORPHI HIC HARDWARE

Limits its of nume meric rical al appr proache

• aches

FACET CETS S neur urom

• morp
• rphic

ic hard rdware are

Spike ikey y - 2006: 2006: 384 neurons 105 synapses

FACET CETS S neur urom

• morp
• rphic

ic hard rdware are

Spike ikey y - 2006: 2006: 384 neurons 105 synapses HICANN NN - 2010 2010 512 neurons 1.3  105 synapses

FACET CETS S neur urom

• morp
• rphic

ic hard rdware are

Spike ikey y - 2006: 2006: 384 neurons 105 synapses HICANN NN - 2010: 2010: 512 neurons 1.3  105 synapses Waf afer er - 2011: 2011: 16  104 neurons 4  107 synapses

FACET CETS S neur urom

• morp
• rphic

ic hard rdware are

Spike ikey y - 2006: 2006: 384 neurons 105 synapses HICANN NN - 2010: 2010: 512 neurons 1.3  105 synapses Waf afer er - 2011: 2011: 16  104 neurons 4  107 synapses Rac ack – 20??: 16  105 neurons 4  108 synapses

Hardw rdware are vs. biology logy

Biological gical neural ral comp mputat tation 1011 neurons, 1015 synapses 10.000 synapses per neuron vast range of neuron categories and parameters long term, short term local, global various time constants and delays FACET ACETS wafer-scale scale hard rdwar ware 105 Neurons, 107 Synapses arbitrarily configurable multi-compartment Adaptive Exponential Integrate and Fire neurons Short Term Plasticity Spike Timing Dependent Plasticity adjustable time constants, but no on-wafer delays modular, high bandwidth, low power, fault tolerant Connectivity Diversity Plasticity Timing Scalability up to 105 speedup

Neuro uron n model del of choice ice

• R. Naud et al.: Firing patterns in the adaptive-exponential integrate-and fire-model, BiolCybern(2008) 99:335–347

tonic spiking initial burst delayed accelerating transient spiking adaptation regular bursting delayed regular bursting irregular spiking

CMOS OS implem emen entation tation of AdEx Ex neuron uron

Wafe fer-sca scale le integr gration ation

massive configuration space  dedicated mapping tools  versatile control software  distortion analysis and compensation  complex emulation workflow

PART

ART II (A)

WORKFLO

LOW:

BIOLOG

OGY-TO TO-HA HARDWA WARE MAPPING

? ?

Mod

• deling

eling langu guage age

Mod

• deling

eling langu guage age

Softwa ftware re and hard rdware are layers ers

Softwa ftware re and hard rdware are layers ers

Biology logy-to to-hard ardware ware mappin ping Graph aph model el (TUD UD)

Biology logy-to to-hard ardware ware mappin ping Graph aph model el (TUD UD)

Hardw rdware are grap aph

Biology logy-to to-hard ardware ware mappin ping Graph aph model el (TUD UD)

Nforce

• rce cluster

ster algor

• rithm

ithm

Placing cing opti timiz mizatio ation

Mapping pping algor

• rithm

ithm perfor rforman mance ce

PART II (B) WORKFLO

LOW:

DIST

STOR ORTION ION EVALUATION ON AND COMPENSA NSATION ION

? ?

Attra tractor ctor memory

• ry schem

ematic atic

Spiki king g patter terns ns

Trajector ajectories es in voltage age space ce

Networ twork k dynamics amics

Networ twork k dynamics amics Motiv tivatio ation

• hardware imperfections
• nonisomorphic simulation/emulation environments

e.g. neuron model, digitized weights, …

• mapping/routing losses

robustness is an essential characteristic of biological neural networks  hardware independent research Relev evant ant para rame meter ters

• STP
• adaptation
• delays
• synaptic weights
• neuron loss
• synapse loss
• number of MC per HC
• number of HC
• total number of MC

( network size) model- independent model- specific

The e importan

• rtance

ce of STP

without STP (Poisson input: 1 kHz) with STP (Poisson input: 4 kHz)

The e importan

• rtance

ce of adap aptatio tation n and delays ays

+ adaptation + delays mean firing rate in ON state: 30 Hz + adaptation - delays mean firing rate in ON state: 28 Hz

• adaptation - delays

mean firing rate in ON state: 116 Hz

Dwel well l times s and neuro uron n loss

Syna napse pse loss 0% loss 10% % loss 25% % loss 40% % loss

Dwel well l times s and synapse apse loss

5 % 10 % 20 % 0 %

Firin ing g rates es and d synap apse se loss

Networ twork k scaling ing Relev evant ant para rame meter ters

• STP
• adaptation
• delays
• synaptic weights
• neuron loss
• synapse loss

model- independent

• number of MC per HC
• number of HC
• total number of MC

( network size) model- specific scaling may influence behavior !



Networ twork k scaling ing Relev evant ant para rame meter ters

• STP
• adaptation
• delays
• synaptic weights
• neuron loss
• synapse loss

model- independent

• number of MC per HC
• number of HC
• total number of MC

( network size) model- specific

Scaling ling throu rough gh modific ification tion of conn nnectio ection n probabili

• babilitie

ties

1 2 3 1 2 3

scaling may influence behavior !



1 2 3 1 2 1 2

Scaling ling and d rob

• bustne

ustness ss

0% synapse loss

3 HC 3 MC

20% synapse loss

Scaling ling and d rob

• bustne

ustness ss

9 HC 9 MC

0% synapse loss 20% synapse loss

Patter tern compl mpleti etion

• n

stor

• red

ed image mages

Spontan

• ntaneo

eous pattern tern generati neration

• n

Patter tern compl mpleti etion:

• n: small

l distor tortio tion input put image

Patter tern compl mpleti etion:

• n: large

ge distorti tortion

• n

input put image

Patter tern compl mpleti etion:

• n: two
• patter

terns input put image

Patter tern compl mpleti etion:

• n: a more

e biological logical appr proach

• ach

Synfire nfire chain ain schem ematic atic same parameters in our model exc 100 regular spiking neurons inh 25 fast spiking neurons

Synfire nfire chain ain simulations lations

Syna napse pse loss

The e proble

• blem

m of limited ed input ut

• nly 64 external inputs

with max. 100 Hz / channel for 192 neurons 4000 Hz independent Poisson input per neuron

The e proble

• blem

m of limited ed input ut

Problem I how to quantify and predict correlations which arise from shared inputs ? Problem II given a limited set of input channels and a minimum requirement for inputs per neuron, can we find a corresponding mapping ?

Singl gle e neuron uron beha havi vior

• r

The The Load Function the neuron fires if

 

mem syn 

  , max 

   



    

i i i

t w t

spikes i

exp t

• 

L

thresh

L L 

with

Statistic tistical al treatmen atment t of neural ural activity vity

Gaussian distribution: , for example two channels: shared and private two neurons sharing inputs:  multivariate normal distributions numerical integration: conditional probability:

 

 ,

M 

N

 

2 1

,

L N

 

s

L

 

p

L  

2 2 1

, ) ( ) ( ) (

p s

p s p s A

dx x a P x P a P

L L

L L N L L L

         



  



  

        dx x b P x a P x P b a P

p p s B A

) ( ) ( ) ( ) , (

L L L L L

 

) , ( : ,

thresh thresh

L L

  

• b

a P B A P

     

B P B A P B A P , | 

Symme metr tric ic Uncer ertainty tainty

   

) ( ) ( ; 2 2 , Y H X H Y X I R Y X SU   

         

   

 

 

  

1 , 1 ,

log ;

A B

B p A p B A p B A p B A I

features:

• symmetric in X and Y
• pure information theory  highly general
• normalized: SU[0,1]  allows comparison
• ver a wide range of spike train parameters
• no free parameters !
• more than just synchrony

Partial rtial derivati rivative ves

Vthresh = -55 mV simtime = 20 s Vrest = -59 mV wexc= w  0,5 nS mem = 5 ms

The e mapp pping ing pro roblem blem

kmax maximum common inputs per neuron pair n total inputs per neuron

     

N (=64/2) total inputs M (=192) total outputs for given N, n minimize k while keeping M  192

• r

for given N, n (large), kmax (small) can we find enough subsets (M)? k common inputs per neuron pair

A grap aph theore eoreti tical cal approa proach

vertices  subsets edges 

• verlap between subsets

two subsets are connected if they have more than kmax elements in common

        

1 6 , 5 , 1 , 5 , 4 , 2 , 6 , 5 , 4 , 5 , 3 , 1 , 4 , 2 , 1 , 3 , 2 , 1

max 

  k

 

3 , 2 , 1

 

4 , 2 , 1

 

5 , 3 , 1

 

5 , 4 , 2

 

6 , 5 , 1

 

6 , 5 , 4

goal: find maximum number of unconnected vertices a.k.a. MAXIMUM INDEPENDENT VERTEX SET PROBLEM

Results sults

The The hybrid algorit ithm Results idea: 1) use greedy algorithm until 2) use “smart” (vertex-cut) algorithm from that point onward

 

40000 ier smart_barr card   

 n=4, kmax=2  M=1240  n=6, kmax=3  M=1357  n=5, kmax=2  M=348  n=7, kmax=3  M=412

N=32, M192 min(kmax)

Limite ited d output tput

• n-wafer bandwidth:

2 Tbps (Layer 1)

• nly 1% of this data can be read out

voltages: 2 per chip, 384 chips 20 MB/s for one channel front-end data volume @CMS: 2 Tbps

PART

ART III

III THE

HE FACET

ETS S DEMONST

STRATOR OR

The FACETS Demonstrator…

… integrates techniques and tools developed within FACETS … … into a complete workflow … … that allows to use the FACETS wafer-scale hardware system … (currently: a virtual version of it) … for the emulation of benchmark cortical neural network models … … which exhibit functionality that can be demonstrated … … which are written in PyNN … … and therefore can be computed with established software simulators (for verification, performance evaluation etc.)

Simulating ulating the emulator ator

Simulating ulating the emulator ator

Goals als Testing and evaluation of all involved software layers

Virtual hardware allows to – test software before hardware is available – test without possible hardware-specific problems – provide a preliminary PyNN module for off-line testing of experiments

Verification of possible hardware changes

e.g. optionally insert detailed HICANN model

 indispensable framework for preparation and development tasks

The e Demonstra monstrator tor model dels s (so far) r)

 A layer 2/3 attractor memory

(by KTH, Krishnamurty / Lansner)

 A synfire chain model

(by INCM and ALUF, Kremkow / Aertsen / Masson)

 A model of self-sustaining cortical AI states

(by UNIC, Davison / Destexhe)

 Upcoming: Two-layer model by UNIC

All written in PyNN, all scalable, basic versions can be mapped to hardware without synapse loss

L2/3 /3 corti rtical cal attracto ractor memory

• ry (NEST)

ST)

L2/3 /3 corti rtical cal attracto ractor memory

• ry (virtual

rtual HW)

Synfire nfire chain ain with h feedf dforw

• rward

ard inhibition ibition (NEST) ST)

Synfire nfire chain ain with h feedf dforw

• rward

ard inhibition ibition (virtu rtual HW)

Cortica rtical l AI states es (NEUR URON) N)

Cortica rtical l AI states es (virtu rtual HW)

PART

ART IV

IV “SPIKE

PIKEY” - DEMOS

The “Spikey” chip

WTA A ring

WTA A ring

WTA A ring

Synfire nfire chain ain with h feedf dforw

• rward

ard inhibition ibition

Synfire nfire chain ain with h feedf dforw

• rward

ard inhibition ibition

“Hellfire chain”

“Hellfire chain”

L2/3 /3 corti rtical cal attracto ractor memory

• ry

L2/3 /3 corti rtical cal attracto ractor memory

• ry

Talki king ng Spikey ey

Talki king ng Spikey ey

PART

ART V

SUMMARY & & TO

TO-DO DO-LIS LIST

Summar mmary

well-esta tablished d workflow: kflow: 1. 1. write te model el in PyNN 2. 2. run!

Summar mmary

well-esta tablished d workflow: kflow: 1. 1. write te model el in PyNN 2. 2. run! 3.1 mapping tool chooses optimal placing and routing 3.2 graph model used for parameter space configuration 3.3 complex, custom-designed software takes care of communication this is done automatically…

Summar mmary

0.1 evaluate model – check if suitable for HW 0.2 analyze influence of distortions on dynamics 0.3 find (if possible !) suitable compensation mechanisms 0.4 investigate scaling properties, if necessary 0.5 think about input-to-network mapping 0.6 think about readout issues well-esta tablished d workflow: kflow: 1. 1. write te model el in PyNN 2. 2. run! 3.1 mapping tool chooses optimal placing and routing 3.2 graph model used for parameter space configuration 3.3 complex, custom-designed software takes care of communication this is done automatically… however, you still need to use your brain…

To To-do do list

Software and modeling

• Demonstrator benchmark models:

find suitable compensation mechanisms for hardware-specific distortions

• embed input-to-network mapping optimization in mapping algorithm

Hardware and low-level software

• implement multi-Spikey environment
• get a fully functioning wafer-scale system (huge R&D effort for hardware people)

investigate the interplay between software and actual hardware Long-term perspectives

• multi-wafer neuromorphic computation facility

Acknow knowled edge geme ments ts

Electronic Vision(s) Group

Acknow knowled edge geme ments ts

The FACETS Project

Links nks

The FACETS Project www.facets-project.org The Electronic Vision(s) group www.kip.uni-heidelberg.de/cms/groups/vision/home/ PyNN neuralensemble.org/trac/PyNN/