On the Algorithmic Power of Spiking Neural Networks Chi-Ning Chou Kai-Min Chung Chi-Jen Lu Harvard University Academia Sinica Academia Sinica ITCS 2019 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 1/10
What is Spiking Neural Networks (SNNs)? Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 2/10
What is Spiking Neural Networks (SNNs)? • Mathematical models for “biological neural networks”. Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 2/10
What is Spiking Neural Networks (SNNs)? • Mathematical models for “biological neural networks”. * By Pennstatenews https://www.flickr.com/photos/pennstatelive/37247502805 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 2/10
What is Spiking Neural Networks (SNNs)? • Mathematical models for “biological neural networks”. Neurons : Nerve cells * By Pennstatenews https://www.flickr.com/photos/pennstatelive/37247502805 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 2/10
What is Spiking Neural Networks (SNNs)? • Mathematical models for “biological neural networks”. Neurons : Nerve cells Synapses : Connections between neurons * By Pennstatenews https://www.flickr.com/photos/pennstatelive/37247502805 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 2/10
What is Spiking Neural Networks (SNNs)? • Mathematical models for “biological neural networks”. Neurons : Nerve cells Synapses : Connections between neurons Spikes : Instantaneous signals * By Pennstatenews https://www.flickr.com/photos/pennstatelive/37247502805 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 2/10
What is Spiking Neural Networks (SNNs)? • Mathematical models for “biological neural networks”. Neurons Synapses Spikes Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 2/10
What is Spiking Neural Networks (SNNs)? • Mathematical models for “biological neural networks”. Neurons Synapses • Various models since the 1900s. Spikes Integrate-and-fire [Lap07] , Hodgkin-Huxley [HH52] , • their variants [Fit61, Ste65, ML81, HR84, Ger95, KGH97, BL03, FTHVVB03, I+03, TMS14] . Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 2/10
What is Spiking Neural Networks (SNNs)? • Mathematical models for “biological neural networks”. Neurons Synapses • Various models since the 1900s. Spikes Integrate-and-fire [Lap07] , Hodgkin-Huxley [HH52] , • their variants [Fit61, Ste65, ML81, HR84, Ger95, KGH97, BL03, FTHVVB03, I+03, TMS14] . • Study the behaviors/statistics of SNNs, e.g., firing rate. Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 2/10
What is Spiking Neural Networks (SNNs)? • Mathematical models for “biological neural networks”. Neurons Synapses • Various models since the 1900s. Spikes Integrate-and-fire [Lap07] , Hodgkin-Huxley [HH52] , • their variants [Fit61, Ste65, ML81, HR84, Ger95, KGH97, BL03, FTHVVB03, I+03, TMS14] . • Study the behaviors/statistics of SNNs, e.g., firing rate. [Barret-Denève-Machens 2013] empirically showed a connection between • the firing rate of integrate-and-fire SNNs and an optimization problem. Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 2/10
What is Spiking Neural Networks (SNNs)? • Mathematical models for “biological neural networks”. Neurons Synapses • Various models since the 1900s. Spikes Integrate-and-fire [Lap07] , Hodgkin-Huxley [HH52] , • their variants [Fit61, Ste65, ML81, HR84, Ger95, KGH97, BL03, FTHVVB03, I+03, TMS14] . • Study the behaviors/statistics of SNNs, e.g., firing rate. [Barret-Denève-Machens 2013] empirically showed a connection between • the firing rate of integrate-and-fire SNNs and an optimization problem. SNNs seem to have non-trivial computational power. Can we understand them better through the lens of algorithms? Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 2/10
Integrate-and-Fire (IAF) Model [Lapicque 1907] Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10
Integrate-and-Fire (IAF) Model [Lapicque 1907] • Neurons: 𝑜 = {1,2, … , 𝑜} 1 3 6 4 5 2 7 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10
Integrate-and-Fire (IAF) Model [Lapicque 1907] • Neurons: 𝑜 = {1,2, … , 𝑜} 𝜃 𝜃 • Potential: 𝒗 𝑢 ∈ ℝ - 1 3 𝜃 0 0 6 𝜃 0 4 𝜃 𝜃 0 𝜃 5 2 7 0 0 0 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10
Integrate-and-Fire (IAF) Model [Lapicque 1907] • Neurons: 𝑜 = {1,2, … , 𝑜} 𝜃 𝜃 • Potential: 𝒗 𝑢 ∈ ℝ - 1 3 • Dynamics: 𝒗 𝑢 + 1 = 𝒗 𝑢 − 𝐷𝒕 𝑢 + 𝑱 𝜃 0 0 6 𝜃 0 4 𝜃 𝜃 0 𝜃 5 2 7 0 0 0 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10
Integrate-and-Fire (IAF) Model [Lapicque 1907] • Neurons: 𝑜 = {1,2, … , 𝑜} 𝜃 𝜃 • Potential: 𝒗 𝑢 ∈ ℝ - 1 3 • Dynamics: 𝒗 𝑢 + 1 = 𝒗 𝑢 − 𝐷𝒕 𝑢 + 𝑱 𝜃 0 0 6 External charging 𝜃 0 4 𝜃 𝜃 0 𝜃 5 2 7 0 0 0 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10
Integrate-and-Fire (IAF) Model [Lapicque 1907] • Neurons: 𝑜 = {1,2, … , 𝑜} 𝜃 𝜃 • Potential: 𝒗 𝑢 ∈ ℝ - 1 3 • Dynamics: 𝒗 𝑢 + 1 = 𝒗 𝑢 − 𝐷𝒕 𝑢 + 𝑱 𝜃 0 0 6 Spiking effects 𝜃 0 4 𝜃 𝜃 0 𝜃 5 2 7 0 0 0 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10
Integrate-and-Fire (IAF) Model [Lapicque 1907] 𝑱 4 𝑱 8 • Neurons: 𝑜 = {1,2, … , 𝑜} 𝜃 𝜃 𝑱 9 • Potential: 𝒗 𝑢 ∈ ℝ - 1 3 • Dynamics: 𝒗 𝑢 + 1 = 𝒗 𝑢 − 𝐷𝒕 𝑢 + 𝑱 𝑱 𝟓 𝜃 0 0 6 • External charging: 𝑱 ∈ ℝ - 𝜃 𝑱 7 𝑱 5 0 4 𝑱 𝟖 𝜃 𝜃 0 𝜃 5 2 7 0 0 0 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10
Integrate-and-Fire (IAF) Model [Lapicque 1907] • Neurons: 𝑜 = {1,2, … , 𝑜} 𝜃 𝜃 • Potential: 𝒗 𝑢 ∈ ℝ - 1 3 • Dynamics: 𝒗 𝑢 + 1 = 𝒗 𝑢 − 𝐷𝒕 𝑢 + 𝑱 𝜃 0 0 6 • External charging: 𝑱 ∈ ℝ - 𝜃 0 4 • Spikes: 𝒕 𝑢 ∈ 0,1 - Firing Rule 𝜃 𝜃 0 𝒕 < 𝑢 = 1 ⇔ 𝒗 < 𝑢 ≥ 𝜃 𝜃 5 2 7 0 0 0 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10
Integrate-and-Fire (IAF) Model [Lapicque 1907] • Neurons: 𝑜 = {1,2, … , 𝑜} 𝜃 𝜃 • Potential: 𝒗 𝑢 ∈ ℝ - 1 3 • Dynamics: 𝒗 𝑢 + 1 = 𝒗 𝑢 − 𝐷𝒕 𝑢 + 𝑱 𝜃 0 0 −𝐷 8@ −𝐷 4@ 6 • External charging: 𝑱 ∈ ℝ - −𝐷 @@ 𝜃 −𝐷 9@ 0 4 • Spikes: 𝒕 𝑢 ∈ 0,1 - Firing Rule 𝜃 𝜃 0 𝒕 < 𝑢 = 1 ⇔ 𝒗 < 𝑢 ≥ 𝜃 • Connectivity: 𝐷 ∈ ℝ -×- −𝐷 A@ −𝐷 5@ 𝜃 −𝐷 7@ 5 2 7 0 0 0 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10
Integrate-and-Fire (IAF) Model [Lapicque 1907] • Neurons: 𝑜 = {1,2, … , 𝑜} 𝜃 𝜃 • Potential: 𝒗 𝑢 ∈ ℝ - 1 3 • Dynamics: 𝒗 𝑢 + 1 = 𝒗 𝑢 − 𝐷𝒕 𝑢 + 𝑱 𝜃 0 0 −𝐷 8@ −𝐷 4@ 6 • External charging: 𝑱 ∈ ℝ - −𝐷 @@ 𝜃 −𝐷 9@ 0 4 • Spikes: 𝒕 𝑢 ∈ 0,1 - Firing Rule 𝜃 𝜃 0 𝒕 < 𝑢 = 1 ⇔ 𝒗 < 𝑢 ≥ 𝜃 • Connectivity: 𝐷 ∈ ℝ -×- −𝐷 A@ −𝐷 5@ 𝜃 −𝐷 7@ 5 2 7 0 0 0 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10
Integrate-and-Fire (IAF) Model [Lapicque 1907] • Neurons: 𝑜 = {1,2, … , 𝑜} 𝜃 𝜃 • Potential: 𝒗 𝑢 ∈ ℝ - 1 3 • Dynamics: 𝒗 𝑢 + 1 = 𝒗 𝑢 − 𝐷𝒕 𝑢 + 𝑱 𝜃 0 0 6 • External charging: 𝑱 ∈ ℝ - 𝜃 0 4 • Spikes: 𝒕 𝑢 ∈ 0,1 - Firing Rule 𝜃 𝜃 0 𝒕 < 𝑢 = 1 ⇔ 𝒗 < 𝑢 ≥ 𝜃 • Connectivity: 𝐷 ∈ ℝ -×- 𝜃 5 2 7 0 0 0 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10
Integrate-and-Fire (IAF) Model [Lapicque 1907] • Neurons: 𝑜 = {1,2, … , 𝑜} 𝜃 𝜃 • Potential: 𝒗 𝑢 ∈ ℝ - 1 3 • Dynamics: 𝒗 𝑢 + 1 = 𝒗 𝑢 − 𝐷𝒕 𝑢 + 𝑱 𝜃 0 0 6 • External charging: 𝑱 ∈ ℝ - 𝜃 0 4 • Spikes: 𝒕 𝑢 ∈ 0,1 - Firing Rule 𝜃 𝜃 0 𝒕 < 𝑢 = 1 ⇔ 𝒗 < 𝑢 ≥ 𝜃 • Connectivity: 𝐷 ∈ ℝ -×- 𝜃 5 2 7 • Firing rate: 𝒚 𝑢 = (#spikes before time 𝑢)/𝑢 0 0 0 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10
Recommend
More recommend
