Learning Multi-Sensory Integration with Self- Organization and - PowerPoint PPT Presentation

Learning Multi-Sensory Integration with Self- Organization and Statistics Johannes Bauer, Stefan Wermter http://www.informatik.uni-hamburg.de/WTM/

The Superior Colliculus Johannes Bauer Multi-Sensory Integration through Self-Organization and Statistics 2

The Superior Colliculus inverse effectiveness [1] spatial principle [1] sub-additive coincident neural response close additive disparate super-additive uni-sensory baseline optimal integration [2] Johannes Bauer Multi-Sensory Integration through Self-Organization and Statistics 3

What’s Interesting About That? ? Place in SC Meaning of Input Johannes Bauer Multi-Sensory Integration through Self-Organization and Statistics 4

The Algorithm [3] 𝜍: input stimulus 𝑏 𝑙 : activity of 𝑗 𝑙 𝜍 𝑚 : preferred value of 𝑝 𝑚 Johannes Bauer Multi-Sensory Integration through Self-Organization and Statistics 5

The Algorithm [3] 𝜍: input stimulus 𝑏 𝑙 : activity of 𝑗 𝑙 𝜍 𝑚 : preferred value of 𝑝 𝑚 ∼ 𝑄 𝑏 1 , 𝑏 2 , … , 𝑏 𝑛 𝜍 = 𝜍 𝑚 ) 𝑄 𝜍 = 𝜍 𝑚 𝑏 1 , 𝑏 2 , … , 𝑏 𝑛 𝑄(𝜍 = 𝜍 𝑚 ) 𝑄 𝑏 1 , 𝑏 2 , … , 𝑏 𝑛 Johannes Bauer Multi-Sensory Integration through Self-Organization and Statistics 6

The Algorithm [3] 𝜍: input stimulus 𝑏 𝑙 : activity of 𝑗 𝑙 𝜍 𝑚 : preferred value of 𝑝 𝑚 ∼ 𝑄 𝑏 1 , 𝑏 2 , … , 𝑏 𝑛 𝜍 = 𝜍 𝑚 ) 𝑄 𝜍 = 𝜍 𝑚 𝑏 1 , 𝑏 2 , … , 𝑏 𝑛 𝑄(𝜍 = 𝜍 𝑚 ) 𝑄 𝑏 1 , 𝑏 2 , … , 𝑏 𝑛 Johannes Bauer Multi-Sensory Integration through Self-Organization and Statistics 6

The Algorithm [3] 𝜍: input stimulus 𝑏 𝑙 : activity of 𝑗 𝑙 𝜍 𝑚 : preferred value of 𝑝 𝑚 ∼ 𝑄(𝑏 𝑙 ∣ 𝜍 = 𝜍 𝑚 ) 𝑙 𝑄 𝜍 = 𝜍 𝑚 𝑏 1 , 𝑏 2 , … , 𝑏 𝑛 𝑄(𝜍 = 𝜍 𝑚 ) 𝑄(𝑏 𝑙 ) 𝑙 Noise independent. Johannes Bauer Multi-Sensory Integration through Self-Organization and Statistics 6

The Algorithm [3] 𝜍: input stimulus 𝑏 𝑙 : activity of 𝑗 𝑙 𝜍 𝑚 : preferred value of 𝑝 𝑚 ∼ 𝑄(𝑏 𝑙 ∣ 𝜍 = 𝜍 𝑚 ) 𝑙 𝑄 𝜍 = 𝜍 𝑚 𝑏 1 , 𝑏 2 , … , 𝑏 𝑛 𝑄(𝑏 𝑙 ) 𝑙 Noise independent. 𝜍 unif. dist. Johannes Bauer Multi-Sensory Integration through Self-Organization and Statistics 6

The Algorithm [3] 𝜍: input stimulus 𝑏 𝑙 : activity of 𝑗 𝑙 𝜍 𝑚 : preferred value of 𝑝 𝑚 𝑄 𝜍 = 𝜍 𝑚 𝑏 1 , 𝑏 2 , … , 𝑏 𝑛 ∼ 𝑄(𝑏 𝑙 ∣ 𝜍 = 𝜍 𝑚 ) 𝑙 Noise independent. 𝜍 unif. dist. Johannes Bauer Multi-Sensory Integration through Self-Organization and Statistics 6

The Algorithm [3] 𝑄 𝜍 = 𝜍 𝑚 𝑏 1 , 𝑏 2 , … , 𝑏 𝑛 ∼ 𝑄(𝑏 𝑙 ∣ 𝜍 = 𝜍 𝑚 ) 𝑙 Johannes Bauer Multi-Sensory Integration through Self-Organization and Statistics 11

The Algorithm [3] can be adapted SOM-like 𝑝𝑜 𝑘,𝑏 [𝑏 𝑙 ] Assume 𝑄 𝑏 𝑙 ∣ 𝜍 = 𝜍 𝑘 = ∑𝑝𝑜 𝑘,𝑏 , then 𝑄 𝜍 = 𝜍 𝑘 ∣ 𝑏 1 , 𝑏 2 , … , 𝑏 𝑛 ∼ 𝑝𝑜 𝑘,𝑚 𝑏 𝑚 ∑ 𝑝𝑜 𝑘,𝑚 𝑚 Johannes Bauer Multi-Sensory Integration through Self-Organization and Statistics 12

Comparison to regular SOM Johannes Bauer Multi-Sensory Integration through Self-Organization and Statistics 13

Comparison to regular SOM Johannes Bauer Multi-Sensory Integration through Self-Organization and Statistics 13

The Network in Action ‘visual’ input ‘auditory’ input population-coded PDF Johannes Bauer Multi-Sensory Integration through Self-Organization and Statistics 15

The Network in Action ‘visual’ input ‘auditory’ input population-coded PDF Johannes Bauer Multi-Sensory Integration through Self-Organization and Statistics 16

Performance The network integrates multi-sensory information. errors given ‘visual’ input errors given ‘auditory’ input errors given multi-sensory input The network replicates biological phenomena. Spatial Principle Inverse Effectiveness Johannes Bauer Multi-Sensory Integration through Self-Organization and Statistics 17

Conclusion presented a novel self-organizing ANN algorithm which  learns to combine information near-optimally  shows spatial principle and MLE-like behavior  shows benefit of multisensory integration  learns to compute a PDF for latent variables  is unsupervised  has few inbuilt assumptions Johannes Bauer Multi-Sensory Integration through Self-Organization and Statistics 18

The End References: [1]: Stanford, T. R., Quessy, S., Stein, B. E., Jul. 2005. Evaluating the operations underlying multisensory integration in the cat superior colliculus. The Journal of Neuroscience 25 (28), 6499 – 6508. [2]: Alais, D., Burr, D., Feb. 2004. The ventriloquist effect results from Near-Optimal bimodal integration . Current Biology 14 (3), 257 – 262. [3]: Bauer, J. and Wermter, S., Sept. 2013. Self-organized neural learning of statistical inference from high-dimensional data. In: Proceedings of the International Joint Conference on Artificial Intelligence 2013 . Johannes Bauer Multi-Sensory Integration through Self-Organization and Statistics 14

Performance – Behavioral * *simulation parameters differ from rest of talk. auditory 𝜏 𝑏 = 4.594 ∙ 10 −4 𝑞 𝑏 = 1.680 ∙ 10 −1 visual 𝜏 𝑤 = 1.061 ∙ 10 −4 𝑞 𝑤 = 8.272 ∙ 10 −1 optimal 1 ≈ 4.185 ∙ 10 −5 𝜏 𝑛,𝑝𝑞𝑢 = 1 2 + 1 2 𝜏𝑤 𝜏𝑏 multi-sensory 𝑞 𝑏,𝑝𝑞𝑢 ≈ 1.876 ∙ 10 −1 𝜏 𝑛 = 8.153 ∙ 10 −5 𝑞 𝑤,𝑝𝑞𝑢 ≈ 8.124 ∙ 10 −1 Johannes Bauer Multi-Sensory Integration through Self-Organization and Statistics 20