universality and individuality in recurrent neural
play

Universality and Individuality in Recurrent Neural Networks Niru - PowerPoint PPT Presentation

Apr 21, 2023 •43 likes •183 views

Universality and Individuality in Recurrent Neural Networks Niru Maheswaranathan, Alex Williams, Matthew D Golub, Surya Ganguli, David Sussillo Google Brain & Stanford University NeurIPS 2019 Poster #179 arxiv:1907.08549 Artificial neural

  1. Universality and Individuality in Recurrent Neural Networks Niru Maheswaranathan, Alex Williams, Matthew D Golub, Surya Ganguli, David Sussillo Google Brain & Stanford University NeurIPS 2019 Poster #179 arxiv:1907.08549

  2. Artificial neural networks in neuroscience Advantages: Can train ANNs to accomplish tasks analogous to those studied in animals. • Can inspect/probe/dissect artificial networks very easily. • Can easily initiate a huge number of in silico studies • ... Poster #179 arxiv:1907.08549

  3. Artificial & biological neural networks Motion context a b c Dots off Line Rotate Sort by attractor Attractor irrelevant Dots colour Dots on off Motion Colour Colour 1 Choice 1 Choice 2 Choice 1 Choice 2 Choice 1 Choice 2 6 Choice Choice Choice Motion Colour Strong Weak Strong Strong Weak Strong To choice 1 To choice 2 To choice 1 To choice 2 Colour context Motion context a b c Maheswaranathan et al, 2018 Sort by irrelevant colour Choice Rotate axis Dots Dots on off Dots a c Units off Motion Colour Colour 1 Go Unit 145 Go 1 Yamins & DiCarlo, 2014 RT Go Activitity (a.u.) 1.0 1.5 Dly Go Choice 1 Choice 2 Choice 1 Choice 2 Choice 1 Choice 2 Anti Choice Choice Choice 0.5 RT Anti Dly Anti Normalized task variance Motion Colour 0.0 DM 1 Strong Weak Strong Strong Weak Strong DM 2 0.0 Time (s) 1.5 Ctx DM 1 To choice 1 To choice 2 To choice 1 To choice 2 Ctx DM 2 MultSen DM b Dly DM 1 a Unit 145 Dly DM 2 grid-like Mante & Sussillo et al. Nature 2013 Ctx Dly DM 1 Task variance 0.05 Ctx Dly DM 2 MultSen Dly DM DMS DNMS 0.00 DMC Go DNMC DNMC 0 Task 1 2 3 4 5 6 7 8 9 10 11 12 Clusters b band-like d e 0.5 Go tSNE RT Go Dly Go Anti RT Anti Performance change after lesioning � Dly Anti DM 1 DM 2 Ctx DM 1 Ctx DM 2 c border MultSen DM Cueva & Wei, ICLR , 2018 Dly DM 1 Dly DM 2 Ctx Dly DM 1 Ctx Dly DM 2 Carnevale et al, NEURON , 2015 MultSen Dly DM DMS DNMS DMC DNMC –0.5 1 2 3 4 5 6 7 8 9 10 11 12 Clusters Yang, et al., Nature Neuroscience 2019 d e Rajan, Harvey, Tank, Neuron , 2016 Normalized activity target move onset Sussillo et al., Nature Neuroscience , 2015 Zipser & Andersen, Science , 1988 Orhan & Ma, Nature Neuroscience , 2019

  4. Artificial & biological neural networks Networks have surprisingly similar representations… Motion context a b c Dots off Line Rotate Sort by attractor Attractor irrelevant Dots colour Dots on off Motion Colour Colour 1 Choice 1 Choice 2 Choice 1 Choice 2 Choice 1 Choice 2 6 Choice Choice Choice Motion Colour Strong Weak Strong Strong Weak Strong To choice 1 To choice 2 To choice 1 To choice 2 Colour context Motion context a b c Maheswaranathan et al, 2018 Sort by irrelevant colour Choice Rotate axis Dots Dots on off Dots a c Units off Motion Colour Colour 1 Go Unit 145 Go 1 Yamins & DiCarlo, 2014 RT Go Activitity (a.u.) 1.0 1.5 Dly Go Choice 1 Choice 2 Choice 1 Choice 2 Choice 1 Choice 2 Anti Choice Choice Choice 0.5 RT Anti Dly Anti Normalized task variance Motion Colour 0.0 DM 1 Strong Weak Strong Strong Weak Strong DM 2 0.0 Time (s) 1.5 Ctx DM 1 To choice 1 To choice 2 To choice 1 To choice 2 Ctx DM 2 MultSen DM b Dly DM 1 a Unit 145 Dly DM 2 grid-like Mante & Sussillo et al. Nature 2013 Ctx Dly DM 1 Task variance 0.05 Ctx Dly DM 2 MultSen Dly DM DMS DNMS 0.00 DMC Go DNMC DNMC 0 Task 1 2 3 4 5 6 7 8 9 10 11 12 Clusters b band-like d e 0.5 Go tSNE RT Go Dly Go Anti RT Anti Performance change after lesioning � Dly Anti DM 1 DM 2 Ctx DM 1 Ctx DM 2 c border MultSen DM Cueva & Wei, ICLR , 2018 Dly DM 1 Dly DM 2 Ctx Dly DM 1 Ctx Dly DM 2 Carnevale et al, NEURON , 2015 MultSen Dly DM DMS DNMS DMC DNMC –0.5 1 2 3 4 5 6 7 8 9 10 11 12 Clusters Yang, et al., Nature Neuroscience 2019 d e Rajan, Harvey, Tank, Neuron , 2016 Normalized activity target move onset Sussillo et al., Nature Neuroscience , 2015 Zipser & Andersen, Science , 1988 Orhan & Ma, Nature Neuroscience , 2019

  5. Artificial & biological neural networks …but are composed of drastically different elements! Artificial neuron Biological neuron Poster #179 arxiv:1907.08549

  6. Central question When trained to perform the same task , why should we expect artificial and biological networks to be similar , given the drastic di ff erences in underlying mechanism ? Poster #179 arxiv:1907.08549

  7. This work: an empirical approach Network mechanisms RNN architectures (e.g. LSTMs, GRUs, …) Nonlinearities (e.g. ReLU, tanh) … Similarity measures Canonical correlation analysis (CCA) Centered kernel alignment (CKA) Tasks Decision making Pattern generation Working memory Poster #179 arxiv:1907.08549

  8. Evidence of both universality and individuality Poster #179 arxiv:1907.08549

  9. Evidence of both universality and individuality Pattern generation task Poster #179 arxiv:1907.08549

  10. Evidence of both universality and individuality Analyzing trained networks Pattern generation task PC #3 PC #1 PC #2 Poster #179 arxiv:1907.08549

  11. Evidence of both universality and individuality Analyzing trained networks Pattern generation task PC #3 PC #1 PC #2 Poster #179 arxiv:1907.08549

  12. Evidence of both universality and individuality Analyzing trained networks Pattern generation task PC #3 PC #1 PC #2 Network representations show individuality Poster #179 arxiv:1907.08549

  13. Evidence of both universality and individuality Analyzing trained networks Pattern generation task PC #3 PC #1 PC #2 Network representations show individuality but aspects of the computation are universal prediction Linear Target Poster #179 arxiv:1907.08549

  14. Learn more at Poster #179 Poster #179 arxiv:1907.08549

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

CHAPTER II I CHAPTER I Recurrent Neural Networks Recurrent Neural Networks CHAPTER II : I :

CHAPTER II I CHAPTER I Recurrent Neural Networks Recurrent Neural Networks CHAPTER II : I :

Ugur HALICI - METU EEE - ANKARA 11/18/2004 CHAPTER II I CHAPTER I Recurrent Neural Networks Recurrent Neural Networks CHAPTER II : I : Recurrent Neural Networks CHAPTER I Recurrent Neural Networks Introduction In this chapter first the

404 views • 27 slides
Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural circuits David Sussillo with Valerio Mante and Bill Newsome Table of contents Introduc(on Training recurrent neural networks(RNNs) Understanding how

1.57k views • 105 slides
Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020

Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020

Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020 Introduction Recurrent Neural Network LSTM Agenda Get introduced to different recurrent neural architecture e.g. , RNNs, LSTMs, GRUs etc. Get

743 views • 48 slides
Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The

Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The

Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The Problem of Long-Term Dependencies RNN short-term dependencies Language model trying to predict the next word based on the previous ones the clouds

729 views • 27 slides
IN5550 Neural Methods in Natural Language Processing Applications of Recurrent Neural Networks

IN5550 Neural Methods in Natural Language Processing Applications of Recurrent Neural Networks

IN5550 Neural Methods in Natural Language Processing Applications of Recurrent Neural Networks Stephan Oepen University of Oslo March 24, 2019 Our Roadmap Last Week Language structure: sequences, trees, graphs Recurrent Neural

217 views • 19 slides
CSEP 517: Natural Language Processing Recurrent Neural Networks Autumn 2018 Luke Zettlemoyer

CSEP 517: Natural Language Processing Recurrent Neural Networks Autumn 2018 Luke Zettlemoyer

CSEP 517: Natural Language Processing Recurrent Neural Networks Autumn 2018 Luke Zettlemoyer University of Washington [most slides from Yejin Choi] RECURRENT NEURAL L NE NETWOR WORKS Recurrent Neural Networks (RNNs) Each input

787 views • 29 slides
CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class Recurrent Neural Network Cell Recurrent Neural Networks (RNNs) Bi-Directional Recurrent Neural Networks (Bi-RNNs) Multiple-layer /

583 views • 47 slides
CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class Recurrent Neural Network Cell Recurrent Neural Networks (RNNs) Bi-Directional Recurrent Neural Networks (Bi-RNNs) Multiple-layer /

793 views • 21 slides
IN5550 Neural Methods in Natural Language Processing Recurrent Neural Networks Stephan Oepen

IN5550 Neural Methods in Natural Language Processing Recurrent Neural Networks Stephan Oepen

IN5550 Neural Methods in Natural Language Processing Recurrent Neural Networks Stephan Oepen University of Oslo March 10, 2019 Our Roadmap Today Language structure: sequences, trees, graphs Recurrent Neural Networks Different

780 views • 76 slides
Recurrent Neural Networks Greg Mori - CMPT 419/726 Goodfellow, Bengio, and Courville: Deep

Recurrent Neural Networks Greg Mori - CMPT 419/726 Goodfellow, Bengio, and Courville: Deep

Recurrent Neural Networks Long Short-Term Memory Temporal Convolutional Networks Examples Recurrent Neural Networks Greg Mori - CMPT 419/726 Goodfellow, Bengio, and Courville: Deep Learning textbook Ch. 10 Recurrent Neural Networks Long

667 views • 18 slides
Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang

Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang

Deep Learning Basics Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang Introduction Recurrent neural networks Dates back to (Rumelhart et al. , 1986) A family of neural networks for handling

348 views • 34 slides
Introduction to Recurrent Neural Networks Jakob Verbeek Modeling sequential data with Recurrent

Introduction to Recurrent Neural Networks Jakob Verbeek Modeling sequential data with Recurrent

Introduction to Recurrent Neural Networks Jakob Verbeek Modeling sequential data with Recurrent Neural Networks Compact schematic drawing of standard multi-layer perceptron (MLP) output hidden input Modeling sequential data So far we

796 views • 50 slides
Recurrent Neural Networks (RNN) Artificial Intelligence @ Allegheny College Janyl Jumadinova

Recurrent Neural Networks (RNN) Artificial Intelligence @ Allegheny College Janyl Jumadinova

Recurrent Neural Networks (RNN) Artificial Intelligence @ Allegheny College Janyl Jumadinova March 9, 2020 Alex Graves, Supervised Sequence Labelling with Recurrent Neural Networks http://colah.github.io/posts/2015-08-Understanding-LSTMs/

538 views • 18 slides
Recurrent Neural Network Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline Dependent Random

Recurrent Neural Network Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline Dependent Random

Recurrent Neural Network Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline Dependent Random Variables Text Preprocessing Language Modeling Recurrent Neural Networks (RNN) LSTM Bidirectional RNN Deep RNN d2l.ai

798 views • 55 slides
RNN Input Layer RNN Hidden Layer RNN h t-1 h t x t (Picture adapted from Andrej

RNN Input Layer RNN Hidden Layer RNN h t-1 h t x t (Picture adapted from Andrej

Outline Recurrent Neural Networks (RNNs) Neural Machine Transla/on NMT basics (Sutskever et al., 2014) ABenCon mechanism (Bahdanau et al., 2015) Marcello Federico 2016 Recurrent Neural Networks (RNNs) Based on slides kindly provided

437 views • 24 slides
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung Chung, Caglar Gulcehre, KyungHyun Cho and Yoshua Bengio Presenter: Yu-Wei Lin Background: Recurrent Neural Network Traditional RNNs encounter

229 views • 18 slides
The Potjans-Diesmann local microcircuit model using different neuron classes for excitatory and

The Potjans-Diesmann local microcircuit model using different neuron classes for excitatory and

The Potjans-Diesmann local microcircuit model using different neuron classes for excitatory and inhibitory neurons Nilton L. Kamiji, Renan O. Shimoura, Vinicius L. Cordeiro, Rodrigo F.O. Pena & Antonio C. Roque Department of Physics,

248 views • 11 slides
Neural Networks: Design Shan-Hung Wu shwu@cs.nthu.edu.tw Department of Computer Science,

Neural Networks: Design Shan-Hung Wu shwu@cs.nthu.edu.tw Department of Computer Science,

Neural Networks: Design Shan-Hung Wu shwu@cs.nthu.edu.tw Department of Computer Science, National Tsing Hua University, Taiwan Machine Learning Shan-Hung Wu (CS, NTHU) NN Design Machine Learning 1 / 49 Outline The Basics 1 Example:

1.91k views • 147 slides
Lecture 1: Neurons Lecture 2: Coding with spikes Lecture 3: Tuning curves and receptive fields

Lecture 1: Neurons Lecture 2: Coding with spikes Lecture 3: Tuning curves and receptive fields

Lecture 1: Neurons Lecture 2: Coding with spikes Lecture 3: Tuning curves and receptive fields Learning objectives: To gain a basic understanding of how neurons represent the environment 1. Name some of the parameters that one can extract from

692 views • 52 slides
Neural Networks: Introduction Machine Learning Based on slides and material from Geoffrey Hinton,

Neural Networks: Introduction Machine Learning Based on slides and material from Geoffrey Hinton,

Neural Networks: Introduction Machine Learning Based on slides and material from Geoffrey Hinton, Richard Socher, Dan Roth, 1 Yoav Goldberg, Shai Shalev-Shwartz and Shai Ben-David, and others Where are we? Learning algorithms General learning

412 views • 30 slides
a i a j Input Input Activation Output Output Links Function Function Links Input

a i a j Input Input Activation Output Output Links Function Function Links Input

Neural Learning Methods } An obvious source of biological inspiration for learning research: the brain } The work of McCulloch and Pitts on the perceptron (1943) started as research into how we could precisely model the neuron and the network of

347 views • 6 slides
for NeuronBank Ontology Weiling Li, Rajshekhar Sunderraman, and Paul Katz Georgia State

for NeuronBank Ontology Weiling Li, Rajshekhar Sunderraman, and Paul Katz Georgia State

A Visual Web Query System for NeuronBank Ontology Weiling Li, Rajshekhar Sunderraman, and Paul Katz Georgia State University, Atlanta, GA Outline Introduction Query sub-system overview Comparison with other visual query systems

796 views • 37 slides
Tutorial on Methods for Interpreting and Understanding Deep Neural Networks Wojciech Samek

Tutorial on Methods for Interpreting and Understanding Deep Neural Networks Wojciech Samek

Tutorial on Methods for Interpreting and Understanding Deep Neural Networks Wojciech Samek Grgoire Montavon Klaus-Robert Mller (Fraunhofer HHI) (TU Berlin) (TU Berlin) 1:30 - 2:00 Part 1: Introduction 2:00 - 3:00 Part 2a: Making Deep

431 views • 29 slides
CMP784 DEEP LEARNING Lecture #03 Multi-layer Perceptrons Aykut Erdem // Hacettepe University

CMP784 DEEP LEARNING Lecture #03 Multi-layer Perceptrons Aykut Erdem // Hacettepe University

Image: Jose-Luis Olivares CMP784 DEEP LEARNING Lecture #03 Multi-layer Perceptrons Aykut Erdem // Hacettepe University // Spring 2018 Breaking news! Practical 1 is out! Learning neural word embeddings Due Friday, Mar. 16,

1.08k views • 64 slides

More recommend