modular neural networks
play

Modular Neural Networks CPSC 533 Franco Lee Ian Ko Modular Neural - PowerPoint PPT Presentation

Mar 30, 2023 •39 likes •289 views

Modular Neural Networks CPSC 533 Franco Lee Ian Ko Modular Neural Networks What is it ? Dif f erent models of neural net w orks combined int o a single syst em. Each single net w ork is made int o a module t hat can be f reely int

  1. Modular Neural Networks CPSC 533 Franco Lee Ian Ko

  2. Modular Neural Networks What is it ? Dif f erent models of neural net w orks combined int o a single syst em. Each single net w ork is made int o a module t hat can be f reely int ermixed w it h modules of ot her t ypes in t hat syst em.

  3. Agenda 1) Issues leading to the development of modular neural networks 2) Problems in Neural Network Modeling 3) Cascade Correlation - characteristics - algorithm - mathematical background - examples

  4. Issues Leading to Modular Networks -> Reducing Model Complexity -> Incorporating Knowledge -> Data Fusion and Prediction Averaging -> Combination of Techniques -> Learning Different Tasks Simultaneously -> Robustness and Incrementality

  5. Problems in Neural Network Modeling: -> The selection of the appropriate number of hidden units -> Inefficiencies of Back-Propagation: 1) Slow Learning 2) Moving Target problem

  6. Problems in Neural Network Modeling: Se le ct ion of hidde n unit s: Courtesy of Neural Nets Using Back-propagation presentation - CPSC 533

  7. Problems in Neural Network Modeling: Slow Le a r ning: When training a network with back- propagation, all input weights into the hidden units must be re-adjusted to minimize the residual error.

  8. Problems in Neural Network Modeling: M ov ing Ta r ge t Pr oble m : Each unit within the network is trying to evolve into a feature detector but input problems are changing constantly. This causes all hidden units to be in a chaotic state, and it takes a long time to settle down.

  9. Problems in Neural Network Modeling: M ov ing Ta r ge t Pr oble m : The Herd Effect: Suppose we have a number of hidden units to solve two tasks. Each unit can not communicate with one another, so they must decide independently which task to tackle. If one task generates a larger error signal, then all units tend to solve this task and ignore the other. Once it has been solved, then all units moves to the second task, but the first problem will re-appear.

  10. Cascade Correlation (CC): Characteristics -> supervised learning algorithm - evaluated on its performance via external source -> a network that determines its own size and topology - starts with input/output layer - builds a minimal multi-layer network by creatingits own hidden layer

  11. Cascade Correlation (CC): Characteristics -> recruits new units according to the residual approximation error - trains and adds hidden units one by one to tackle new tasks, hence “ Cascade ” - the residual error “ Correlat ion ” between the new units and its output is maximized - input weights going into the new hidden unit become frozen (fixed)

  12. Cascade Correlation (CC) -> CC combines two ideas: - cascade architecture: hidden units added one at a time and is frozen - learning algorithm: trains and installs new hidden units

  13. CC Algorithm -> starts with minimal network consisting of a input and output layer. -> train the network with a learning algorithm (ie. Gradient Descent, Simulating Annealing) -> train until no significant error reduction can be measured -> add new hidden unit to reduce residual error

  14. CC Algorithm -> hidden units are added one by one to the network which is connected by all input units and to every pre-existing hidden unit -> freeze all incoming weights of the hidden unit -> repeat until desired performance is reached

  15. Cascade Correlation - Diagram

  16. CC Mathematical Background We want to maximize ‘S’ where S is the sum of all output units. This leads to the creation of very powerful and organized feature detectors (the hidden units).

  17. Example: Speech Recognition The difficulties with speech recognition: -> deciphering different phonetics sounds -> everyone has a different voice!

  18. Example: Speech Recognition A simple example: Designing a network which can classify speech data into one of 10 different phonemes.

  19. Example: Speech Recognition -> train 10 hidden units separately, put them together and train the output unit one by one -> adding new phoneme: train new hidden units for this phoneme and add to the network, and then retrain the output layer

  20. Example: Two-Spirals Problem A primary benchmark for the back- propagation algorithms because it is an extremely hard problem to solve.

  21. Example: Two-Spirals Problem

  22. Example: Two-Spirals Problem

  23. Cascade Correlation (CC) Advantages: -> reduces learning time -> transparent -> creates a structured network

  24. Cascade Correlation (CC) Disadvantages: Can lead to specialization of just the training sets.

  25. Cascade Correlation References 1. Rojas, R. (1996). Neural Networks - A Systematic Introduction. Springer - Verlag Berlin Heidelberg. 2. http://www.mass.u- bordeaux2.fr/~corsini/SNNS_Manual/node164.html 3. ftp://archive.cis.ohio-state.edu/pub/neuroprose/fahlman.cascor- tr.ps.Z

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

852 views • 33 slides
Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

1.17k views • 91 slides
CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides
Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.63k views • 119 slides
Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.57k views • 124 slides
Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography Hopfield, J. J., "Neural networks and physical systems with emergent collective computational abilities," Proceedings of the National Academy

367 views • 19 slides
Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides
(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning objectives What are neural networks? What are deep neural networks? How do we train neural networks? What variants of neural network

331 views • 31 slides
How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . .

How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . .

Need for Machine . . . Neural Networks . . . How to Speed Up . . . Neural Networks: A . . . How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . . . Resulting Algorithm Faster by Taking Into How to

464 views • 17 slides
Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron.

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron.

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron. A look at some current works. Adding an evolutionary strategy. Paul Nuytten CPSC 607 Why Model Neural Networks? The Neuron The nervous

697 views • 4 slides
Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In

987 views • 40 slides
Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Learning in Neural Networks w 1,3 w 3,5 1 3 5 w w 1,4 3,6 w w 2,3 4,5 2 4 6 w w 2,4 4,6 Class #18: Back-Propagation; } A neural network can learn a classification function by Tuning Hyper-Parameters adjusting its weights to

259 views • 4 slides
Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What

Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What

Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What well cover ... f ( x ) computer vision architectures - convolutional networks - data augmentation 1 - residual networks ... ...

1.29k views • 75 slides
Neural Networks Oskar Taubert (SCC) SCC 1 15.01.2020 Oskar Taubert - Neural Networks SCC

Neural Networks Oskar Taubert (SCC) SCC 1 15.01.2020 Oskar Taubert - Neural Networks SCC

Neural Networks Oskar Taubert (SCC) SCC 1 15.01.2020 Oskar Taubert - Neural Networks SCC www.kit.edu KIT The Research University in the Helmholtz Association Neural Network Concept (very) remotely brain inspired computational system

479 views • 25 slides
Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova November 21, 2016 Neural Networks 2/20 Neural Networks 3/20 Neural Networks Neural computing requires a number of neurons , to be connected

813 views • 21 slides
Neural Network Backpropagation 3-2-16 Recall from Monday... Perceptrons can only classify

Neural Network Backpropagation 3-2-16 Recall from Monday... Perceptrons can only classify

Neural Network Backpropagation 3-2-16 Recall from Monday... Perceptrons can only classify linearly separable data. Multi-layer networks Can represent any boolean function. We dont want to build them by hand, so we need a way to

570 views • 11 slides
Lecture 16: Introduction to Neural Networks, Feed-forward Networks and Back-propagation Dr.

Lecture 16: Introduction to Neural Networks, Feed-forward Networks and Back-propagation Dr.

Lecture 16: Introduction to Neural Networks, Feed-forward Networks and Back-propagation Dr. Chengjiang Long Computer Vision Researcher at Kitware Inc. Adjunct Professor at RPI. Email: longc3@rpi.edu Recap Previous Lecture 2 C. Long Lecture

1.12k views • 68 slides
Multi-layer Perceptrons & the Back-propagation Algorithm Instructor: Sham Kakade Please email

Multi-layer Perceptrons & the Back-propagation Algorithm Instructor: Sham Kakade Please email

CSE 446: Machine Learning Lecture Multi-layer Perceptrons & the Back-propagation Algorithm Instructor: Sham Kakade Please email the staff mailing list should you find typos/errors in the notes. Thank you! The treatment in Bishop is good as

288 views • 4 slides
Backpropagation I2DL: Prof. Niessner, Prof. Leal-Taix 1 Lecture 3 Recap I2DL: Prof. Niessner,

Backpropagation I2DL: Prof. Niessner, Prof. Leal-Taix 1 Lecture 3 Recap I2DL: Prof. Niessner,

Optimization and Backpropagation I2DL: Prof. Niessner, Prof. Leal-Taix 1 Lecture 3 Recap I2DL: Prof. Niessner, Prof. Leal-Taix 2 Neural Network Linear score function = On CIFAR-10 Credit: Li/Karpathy/Johnson On

1.75k views • 75 slides
Circuit-GNN: Graph Neural Networks for Distributed Circuit Design Guo Zhang Hao He Dina Katabi

Circuit-GNN: Graph Neural Networks for Distributed Circuit Design Guo Zhang Hao He Dina Katabi

Circuit-GNN: Graph Neural Networks for Distributed Circuit Design Guo Zhang Hao He Dina Katabi Poster #248 Distributed Circuit Design Design high frequency circuits Important for 5G and future cellular networks! Distributed Circuit Design

380 views • 34 slides
A brief history of deep learning 1 Andrew Kurenkov. This summary is based on A Brief

A brief history of deep learning 1 Andrew Kurenkov. This summary is based on A Brief

CS 486/686 Lecture 21 A brief history of deep learning model of neurons in our brain. The output depends only on inputs. Model this by multiplying each input by a continuous valued weight. strengths. Links between neurons are called

392 views • 13 slides
Machine Learning: Chenhao Tan University of Colorado Boulder LECTURE 7 Slides adapted from

Machine Learning: Chenhao Tan University of Colorado Boulder LECTURE 7 Slides adapted from

Machine Learning: Chenhao Tan University of Colorado Boulder LECTURE 7 Slides adapted from Jordan Boyd-Graber, Chris Ketelsen Machine Learning: Chenhao Tan | Boulder | 1 of 39 Final projects WSDM Cup SemEval 2018 Machine Learning:

1.11k views • 67 slides
Cost function Machine Learning Neural Network (Classification) total no. of layers in network

Cost function Machine Learning Neural Network (Classification) total no. of layers in network

Neural Networks: Learning Cost function Machine Learning Neural Network (Classification) total no. of layers in network no. of units (not counting bias unit) in layer Layer 1 Layer 2 Layer 3 Layer 4 Multi-class classification (K classes)

711 views • 33 slides

More recommend