lecture 16 introduction to neural networks feed forward
play

Lecture 16: Introduction to Neural Networks, Feed-forward Networks - PowerPoint PPT Presentation

Jul 13, 2023 •440 likes •1.12k views

Lecture 16: Introduction to Neural Networks, Feed-forward Networks and Back-propagation Dr. Chengjiang Long Computer Vision Researcher at Kitware Inc. Adjunct Professor at RPI. Email: longc3@rpi.edu Recap Previous Lecture 2 C. Long Lecture

  1. Lecture 16: Introduction to Neural Networks, Feed-forward Networks and Back-propagation Dr. Chengjiang Long Computer Vision Researcher at Kitware Inc. Adjunct Professor at RPI. Email: longc3@rpi.edu

  2. Recap Previous Lecture 2 C. Long Lecture 16 April 22, 2018

  3. Outline Introduction to Nueral Networks • Feed-Forward Networks • Single-layer Perceptron (SLP) • Multi-layer Perceptron (MLP) • Back-propagation Learning • 3 C. Long Lecture 16 April 22, 2018

  4. Outline Introduction to Nueral Networks • Feed-Forward Networks • Single-layer Perceptron (SLP) • Multi-layer Perceptron (MLP) • Back-propagation Learning • 4 C. Long Lecture 16 April 22, 2018

  5. Neural networks Neural networks are made up of many artificial neurons . • Each input into the neuron has its own weight • associated with it illustrated by the red circle . A weight is simply a floating point number and it ' s these • we adjust when we eventually come to train the network . 5 C. Long Lecture 16 April 22, 2018

  6. Neural networks A neuron can have any number of inputs from one to n, • where n is the total number of inputs. The inputs may be represented therefore as x 1 , x 2 , x 3 … • x n . And the corresponding weights for the inputs as w 1 , w 2 , • w 3 … w n . Output a = x 1 w 1 +x 2 w 2 +x 3 w 3 ... +x n w n • 6 C. Long Lecture 16 April 22, 2018

  7. McCulloch–Pitts “unit” Output is a “squashed” linear function of the inputs : • A gross oversimplification of real neurons , but its • purpose is to develop understanding of what networks of simple units can do 7 C. Long Lecture 16 April 22, 2018

  8. Activation functions 8 C. Long Lecture 16 April 22, 2018

  9. Neural Networks by an Example let ' s design a neural network that will detect the number '4'. • Given a panel made up of a grid of lights which can be either on or • off , we want our neural net to let us know whenever it thinks it sees the character '4'. The panel is eight cells square and looks like this : • the neural net will have 64 inputs , each one representing a • particular cell in the panel and a hidden layer consisting of a number of neurons ( more on this later ) all feeding their output into just one neuron in the output layer 9 9 C. Long Lecture 16 April 22, 2018

  10. Neural Networks by an Example Initialize the neural net with random weights • Feed it a series of inputs which represent , in this • example , the different panel configurations For each configuration we check to see what its output • is and adjust the weights accordingly so that whenever it sees something looking like a number 4 it outputs a 1 and for everything else it outputs a zero . http://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html 10 C. Long Lecture 16 April 22, 2018

  11. A Cheat Sheet of Neural Network Architectures 11 C. Long Lecture 16 April 22, 2018

  12. A Cheat Sheet of Neural Network Architectures 12 C. Long Lecture 16 April 22, 2018

  13. Outline Introduction to Nueral Networks • Feed-Forward Networks • Single-layer Perceptron (SLP) • Multi-layer Perceptron (MLP) • Back-propagation Learning • 13 C. Long Lecture 16 April 22, 2018

  14. Network structures Feed - forward networks : • – single - layer perceptrons – multi - layer perceptrons Feed - forward networks implement functions , have no • internal state 14 C. Long Lecture 16 April 22, 2018

  15. Feed-forward example Feed - forward network = a parameterized family of • nonlinear functions : Adjusting weights changes the function : do learning • this way ! 15 C. Long Lecture 16 April 22, 2018

  16. Outline Introduction to Nueral Networks • Feed-Forward Networks • Single-layer Perceptron (SLP) • Multi-layer Perceptron (MLP) • Back-propagation Learning • 16 C. Long Lecture 16 April 22, 2018

  17. Single-layer perceptrons Output units all operate separately—no shared weights • Adjusting weights moves the location , orientation , and • steepness of cliff 17 C. Long Lecture 16 April 22, 2018

  18. Expressiveness of perceptrons Consider a perceptron with g = step function • ( Rosenblatt , 1957, 1960) Can represent AND , OR , NOT , majority , etc ., but not • XOR Represents a linear separator in input space : • Minsky & Papert (1969) pricked the neural network balloon 18 C. Long Lecture 16 April 22, 2018

  19. Perceptron learning 19 C. Long Lecture 16 April 22, 2018

  20. Perceptron learning contd. Perceptron learning rule converges to a consistent • function for any linearly separable data set Perceptron learns majority function easily, DTL is hopeless DTL learns restaurant function easily, perceptron cannot represent it. 20 C. Long Lecture 16 April 22, 2018

  21. Outline Introduction to Nueral Networks • Feed-Forward Networks • Single-layer Perceptron (SLP) • Multi-layer Perceptron (MLP) • Back-propagation Learning • 21 C. Long Lecture 16 April 22, 2018

  22. Multilayer perceptrons Layers are usually fully connected , and numbers of • hidden units typically chosen by hand 22 C. Long Lecture 16 April 22, 2018

  23. Multi-Layer Perceptron We will introduce the MLP and the backpropagation • algorithm which is used to train it MLP used to describe any general feedforward (no recurrent • connections) network However, we will concentrate on nets with units arranged in • layers x 1 x n 23 C. Long Lecture 16 April 22, 2018

  24. Multi-Layer Perceptron x 1 x n Different books refer to the above as either 4 layer (no. of • layers of neurons) or 3 layer (no. of layers of adaptive weights). We will follow the latter convention. What do the extra layers gain you? Start with looking at • what a single layer can’t do. 24 C. Long Lecture 16 April 22, 2018

  25. How do we actually use an artificial neuron? Feedforward network: The neurons in each layer feed • their output forward to the next layer until we get the final output from the neural network. There can be any number of hidden layers within a • feedforward network. The number of neurons can be completely arbitrary. • 25 C. Long Lecture 16 April 22, 2018

  26. Perceptron Learning Theorem Recap : A perceptron ( threshold unit ) can learn • anything that it can represent ( i . e . anything separable with a hyperplane ) Logical OR Function x1 x2 y 0 0 0 0 1 1 1 0 1 1 1 1 26 C. Long Lecture 16 April 22, 2018

  27. The Exclusive OR problem A Perceptron cannot represent Exclusive OR since it is not linearly separable . Logical XOR Function x1 x2 y 0 0 0 0 1 1 1 0 1 1 1 0 27 27 C. Long Lecture 16 April 22, 2018

  28. Piecewise linear classification using an MLP Minsky & Papert (1969) offered solution to XOR problem • by combining perceptron unit responses using a second layer of Units. Piecewise linear classification using an MLP with threshold (perceptron) units 28 C. Long Lecture 16 April 22, 2018

  29. Piecewise linear classification using an MLP Three-layer networks 29 C. Long Lecture 16 April 22, 2018

  30. Properties of architecture No connections within a layer • No direct connections between input and output layers • Fully connected between layers • Often more than 3 layers • Number of output units need not equal number of input units • Number of hidden units per layer can be more or less than input or • output units Each unit is a perceptron m = + å y f ( w x b ) i i j j i = j 1 30 C. Long Lecture 16 April 22, 2018

  31. What do each of the layers do? 1st layer draws linear 2nd layer combines the 3rd layer can generate arbitrarily boundaries boundaries complex boundaries 31 C. Long Lecture 16 April 22, 2018

  32. Outline Introduction to Nueral Networks • Feed-Forward Networks • Single-layer Perceptron (SLP) • Multi-layer Perceptron (MLP) • Back-propagation Learning • 32 C. Long Lecture 16 April 22, 2018

  33. Backpropagation learning algorithm BP Solution to credit assignment problem in MLP. Rumelhart, • Hinton and Williams (1986) ( though actually invented earlier in a PhD thesis relating to economics) BP has two phases : • Forward pass phase : computes ‘functional signal’, feed forward propagation of input pattern signals through network. Backward pass phase : computes ‘error signal’, propagates the error backwards through network starting at output units (where the error is the difference between actual and desired output values) 33 C. Long Lecture 16 April 22, 2018

  34. Conceptually: Forward Activity - Backward Error Link between hidden node j Output node i and output node i: Wji Hidden node j Link between input node k Input node k and hidden node j: Wkj 34 C. Long Lecture 16 April 22, 2018

  35. Back-propagation derivation The squared error on a single example is defined as • where the sum is over the nodes in the output layer. 35 C. Long Lecture 16 April 22, 2018

  36. Back-propagation derivation contd. 36 C. Long Lecture 16 April 22, 2018

  37. Back-propagation Learning Output layer: same as the single-layer perceptron • where Hidden layer: back-propagation the error from the output • layer. Upadate rules for weights in the hidden layers. • (Most neuroscientists deny that back-propagation occurs in the brain) 37 C. Long Lecture 16 April 22, 2018

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
2nd Session Machine learning: feed-forward neural networks and self-organizing maps 1

2nd Session Machine learning: feed-forward neural networks and self-organizing maps 1

2nd Session Machine learning: feed-forward neural networks and self-organizing maps 1 Recommended reading J. Zupan, J. Gasteiger, Neural Networks in Chemistry and Drug Design: An Introduction, Wiley-VCH, Weinheim, 1999.

959 views • 62 slides
Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network architecture Geoffrey Hinton with Nitish Srivastava Kevin Swersky Feed-forward neural networks These are the commonest type of neural

534 views • 32 slides
Lecture 20: Neural Networks for NLP Zubin Pahuja zpahuja2@illinois.edu

Lecture 20: Neural Networks for NLP Zubin Pahuja zpahuja2@illinois.edu

Lecture 20: Neural Networks for NLP Zubin Pahuja zpahuja2@illinois.edu courses.engr.illinois.edu/cs447 CS447: Natural Language Processing 1 Todays Lecture Feed-forward neural networks as classifiers simple architecture in which

724 views • 37 slides
Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Feed-forward Networks Network Training Error Backpropagation Deep Learning Feed-forward Networks Network Training Error Backpropagation Deep Learning Neural Networks Neural networks arise from attempts to model Neural Networks

380 views • 9 slides
Neural Networks Stefan Edelkamp 1 Overview - Introduction - Percepton - Hofield-Nets -

Neural Networks Stefan Edelkamp 1 Overview - Introduction - Percepton - Hofield-Nets -

Neural Networks Stefan Edelkamp 1 Overview - Introduction - Percepton - Hofield-Nets - Self-Organizing Maps - Feed-Forward Neural Networks - Backpropagation Overview 1 2 Introduction Idea: Mimic principle of biological neural networks

468 views • 34 slides
CS480/680 Lecture 18: July 8, 2019 Recurrent and Recursive Neural Networks [GBC] Chap. 10

CS480/680 Lecture 18: July 8, 2019 Recurrent and Recursive Neural Networks [GBC] Chap. 10

CS480/680 Lecture 18: July 8, 2019 Recurrent and Recursive Neural Networks [GBC] Chap. 10 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Variable length data Traditional feed forward neural networks can only handle fixed

609 views • 18 slides
Neural Networks Oliver Schulte - CMPT 726 Bishop PRML Ch. 5 Feed-forward Networks Network

Neural Networks Oliver Schulte - CMPT 726 Bishop PRML Ch. 5 Feed-forward Networks Network

Feed-forward Networks Network Training Error Backpropagation Applications Neural Networks Oliver Schulte - CMPT 726 Bishop PRML Ch. 5 Feed-forward Networks Network Training Error Backpropagation Applications Neural Networks Neural

462 views • 43 slides
Neural Networks Greg Mori - CMPT 419/726 Bishop PRML Ch. 5 Feed-forward Networks Network

Neural Networks Greg Mori - CMPT 419/726 Bishop PRML Ch. 5 Feed-forward Networks Network

Feed-forward Networks Network Training Error Backpropagation Deep Learning Neural Networks Greg Mori - CMPT 419/726 Bishop PRML Ch. 5 Feed-forward Networks Network Training Error Backpropagation Deep Learning Neural Networks Neural

712 views • 54 slides
Artificial Neural Networks Oliver Schulte - CMPT 726 Feed-forward Networks Network Training

Artificial Neural Networks Oliver Schulte - CMPT 726 Feed-forward Networks Network Training

Feed-forward Networks Network Training Error Backpropagation Applications Artificial Neural Networks Oliver Schulte - CMPT 726 Feed-forward Networks Network Training Error Backpropagation Applications Neural Networks Neural networks

956 views • 46 slides
Neural Networks: Introduction LING572 Advanced Statistical Methods for NLP February 25 2020 1

Neural Networks: Introduction LING572 Advanced Statistical Methods for NLP February 25 2020 1

Neural Networks: Introduction LING572 Advanced Statistical Methods for NLP February 25 2020 1 Unit Overview Introduction: History; Main Ideas / Basic Computation; Landscape Computation in feed-forward networks + Beginning of Learning

516 views • 41 slides
1 Kinds of Networks Feed-forward Single layer Multi-layer Recurrent Kinds of

1 Kinds of Networks Feed-forward Single layer Multi-layer Recurrent Kinds of

CSE 592 Applications of Artificial Intelligence Neural Networks & Data Mining Henry Kautz Winter 2003 1 Kinds of Networks Feed-forward Single layer Multi-layer Recurrent Kinds of Networks Kinds of Networks

308 views • 18 slides
Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) CMSC 678 UMBC Recap

Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) CMSC 678 UMBC Recap

Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) CMSC 678 UMBC Recap from last time Feed-Forward Neural Network: Multilayer Perceptron + 0 ) = (

1.04k views • 93 slides
MICROBOONE Taritree Wongjirad DPF 2017 Tufts/MIT Outline Convolutional neural networks

MICROBOONE Taritree Wongjirad DPF 2017 Tufts/MIT Outline Convolutional neural networks

Run 3493 Event 41075, Oct. 23rd, 2015 CONVOLUTIONAL NEURAL NETWORKS IN MICROBOONE Taritree Wongjirad DPF 2017 Tufts/MIT Outline Convolutional neural networks (CNNs) are a type of deep, feed-forward neural networks that have been

831 views • 43 slides
L101: Feed Forward Neural Networks Linear classifiers e.g. binary logistic regression: And

L101: Feed Forward Neural Networks Linear classifiers e.g. binary logistic regression: And

L101: Feed Forward Neural Networks Linear classifiers e.g. binary logistic regression: And their limitations: http://www.ece.utep.edu/research/webfuzzy/docs/kk-thesis/kk-thesis-html/node19.html What if we could use multiple classifiers?

579 views • 19 slides
The Potential of Memory Augmented Neural Networks Dalton Caron Montana Technological University

The Potential of Memory Augmented Neural Networks Dalton Caron Montana Technological University

The Potential of Memory Augmented Neural Networks Dalton Caron Montana Technological University November 15, 2019 Overview Review of Perceptron and Feed Forward Networks Recurrent Neural Networks Neural Turing Machines

1.27k views • 57 slides
Multi-layer Perceptrons & the Back-propagation Algorithm Instructor: Sham Kakade Please email

Multi-layer Perceptrons & the Back-propagation Algorithm Instructor: Sham Kakade Please email

CSE 446: Machine Learning Lecture Multi-layer Perceptrons & the Back-propagation Algorithm Instructor: Sham Kakade Please email the staff mailing list should you find typos/errors in the notes. Thank you! The treatment in Bishop is good as

288 views • 4 slides
Backpropagation I2DL: Prof. Niessner, Prof. Leal-Taix 1 Lecture 3 Recap I2DL: Prof. Niessner,

Backpropagation I2DL: Prof. Niessner, Prof. Leal-Taix 1 Lecture 3 Recap I2DL: Prof. Niessner,

Optimization and Backpropagation I2DL: Prof. Niessner, Prof. Leal-Taix 1 Lecture 3 Recap I2DL: Prof. Niessner, Prof. Leal-Taix 2 Neural Network Linear score function = On CIFAR-10 Credit: Li/Karpathy/Johnson On

1.75k views • 75 slides
Lecture 4: Backpropagation and Neural Networks part 1 Fei-Fei Li & Andrej Karpathy &

Lecture 4: Backpropagation and Neural Networks part 1 Fei-Fei Li & Andrej Karpathy &

Lecture 4: Backpropagation and Neural Networks part 1 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 1 Administrative A1 is due

1.26k views • 84 slides
The Embedded Graphs of a Knot and the Partial Duals of a Plane Graph Iain Moffatt University of

The Embedded Graphs of a Knot and the Partial Duals of a Plane Graph Iain Moffatt University of

The Embedded Graphs of a Knot and the Partial Duals of a Plane Graph Iain Moffatt University of South Alabama SIAM Conference on Discrete Mathematics, 14 th June 2010 I. Moffatt (South Alabama) Partial Duals of a Plane Graph SIAM DM 10 1

542 views • 22 slides
Neural Network Backpropagation 3-2-16 Recall from Monday... Perceptrons can only classify

Neural Network Backpropagation 3-2-16 Recall from Monday... Perceptrons can only classify

Neural Network Backpropagation 3-2-16 Recall from Monday... Perceptrons can only classify linearly separable data. Multi-layer networks Can represent any boolean function. We dont want to build them by hand, so we need a way to

570 views • 11 slides
Modular Neural Networks CPSC 533 Franco Lee Ian Ko Modular Neural Networks What is it ? Dif

Modular Neural Networks CPSC 533 Franco Lee Ian Ko Modular Neural Networks What is it ? Dif

Modular Neural Networks CPSC 533 Franco Lee Ian Ko Modular Neural Networks What is it ? Dif f erent models of neural net w orks combined int o a single syst em. Each single net w ork is made int o a module t hat can be f reely int

289 views • 25 slides
Circuit-GNN: Graph Neural Networks for Distributed Circuit Design Guo Zhang Hao He Dina Katabi

Circuit-GNN: Graph Neural Networks for Distributed Circuit Design Guo Zhang Hao He Dina Katabi

Circuit-GNN: Graph Neural Networks for Distributed Circuit Design Guo Zhang Hao He Dina Katabi Poster #248 Distributed Circuit Design Design high frequency circuits Important for 5G and future cellular networks! Distributed Circuit Design

380 views • 34 slides
A brief history of deep learning 1 Andrew Kurenkov. This summary is based on A Brief

A brief history of deep learning 1 Andrew Kurenkov. This summary is based on A Brief

CS 486/686 Lecture 21 A brief history of deep learning model of neurons in our brain. The output depends only on inputs. Model this by multiplying each input by a continuous valued weight. strengths. Links between neurons are called

392 views • 13 slides

More recommend