neural networks ch 12 back propagation
play

Neural networks (Ch. 12) Back-propagation The neural network is as - PowerPoint PPT Presentation

May 30, 2023 •976 likes •1.18k views

Neural networks (Ch. 12) Back-propagation The neural network is as good as it's structure and weights on edges Structure we will ignore (more complex), but there is an automated way to learn weights Whenever a NN incorrectly answer a problem,

  1. Neural networks (Ch. 12)

  2. Back-propagation The neural network is as good as it's structure and weights on edges Structure we will ignore (more complex), but there is an automated way to learn weights Whenever a NN incorrectly answer a problem, the weights play a “blame game”... - Weights that have a big impact to the wrong answer are reduced

  3. Back-propagation To do this blaming, we have to find how much each weight influenced the final answer Steps: 1. Find total error 2. Find derivative of error w.r.t. weights 3. Penalize each weight by an amount proportional to this derivative

  4. Back-propagation Consider this example: 4 nodes, 2 layers w 1 w 5 out 1 in 1 3 1 w 3 w 7 w 6 w 2 out 2 in 2 2 4 w 4 w 8 b 1 b 2 1 This node as a constant bias of 1

  5. Neural network: feed-forward One commonly used function is the sigmoid:

  6. Back-propagation .4 out 1 in 1 .15 3 1 0.05 .5 .25 .45 .2 out 2 0.1 in 2 4 2 .55 .3 0.35 1 0.6 Node 1: 0.15*0.05 + 0.2*0.1 +0.35 as input thus it outputs (all edges) S(0.3775)=0.59327

  7. Back-propagation .4 out 1 in 1 .15 3 1 0.05 .5 .25 .45 .2 out 2 0.1 in 2 4 2 .55 .3 0.35 1 0.6 Eventually we get: out 1 = 0.7513, out 2 = 0.7729 Suppose wanted: out 1 = 0.01, out 2 = 0.99

  8. Back-propagation We will define the error as: (you will see why shortly) Suppose we want to find how much w 5 is to blame for our incorrectness We then need to find: Apply the chain rule:

  9. Back-propagation

  10. Back-propagation In a picture we did this: Now that we know w5 is 0.08217 part responsible, we update the weight by: w 5 ←w 5 - α * 0.08217 = 0.3589 (from 0.4) α is learning rate, set to 0.5

  11. Back-propagation Updating this w 5 to w 8 gives: w 5 = 0.3589 w 6 = 0.4067 w 7 = 0.5113 w 8 = 0.5614 For other weights, you need to consider all possible ways in which they contribute

  12. Back-propagation For w 1 it would look like: (book describes how to dynamic program this)

  13. Back-propagation Specifically for w 1 you would get: Next we have to break down the top equation...

  14. Back-propagation

  15. Back-propagation Similarly for Error 2 we get: You might notice this is small... This is an issue with neural networks, deeper the network the less earlier nodes update

  16. NN examples Despite this learning shortcoming, NN are useful in a wide range of applications: Reading handwriting Playing games Face detection Economic predictions Neural networks can also be very powerful when combined with other techniques (genetic algorithms, search techniques, ...)

  17. NN examples Examples: https://www.youtube.com/watch?v=umRdt3zGgpU https://www.youtube.com/watch?v=qv6UVOQ0F44 https://www.youtube.com/watch?v=xcIBoPuNIiw https://www.youtube.com/watch?v=0Str0Rdkxxo https://www.youtube.com/watch?v=l2_CPB0uBkc https://www.youtube.com/watch?v=0VTI1BBLydE

  18. NN examples AlphaGo/Zero has been in the news recently, and is also based on neural networks AlphaGo uses Monte-Carlo tree search guided by the neural network to prune useless parts Often limiting Monte-Carlo in a static way reduces the effectiveness, much like mid-state evaluations can limit algorithm effectiveness

  19. NN examples Basically, AlphaGo uses a neural network to “prune” parts for a Monte-carlo search

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Propagating Error Backward A Back-Propagation Example /* Propagate deltas backward from output

Propagating Error Backward A Back-Propagation Example /* Propagate deltas backward from output

Learning in Neural Networks w 1,3 w 3,5 1 3 5 w w 1,4 3,6 w w 2,3 4,5 2 4 6 w w 2,4 4,6 Class #21: Back-Propagation; } A neural network can learn a classification function by Tuning Hyper-Parameters adjusting its weights to

342 views • 6 slides
Neural Networks: Multi-Layer Networks & Back-Propagation M. Soleymani Artificial

Neural Networks: Multi-Layer Networks & Back-Propagation M. Soleymani Artificial

Neural Networks: Multi-Layer Networks & Back-Propagation M. Soleymani Artificial Intelligence Sharif University of Technology Spring 2019 Most slides have been adapted from Bhiksha Raj, 11-785, CMU 2019 and some from Fei Fei Li et. al,

1.01k views • 100 slides
Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Learning in Neural Networks w 1,3 w 3,5 1 3 5 w w 1,4 3,6 w w 2,3 4,5 2 4 6 w w 2,4 4,6 Class #18: Back-Propagation; } A neural network can learn a classification function by Tuning Hyper-Parameters adjusting its weights to

259 views • 4 slides
Lecture 16: Introduction to Neural Networks, Feed-forward Networks and Back-propagation Dr.

Lecture 16: Introduction to Neural Networks, Feed-forward Networks and Back-propagation Dr.

Lecture 16: Introduction to Neural Networks, Feed-forward Networks and Back-propagation Dr. Chengjiang Long Computer Vision Researcher at Kitware Inc. Adjunct Professor at RPI. Email: longc3@rpi.edu Recap Previous Lecture 2 C. Long Lecture

1.12k views • 68 slides
Introduction to Data Science: Neural [ 1 , 2 , , p ] g x w h m g h g f w old M

Introduction to Data Science: Neural [ 1 , 2 , , p ] g x w h m g h g f w old M

Projection-Pursuit Regression Feed-forward Neural Networks Fitting with back propagation Multiple Minima Fitting with back propagation Projection-Pursuit Regression Multiple Minima Multiple Minima Fitting with back propagation Multiple

1.38k views • 111 slides
Layer-wise Relevance Propagation in Neural Neural Networks Deep Learning Shortcomings Networks

Layer-wise Relevance Propagation in Neural Neural Networks Deep Learning Shortcomings Networks

LRP Ariyan Zarei Motivation Having More interpretable Layer-wise Relevance Propagation in Neural Neural Networks Deep Learning Shortcomings Networks to have more interpretable Papers and Demo Introduction Machine Learning models

373 views • 35 slides
ECON 950 Winter 2020 Prof. James MacKinnon 11. Neural Networks Neural networks go back many

ECON 950 Winter 2020 Prof. James MacKinnon 11. Neural Networks Neural networks go back many

ECON 950 Winter 2020 Prof. James MacKinnon 11. Neural Networks Neural networks go back many decades, but they have recently become a very hot topic because of major improvements in performance. ESL says that the most widely used neural

307 views • 17 slides
A signal propagation perspective for pruning neural networks at initialization Namhoon Lee 1 ,

A signal propagation perspective for pruning neural networks at initialization Namhoon Lee 1 ,

A signal propagation perspective for pruning neural networks at initialization Namhoon Lee 1 , Thalaiyasingam Ajanthan 2 , Stephen Gould 2 , Philip Torr 1 1 University of Oxford, 2 Australian National University ICLR 2020 Spotlight presentation

242 views • 21 slides
Learning From Data Lecture 21 Neural Networks: Backpropagation Forward propagation: algorithmic

Learning From Data Lecture 21 Neural Networks: Backpropagation Forward propagation: algorithmic

Learning From Data Lecture 21 Neural Networks: Backpropagation Forward propagation: algorithmic computation h ( x ) e ( x ) Backpropagation: algorithmic computation of weights M. Magdon-Ismail CSCI 4100/6100 recap: The Neural Network

294 views • 14 slides
Algorithms in Nature Pruning in neural networks Neural network development 1. Efficient signal

Algorithms in Nature Pruning in neural networks Neural network development 1. Efficient signal

Algorithms in Nature Pruning in neural networks Neural network development 1. Efficient signal propagation [e.g. information processing & integration] 2. Robust to noise and failures [e.g. cell or synapse failure] 3. Cost-aware design [e.g.

874 views • 33 slides
Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang

Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang

Deep Learning Basics Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang Introduction Recurrent neural networks Dates back to (Rumelhart et al. , 1986) A family of neural networks for handling

348 views • 34 slides
Multi-Layer Networks & Back-Propagation M. Soleymani Deep Learning Sharif University of

Multi-Layer Networks & Back-Propagation M. Soleymani Deep Learning Sharif University of

Multi-Layer Networks & Back-Propagation M. Soleymani Deep Learning Sharif University of Technology Spring 2019 Most slides have been adapted from Bhiksha Raj, 11-785, CMU 2019 and some from Fei Fei Li et. al, cs231n, Stanford 2017 These

1.47k views • 120 slides
Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

852 views • 33 slides
Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

1.17k views • 91 slides
Neural Networks in Supreme Commander 2 Mike Sorian Robbins Gameplay Engineer

Neural Networks in Supreme Commander 2 Mike Sorian Robbins Gameplay Engineer

Neural Networks in Supreme Commander 2 Mike Sorian Robbins Gameplay Engineer twitter.com/merobbins The Four Neural Networks Back Propagating Neural Network DPS HP Speed Input 34 nodes Hidden 98 nodes Output 15 nodes Attack

480 views • 28 slides
COMP30019 Graphics and Interaction Rendering pipeline & object modelling Adrian Pearce

COMP30019 Graphics and Interaction Rendering pipeline & object modelling Adrian Pearce

Introduction to Modelling Polygonal geometry The rendering pipeline High Performance Visible Surface Detection COMP30019 Graphics and Interaction Rendering pipeline & object modelling Adrian Pearce Department of Computing and Information

629 views • 20 slides
Surveillance Monitoring Tnis Uiboupin Pejman Rasti (Head of Image Processing division of iCV

Surveillance Monitoring Tnis Uiboupin Pejman Rasti (Head of Image Processing division of iCV

Facial Image Super Resolution Using Sparse Representation for Improving Face Recognition in Surveillance Monitoring Tnis Uiboupin Pejman Rasti (Head of Image Processing division of iCV Group) Gholamreza Anbarjafari, Outline Problem

595 views • 28 slides
Non Photorealistic Rendering BY DMYTRO TKACHUK COMPUTER GRAPHICS SEMINAR Non photorealistic

Non Photorealistic Rendering BY DMYTRO TKACHUK COMPUTER GRAPHICS SEMINAR Non photorealistic

Non Photorealistic Rendering BY DMYTRO TKACHUK COMPUTER GRAPHICS SEMINAR Non photorealistic rendering Non photorealistic rendering (NPR) is a process by which computer engineers try to animate and represent items inspired by paintings,

593 views • 35 slides
Visible Surface Detection (Chapt. 15 in FVD, Chapt. 13 in Hearn & Baker) 1 Given a set

Visible Surface Detection (Chapt. 15 in FVD, Chapt. 13 in Hearn & Baker) 1 Given a set

Visible Surface Detection (Chapt. 15 in FVD, Chapt. 13 in Hearn & Baker) 1 Given a set of 3D objects and a viewing specifications, determine which lines or surfaces of the objects should be visible. A surface might be occluded by

1.13k views • 56 slides
In Search of a Unifying Theory for Image Interpretation Donald Gem an Department of Applied

In Search of a Unifying Theory for Image Interpretation Donald Gem an Department of Applied

In Search of a Unifying Theory for Image Interpretation Donald Gem an Department of Applied Mathematics and Statistics and Center for Imaging Science, Whitaker Institute Johns Hopkins University Outline Semantic Scene Interpretation

992 views • 84 slides
MICROBOONE Taritree Wongjirad DPF 2017 Tufts/MIT Outline Convolutional neural networks

MICROBOONE Taritree Wongjirad DPF 2017 Tufts/MIT Outline Convolutional neural networks

Run 3493 Event 41075, Oct. 23rd, 2015 CONVOLUTIONAL NEURAL NETWORKS IN MICROBOONE Taritree Wongjirad DPF 2017 Tufts/MIT Outline Convolutional neural networks (CNNs) are a type of deep, feed-forward neural networks that have been

831 views • 43 slides
Visual Recognition: Prospects for Image & Video Analytics Jitendra Malik University of

Visual Recognition: Prospects for Image & Video Analytics Jitendra Malik University of

Visual Recognition: Prospects for Image & Video Analytics Jitendra Malik University of California at Berkeley Classification & Segmentation Water outdoor Grass wildlife Tiger Sand back Tiger head eye legs tail mouth shadow

427 views • 41 slides
Department of Computer Science IV University of Mannheim, Germany Motivation Part I: Basic

Department of Computer Science IV University of Mannheim, Germany Motivation Part I: Basic

Stephan Kopf Department of Computer Science IV University of Mannheim, Germany Motivation Part I: Basic Retargeting Operations Scaling and cropping Regions of interest Automatic crop & scale Sports video adaptation

949 views • 50 slides

More recommend