csc 411 lecture 11 neural networks ii
play

CSC 411: Lecture 11: Neural Networks II Class based on Raquel - PowerPoint PPT Presentation

Apr 09, 2024 •414 likes •1.42k views

CSC 411: Lecture 11: Neural Networks II Class based on Raquel Urtasun & Rich Zemels lectures Sanja Fidler University of Toronto March 2, 2016 Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 1 / 55 Today

  1. CSC 411: Lecture 11: Neural Networks II Class based on Raquel Urtasun & Rich Zemel’s lectures Sanja Fidler University of Toronto March 2, 2016 Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 1 / 55

  2. Today Deep learning for Object Recognition Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 2 / 55

  3. Neural Nets for Object Recognition People are very good at recognizing shapes Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 3 / 55

  4. Neural Nets for Object Recognition People are very good at recognizing shapes I Intrinsically di ffi cult, computers are bad at it Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 3 / 55

  5. Neural Nets for Object Recognition People are very good at recognizing shapes I Intrinsically di ffi cult, computers are bad at it Why is it di ffi cult? Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 3 / 55

  6. Why is it a Problem? Di ffi cult scene conditions [From: Grauman & Leibe] Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 4 / 55

  7. Why is it a Problem? Huge within-class variations. Recognition is mainly about modeling variation. [Pic from: S. Lazebnik] Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 5 / 55

  8. Why is it a Problem? Tones of classes [Biederman] Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 6 / 55

  9. Neural Nets for Object Recognition People are very good at recognizing shapes I Intrinsically di ffi cult, computers are bad at it Some reasons why it is di ffi cult: Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 7 / 55

  10. Neural Nets for Object Recognition People are very good at recognizing shapes I Intrinsically di ffi cult, computers are bad at it Some reasons why it is di ffi cult: I Segmentation: Real scenes are cluttered Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 7 / 55

  11. Neural Nets for Object Recognition People are very good at recognizing shapes I Intrinsically di ffi cult, computers are bad at it Some reasons why it is di ffi cult: I Segmentation: Real scenes are cluttered I Invariances: We are very good at ignoring all sorts of variations that do not a ff ect shape Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 7 / 55

  12. Neural Nets for Object Recognition People are very good at recognizing shapes I Intrinsically di ffi cult, computers are bad at it Some reasons why it is di ffi cult: I Segmentation: Real scenes are cluttered I Invariances: We are very good at ignoring all sorts of variations that do not a ff ect shape I Deformations: Natural shape classes allow variations (faces, letters, chairs) Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 7 / 55

  13. Neural Nets for Object Recognition People are very good at recognizing shapes I Intrinsically di ffi cult, computers are bad at it Some reasons why it is di ffi cult: I Segmentation: Real scenes are cluttered I Invariances: We are very good at ignoring all sorts of variations that do not a ff ect shape I Deformations: Natural shape classes allow variations (faces, letters, chairs) I A huge amount of computation is required Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 7 / 55

  14. How to Deal with Large Input Spaces How can we apply neural nets to images? Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 8 / 55

  15. How to Deal with Large Input Spaces How can we apply neural nets to images? Images can have millions of pixels, i.e., x is very high dimensional Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 8 / 55

  16. How to Deal with Large Input Spaces How can we apply neural nets to images? Images can have millions of pixels, i.e., x is very high dimensional How many parameters do I have? Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 8 / 55

  17. How to Deal with Large Input Spaces How can we apply neural nets to images? Images can have millions of pixels, i.e., x is very high dimensional How many parameters do I have? Prohibitive to have fully-connected layers Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 8 / 55

  18. How to Deal with Large Input Spaces How can we apply neural nets to images? Images can have millions of pixels, i.e., x is very high dimensional How many parameters do I have? Prohibitive to have fully-connected layers What can we do? Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 8 / 55

  19. How to Deal with Large Input Spaces How can we apply neural nets to images? Images can have millions of pixels, i.e., x is very high dimensional How many parameters do I have? Prohibitive to have fully-connected layers What can we do? We can use a locally connected layer Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 8 / 55

  20. Locally Connected Layer Example: 200x200 image 40K hidden units Filter size: 10x10 4M parameters Note: This parameterization is good when input image is registered (e.g., 34 face recognition). Ra Ranzato Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 9 / 55

  21. When Will this Work? When Will this Work? Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 10 / 55

  22. When Will this Work? When Will this Work? This is good when the input is (roughly) registered Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 10 / 55

  23. General Images The object can be anywhere [Slide: Y. Zhu] Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 11 / 55

  24. General Images The object can be anywhere [Slide: Y. Zhu] Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 12 / 55

  25. General Images The object can be anywhere [Slide: Y. Zhu] Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 13 / 55

  26. Locally Connected Layer STATIONARITY? Statistics is similar at different locations Example: 200x200 image 40K hidden units Filter size: 10x10 4M parameters Note: This parameterization is good when input image is registered (e.g., 35 face recognition). Ranzato Ra Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 14 / 55

  27. The replicated feature approach Adopt approach apparently used in monkey visual systems The red connections all Use many di ff erent copies of the same have the same weight. feature detector. I Copies have slightly di ff erent positions. I Could also replicate across scale and orientation. I Tricky and expensive I Replication reduces number of free parameters to be learned. Use several di ff erent feature types, each with its own replicated pool of detectors. 5 I Allows each patch of image to be represented in several ways. Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 15 / 55

  28. Convolutional Neural Net Idea: statistics are similar at di ff erent locations (Lecun 1998) Connect each hidden unit to a small input patch and share the weight across space This is called a convolution layer and the network is a convolutional network Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 16 / 55

  29. Convolutional Layer Ra Ranzato K h n X h n − 1 ∗ w n j = max(0 , jk ) k k =1 Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 17 / 55

  30. Convolutional Layer Ra Ranzato K X h n − 1 h n ∗ w n j = max(0 , jk ) k k =1 Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 18 / 55

  31. Convolutional Layer Ra Ranzato K X h n − 1 h n ∗ w n j = max(0 , jk ) k k =1 Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 19 / 55

  32. Convolutional Layer Ra Ranzato K X h n − 1 h n ∗ w n j = max(0 , jk ) k k =1 Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 20 / 55

  33. Convolutional Layer Ra Ranzato K h n X h n − 1 ∗ w n j = max(0 , jk ) k k =1 Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 21 / 55

  34. Convolutional Layer Ra Ranzato K h n X h n − 1 ∗ w n j = max(0 , jk ) k k =1 Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 22 / 55

  35. Convolutional Layer Learn multiple filters. E.g.: 200x200 image 100 Filters Filter size: 10x10 10K parameters 54 Ra Ranzato Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 23 / 55

  36. Convolutional Layer Figure: Left: CNN, right: Each neuron computes a linear and activation function Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 24 / 55

  37. Convolutional Layer Figure: Left: CNN, right: Each neuron computes a linear and activation function Hyperparameters of a convolutional layer: Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 24 / 55

  38. Convolutional Layer Figure: Left: CNN, right: Each neuron computes a linear and activation function Hyperparameters of a convolutional layer: The number of filters (controls the depth of the output volume) Urtasun, Zemel, Fidler (UofT) CSC 411: 11-Neural Networks II March 2, 2016 24 / 55

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network architecture Geoffrey Hinton with Nitish Srivastava Kevin Swersky Feed-forward neural networks These are the commonest type of neural

534 views • 32 slides
Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang

Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang

Deep Learning Basics Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang Introduction Recurrent neural networks Dates back to (Rumelhart et al. , 1986) A family of neural networks for handling

348 views • 34 slides
Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

CSCI 5525 Machine Learning Fall 2019 Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1 Convolutional Neural Networks We will now study a special type of neural networks convolutional neural

662 views • 6 slides
Lecture 20: Neural Networks for NLP Zubin Pahuja zpahuja2@illinois.edu

Lecture 20: Neural Networks for NLP Zubin Pahuja zpahuja2@illinois.edu

Lecture 20: Neural Networks for NLP Zubin Pahuja zpahuja2@illinois.edu courses.engr.illinois.edu/cs447 CS447: Natural Language Processing 1 Todays Lecture Feed-forward neural networks as classifiers simple architecture in which

724 views • 37 slides
Lecture 22 Interpolation, Overfitting, Ridgeless Regression, and Neural Networks Sasha Rakhlin

Lecture 22 Interpolation, Overfitting, Ridgeless Regression, and Neural Networks Sasha Rakhlin

Lecture 22 Interpolation, Overfitting, Ridgeless Regression, and Neural Networks Sasha Rakhlin Nov 21, 2019 1 / 32 In the previous lecture, we derived upper bounds on Rademacher averages of a set of neural networks in terms of norms of weight

640 views • 32 slides
Lecture 16: Introduction to Neural Networks, Feed-forward Networks and Back-propagation Dr.

Lecture 16: Introduction to Neural Networks, Feed-forward Networks and Back-propagation Dr.

Lecture 16: Introduction to Neural Networks, Feed-forward Networks and Back-propagation Dr. Chengjiang Long Computer Vision Researcher at Kitware Inc. Adjunct Professor at RPI. Email: longc3@rpi.edu Recap Previous Lecture 2 C. Long Lecture

1.12k views • 68 slides
Lecture 18: Concluding Convolutional Neural Networks, Graphical Models as Foundation for Recurrent

Lecture 18: Concluding Convolutional Neural Networks, Graphical Models as Foundation for Recurrent

Lecture 18: Concluding Convolutional Neural Networks, Graphical Models as Foundation for Recurrent Neural Networks and Bayesian Networks Reference: We will be referring to sections etc of Deep Learning by Yoshua Bengio, Ian J. Goodfellow

974 views • 54 slides
Evolving Neural Networks This lecture is based on Xin Yaos tutorial slides From Evolving

Evolving Neural Networks This lecture is based on Xin Yaos tutorial slides From Evolving

Evolving Neural Networks This lecture is based on Xin Yaos tutorial slides From Evolving Single Neural Networks to Evolving Ensembles presented at CEC 2007. Literature: Evolving Artificial Neural Networks , Xin Yao, Proceedings of the

281 views • 16 slides
Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
Learning From Data Lecture 22 Neural Networks and Overfitting Approximation vs. Generalization

Learning From Data Lecture 22 Neural Networks and Overfitting Approximation vs. Generalization

Learning From Data Lecture 22 Neural Networks and Overfitting Approximation vs. Generalization Regularization and Early Stopping Minimizing E in More Efficienty M. Magdon-Ismail CSCI 4100/6100 recap: Neural Networks and Fitting the Data

464 views • 15 slides
Decision support systems and machine learning Lecture 11 Lecture 11 p. 1/24 Neural networks:

Decision support systems and machine learning Lecture 11 Lecture 11 p. 1/24 Neural networks:

Decision support systems and machine learning Lecture 11 Lecture 11 p. 1/24 Neural networks: Biological and artificial Consider humans: Properties of artificial neural nets (ANNs): Neuron switching time 0 . 001 second Many

404 views • 36 slides
CS485/685 Lecture 7: Jan 24, 2012 Perceptrons, Neural Networks [B]: Sections 4.1.7, 5.1 CS485/685

CS485/685 Lecture 7: Jan 24, 2012 Perceptrons, Neural Networks [B]: Sections 4.1.7, 5.1 CS485/685

25/01/2012 CS485/685 Lecture 7: Jan 24, 2012 Perceptrons, Neural Networks [B]: Sections 4.1.7, 5.1 CS485/685 (c) 2012 P. Poupart 1 Outline Neural networks Perceptron Supervised learning algorithms for neural networks CS485/685 (c)

326 views • 15 slides
Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

1.17k views • 91 slides
Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

852 views • 33 slides
Lecture 5 Intro to machine learning and single-layer neural networks Kai Olav Ellefsen This

Lecture 5 Intro to machine learning and single-layer neural networks Kai Olav Ellefsen This

Biologically inspired computing Lecture 5 Intro to machine learning and single-layer neural networks Kai Olav Ellefsen This Lecture 1. Introduction to learning/classification 2. Biological neuron 3. Perceptron and artificial neural networks

585 views • 55 slides
Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture

Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture

Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture you should understand the following concepts convolutional neural networks (CNN) convolution and its advantage pooling and its advantage 2

750 views • 50 slides
Advanced Analytics in Business [D0S07a] Big Data Platforms & Technologies [D0S06a] Artificial

Advanced Analytics in Business [D0S07a] Big Data Platforms & Technologies [D0S06a] Artificial

Advanced Analytics in Business [D0S07a] Big Data Platforms & Technologies [D0S06a] Artificial Neural Networks Deep Learning Q-learning Opening the Black Box (Part 2) Overview Artificial neural networks Deep learning (conceptual)

1.67k views • 117 slides
Interactive Media and Game Development 2-D Tiles and Sprites Outline Tiles Sprites

Interactive Media and Game Development 2-D Tiles and Sprites Outline Tiles Sprites

Interactive Media and Game Development 2-D Tiles and Sprites Outline Tiles Sprites More material: Ari Feldman. Designing Arcade Computer Game Graphics , Online at: http://www.gamemaker.nl/feldman/full.zip Tsugumo. So You

550 views • 16 slides
Woodlands odlands Ri Ring g Prima imary y School hool Primary 4 Parents Briefing 30 30

Woodlands odlands Ri Ring g Prima imary y School hool Primary 4 Parents Briefing 30 30

Woodlands odlands Ri Ring g Prima imary y School hool Primary 4 Parents Briefing 30 30 Ja Janu nuar ary y 20 2015 15 Sc Scie ience nce De Depa partme tment nt STEL TELLAR LAR @ P3 @ P3 Primary Science Syllabus How

521 views • 24 slides
Mixing R with other languages JOHN D. COOK, PHD SINGULAR VALUE CONSULTING Why R? Libraries,

Mixing R with other languages JOHN D. COOK, PHD SINGULAR VALUE CONSULTING Why R? Libraries,

Mixing R with other languages JOHN D. COOK, PHD SINGULAR VALUE CONSULTING Why R? Libraries, libraries, libraries De facto standard for statistical research Nice language, as far as statistical languages go Quirky, flawed, and

510 views • 27 slides
Social Media and Opinion Mining ( ) 2016/10/25 ()

Social Media and Opinion Mining ( ) 2016/10/25 ()

Tamkang University Social Media and Opinion Mining ( ) 2016/10/25 () (2:10-5:00pm) 270407407 Min-Yuh Day Assistant

1.69k views • 145 slides
Deep Learning by Doing arconsis IT-Solutions GmbH Who? Wolfgang Frank Achim Baier

Deep Learning by Doing arconsis IT-Solutions GmbH Who? Wolfgang Frank Achim Baier

Deep Learning by Doing arconsis IT-Solutions GmbH Who? Wolfgang Frank Achim Baier @wolfgangfrank @arconsis Lets talk about Cucumber we will get back to that later! Why learn about Machine Learning? Web Search News Search

1.16k views • 76 slides
Introduction to Deep Learning Princeton University COS 495 Instructor: Yingyu Liang What is deep

Introduction to Deep Learning Princeton University COS 495 Instructor: Yingyu Liang What is deep

Introduction to Deep Learning Princeton University COS 495 Instructor: Yingyu Liang What is deep learning? Short answer: recent buzz word Industry Google Facebook Microsoft Musk Toyota Drug Finance

282 views • 26 slides
Budget Trends SOCC Charges Business Model Proposal Discussion Budget Trends

Budget Trends SOCC Charges Business Model Proposal Discussion Budget Trends

OSC Fall Update Budget Trends SOCC Charges Business Model Proposal Discussion Budget Trends Revenue Analysis OSC Revenue Categories: Revenue Trend State Subsidy Sponsored Research Funding User Fees

502 views • 22 slides

More recommend