neural networks
play

Neural Networks Many slides attributable to: Prof. Mike Hughes - PowerPoint PPT Presentation

Oct 18, 2022 •222 likes •674 views

Tufts COMP 135: Introduction to Machine Learning https://www.cs.tufts.edu/comp/135/2019s/ Neural Networks Many slides attributable to: Prof. Mike Hughes Erik Sudderth (UCI), Emily Fox (UW), Finale Doshi-Velez (Harvard) James, Witten, Hastie,

  1. Tufts COMP 135: Introduction to Machine Learning https://www.cs.tufts.edu/comp/135/2019s/ Neural Networks Many slides attributable to: Prof. Mike Hughes Erik Sudderth (UCI), Emily Fox (UW), Finale Doshi-Velez (Harvard) James, Witten, Hastie, Tibshirani (ISL/ESL books) 2

  2. Objectives Today : Neural Networks day 10 • How to learn feature representations • Feed-forward neural nets • Single neuron = linear function + activation • Multi-layer perceptrons (MLPs) • Universal approximation • The Rise of Deep Learning: • Success stories on Images and Language Mike Hughes - Tufts COMP 135 - Fall 2020 3

  3. What will we learn? Evaluation Supervised Training Learning Data, Label Pairs Performance { x n , y n } N measure Task n =1 Unsupervised Learning data label x y Reinforcement Learning Prediction Mike Hughes - Tufts COMP 135 - Fall 2020 4

  4. Task: Binary Classification y is a binary variable Supervised (red or blue) Learning binary classification x 2 Unsupervised Learning Reinforcement Learning x 1 Mike Hughes - Tufts COMP 135 - Fall 2020 5

  5. Example: Hotdog or Not https://www.theverge.com/tldr/2017/5/14/15639784/hbo- silicon-valley-not-hotdog-app-download Mike Hughes - Tufts COMP 135 - Fall 2020 6

  6. Text Sentiment Classification Mike Hughes - Tufts COMP 135 - Fall 2020 7

  7. Image Classification Mike Hughes - Tufts COMP 135 - Fall 2020 8

  8. Feature Transform Pipeline Data, Label Pairs { x n , y n } N n =1 Feature, Label Pairs Performance Task { φ ( x n ) , y n } N measure n =1 label data φ ( x ) y x Mike Hughes - Tufts COMP 135 - Fall 2020 9

  9. Predicted Probas vs Binary Labels Mike Hughes - Tufts COMP 135 - Fall 2020 10

  10. Decision Boundary is Linear { x ∈ R 2 : σ ( w T ˜ x ) = 0 . 5 } ←→ { x ∈ R 2 : w T ˜ x = 0 } Mike Hughes - Tufts COMP 135 - Fall 2020 11

  11. Logistic Regr. Network Diagram 0 or 1 Credit: Emily Fox (UW) https://courses.cs.washington.edu/courses/cse41 6/18sp/slides/ Mike Hughes - Tufts COMP 135 - Fall 2020 12

  12. A “Neuron” or “Perceptron” Unit Non-linear Linear function activation with weights w function Credit: Emily Fox (UW) Mike Hughes - Tufts COMP 135 - Fall 2020 13

  13. “Inspired” by brain biology Slide Credit: Bhiksha Raj (CMU) Mike Hughes - Tufts COMP 135 - Fall 2020 14

  14. Challenge: Find w for these functions X_1 X_2 y X_1 X_2 y 0 0 0 0 0 0 0 1 0 0 1 1 1 0 0 1 0 1 1 1 1 1 1 1 Credit: Emily Fox (UW) Mike Hughes - Tufts COMP 135 - Fall 2020 15

  15. Challenge: Find w for these functions X_1 X_2 y X_1 X_2 y 0 0 0 0 0 0 0 1 0 0 1 1 1 0 0 1 0 1 1 1 1 1 1 1 Mike Hughes - Tufts COMP 135 - Fall 2020 Credit: Emily Fox (UW) 16

  16. What we can’t do with linear decision boundary classifiers X_1 X_2 y 0 0 0 0 1 1 1 0 1 1 1 0 Mike Hughes - Tufts COMP 135 - Fall 2020 17

  17. Idea: Compose Neurons together! φ ( x ) y x transform classify Mike Hughes - Tufts COMP 135 - Fall 2020 18

  18. Can you find w to solve XOR? AND/ ? ? OR ? ? ? ? ? ? ? ? ? Mike Hughes - Tufts COMP 135 - Fall 2020 19

  19. Can you find w to solve XOR? ? ? ? ? ? ? ? ? ? Mike Hughes - Tufts COMP 135 - Fall 2020 20

  20. Can you find w to solve XOR? Mike Hughes - Tufts COMP 135 - Fall 2020 21

  21. 1D Input + 3 hidden units Mike Hughes - Tufts COMP 135 - Fall 2020 22

  22. 1D Input + 3 hidden units Example functions (before final threshold) f ( x 1 ) f ( x 1 ) Intuition: Piece-wise step function Partitioning input space into regions Mike Hughes - Tufts COMP 135 - Fall 2020 23

  23. MLPs can approximate any functions with enough hidden units! Mike Hughes - Tufts COMP 135 - Fall 2020 24

  24. Neuron Design What’s wrong with hard step activation function? Non-linear Linear function activation with weights w function Credit: Emily Fox (UW) Mike Hughes - Tufts COMP 135 - Fall 2020 25

  25. Neuron Design What’s wrong with hard step activation function? Not smooth! Gradient is zero almost everywhere, so hard to train weights! Non-linear Linear function activation with weights w function Credit: Emily Fox (UW) Mike Hughes - Tufts COMP 135 - Fall 2020 26

  26. Which Activation Function? Non-linear Linear function activation with weights w function Credit: Emily Fox (UW) Mike Hughes - Tufts COMP 135 - Fall 2020 27

  27. Activation Functions Advice Credit: Emily Fox (UW) Mike Hughes - Tufts COMP 135 - Fall 2020 28

  28. Exciting Applications: Computer Vision Mike Hughes - Tufts COMP 135 - Fall 2020 29

  29. Object Recognition from Images Mike Hughes - Tufts COMP 135 - Fall 2020 30

  30. Deep Neural Networks for Object Recognition Scores for each possible object category Decision: “ leopard ” Mike Hughes - Tufts COMP 135 - Fall 2020 31

  31. Deep Neural Networks for Object Recognition Scores for each possible object category Decision: “ mushroom ” Mike Hughes - Tufts COMP 135 - Fall 2020 32

  32. Each Layer Extracts “Higher Level” Features Mike Hughes - Tufts COMP 135 - Fall 2020 33

  33. More layers = less error! ImageNet challenge 1000 categories, 1.2 million images in training set top 5 classification error (%) shallow 2010 2011 2012 2013 2014 2015 Credit: KDD Tutorial by Sun, Xiao, & Choi: http://dl4health.org/ Figure idea originally from He et. al., CVPR 2016 Mike Hughes - Tufts COMP 135 - Fall 2020 34

  34. 2012 ImageNet Challenge Winner Mike Hughes - Tufts COMP 135 - Fall 2020 35

  35. State of the art Results Mike Hughes - Tufts COMP 135 - Fall 2020 36

  36. Semantic Segmentation Mike Hughes - Tufts COMP 135 - Fall 2020 37

  37. Object Detection Mike Hughes - Tufts COMP 135 - Fall 2020 38

  38. Exciting Applications: Natural Language (Spoken and Written) Mike Hughes - Tufts COMP 135 - Fall 2020 39

  39. Reaching Human Performance in Speech-to-Text https://arxiv.org/pdf/1610.05256.pdf Mike Hughes - Tufts COMP 135 - Fall 2020 40

  40. Gains in Translation Quality https://ai.googleblog.com/2016/09/a-neural-network-for-machine.html Mike Hughes - Tufts COMP 135 - Fall 2020 41

  41. Any Disadvantages? Mike Hughes - Tufts COMP 135 - Fall 2020 42

  42. Deep Neural Networks can overfit! 1 layer Many layers Few units / layer Many units / layer Underfitting Overfitting Mike Hughes - Tufts COMP 135 - Fall 2020 43

  43. Ways to avoid overfitting • More training data! • L2 / L1 penalties on weights • More tricks (next week) …. • Early stopping • Dropout • Data augmentation Mike Hughes - Tufts COMP 135 - Fall 2020 44

  44. Objectives Today : Neural Networks Unit 1/2 • How to learn feature representations • Feed-forward neural nets • Single neuron = linear function + activation • Multi-layer perceptrons (MLPs) • Universal approximation • The Rise of Deep Learning: • Success stories on Images and Language Mike Hughes - Tufts COMP 135 - Fall 2020 45

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

852 views • 33 slides
Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

1.17k views • 91 slides
CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides
Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.57k views • 124 slides
Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.63k views • 119 slides
Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography Hopfield, J. J., "Neural networks and physical systems with emergent collective computational abilities," Proceedings of the National Academy

367 views • 19 slides
Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides
(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning objectives What are neural networks? What are deep neural networks? How do we train neural networks? What variants of neural network

331 views • 31 slides
How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . .

How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . .

Need for Machine . . . Neural Networks . . . How to Speed Up . . . Neural Networks: A . . . How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . . . Resulting Algorithm Faster by Taking Into How to

464 views • 17 slides
Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron.

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron.

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron. A look at some current works. Adding an evolutionary strategy. Paul Nuytten CPSC 607 Why Model Neural Networks? The Neuron The nervous

697 views • 4 slides
Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In

987 views • 40 slides
Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Learning in Neural Networks w 1,3 w 3,5 1 3 5 w w 1,4 3,6 w w 2,3 4,5 2 4 6 w w 2,4 4,6 Class #18: Back-Propagation; } A neural network can learn a classification function by Tuning Hyper-Parameters adjusting its weights to

259 views • 4 slides
Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What

Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What

Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What well cover ... f ( x ) computer vision architectures - convolutional networks - data augmentation 1 - residual networks ... ...

1.29k views • 75 slides
Neural Networks Oskar Taubert (SCC) SCC 1 15.01.2020 Oskar Taubert - Neural Networks SCC

Neural Networks Oskar Taubert (SCC) SCC 1 15.01.2020 Oskar Taubert - Neural Networks SCC

Neural Networks Oskar Taubert (SCC) SCC 1 15.01.2020 Oskar Taubert - Neural Networks SCC www.kit.edu KIT The Research University in the Helmholtz Association Neural Network Concept (very) remotely brain inspired computational system

479 views • 25 slides
Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova November 21, 2016 Neural Networks 2/20 Neural Networks 3/20 Neural Networks Neural computing requires a number of neurons , to be connected

813 views • 21 slides
I is for Turtle, Bagels, Loop Tracing, Files Part 1 of 4 Identity Who are you?

I is for Turtle, Bagels, Loop Tracing, Files Part 1 of 4 Identity Who are you?

Compsci 101 I is for Turtle, Bagels, Loop Tracing, Files Part 1 of 4 Identity Who are you? Computer Science Student Susan Rodger Invariant September 15, 2020 Reasoning formally and informally about loops Internet

289 views • 10 slides
Forward Guidance by Inflation-Targeting Central Banks Michael Woodford Columbia University

Forward Guidance by Inflation-Targeting Central Banks Michael Woodford Columbia University

Forward Guidance by Inflation-Targeting Central Banks Michael Woodford Columbia University Conference Two Decades of Inflation Targeting Sveriges Riksbank June 3, 2013 Michael Woodford (Columbia) Forward Guidance Riksbank June 2013 1

1.05k views • 78 slides
6.003: Signals and Systems Discrete Approximation of Continuous-Time Systems September 29, 2011 1

6.003: Signals and Systems Discrete Approximation of Continuous-Time Systems September 29, 2011 1

6.003: Signals and Systems Discrete Approximation of Continuous-Time Systems September 29, 2011 1 Mid-term Examination #1 Wednesday, October 5, 7:30-9:30pm, No recitations on the day of the exam. Coverage: CT and DT Systems, Z and Laplace

557 views • 40 slides
R and Reproducibility A Proposal David Smith Revolu0on

R and Reproducibility A Proposal David Smith Revolu0on

R and Reproducibility A Proposal David Smith Revolu0on Analy0cs DSC 2014 Why care about reproducibility? Academic / Clinical Research Verify results

175 views • 16 slides
UBC CS 538B Distributed Systems Abstractions with Ivan Beschastnikh COVID Year 2020, Fall term

UBC CS 538B Distributed Systems Abstractions with Ivan Beschastnikh COVID Year 2020, Fall term

UBC CS 538B Distributed Systems Abstractions with Ivan Beschastnikh COVID Year 2020, Fall term Agenda for our first class Who am I, and who are you? Online format Survey results Course overview; reading research papers Break

959 views • 17 slides
Epipolar Geometry EECS 442 David Fouhey Fall 2019, University of Michigan

Epipolar Geometry EECS 442 David Fouhey Fall 2019, University of Michigan

Epipolar Geometry EECS 442 David Fouhey Fall 2019, University of Michigan http://web.eecs.umich.edu/~fouhey/teaching/EECS442_F19/ Multi-view geometry Image Credit: S. Lazebnik Multi-view geometry problems Recovering structure: ? Given

866 views • 54 slides
AIRS Forward Model Validation and Status AIRS Science Team Meeting: Nov/Dec 2004 L. Strow, S.

AIRS Forward Model Validation and Status AIRS Science Team Meeting: Nov/Dec 2004 L. Strow, S.

AIRS Forward Model Validation and Status AIRS Science Team Meeting: Nov/Dec 2004 L. Strow, S. Hannon, S. DeSouza-Machado, H. Motteler UMBC Physics Department and JCET Estimated AIRS RTA accuracy via ARM-TWP and ECMWF validation studies. RTA

517 views • 36 slides
The Purpose History The Mission The Mission The Commission will bring together persons ...and

The Purpose History The Mission The Mission The Commission will bring together persons ...and

6/6/18 A Way Forward The Process C O N V E R S A T I O N The Purpose History The Mission The Mission The Commission will bring together persons ...and exploring the potential future(s) of our deeply committed to the future(s) of The United

197 views • 8 slides

More recommend