training neural networks
play

Training Neural Networks Some considerations Gaurav Kumar Center - PowerPoint PPT Presentation

Mar 18, 2024 •982 likes •1.19k views

Training Neural Networks Some considerations Gaurav Kumar Center for Language and Speech Processing gkumar@cs.jhu.edu Universal Approximators Neural networks can approximate [1] function. any Capacity Layers hidden layer size

  1. Training Neural Networks Some considerations Gaurav Kumar Center for Language and Speech Processing gkumar@cs.jhu.edu

  2. Universal Approximators • Neural networks can approximate [1] function. any • Capacity • Layers • hidden layer size • A bsence of regularization • Optimal activation functions and hyper-parameters. • Training data [1] K. Hornik, M. Stinchcombe, and H. White. 1989. Multilayer feedforward networks are universal approximators. Neural Netw. 2, 5 (July 1989) : proved this for a specific class of functions.

  3. Universal Approximators • We will focus on two important aspects of training: • Ideal properties of parameters during training • Generalization error • Other things to consider: • Hyper-parameter optimization • Choice of model, loss functions • Learning rates (Use Adadelta or Adam) • …

  4. Properties of Parameters • Responsive to activation functions • Numerically stable

  5. Activation Saturation Sigmoid Relu

  6. Initialization of weight matrices • Are you using a non-recurrent NN ? • Use the Xavier initialization • (use small values to initialize bias vectors) Glorot & Bengio (2010), He et.al (2015)

  7. Initialization of weight matrices (Xavier, He) • Tanh • Sigmoid • Relu

  8. Initialization of weight matrices • Are you using a recurrent NN ? • With LSTMs : Use the Saxe initialization • All weight matrices initialized to be orthonormal (Gaussian noise -> SVD) • Without LSTMS • All weight matrices initialized to identity Saxe et al, 2014,

  9. Watch your input • A high variance in input features may cause saturation very early • Mean subtraction : Same mean across all features • Normalization : Same scale across all features

  10. Numerical stability • Floating point precision causes values to overflow or underflow • Instead, compute

  11. Numerical stability L = − t log( p ) − (1 − t )log(1 − p ) • Cross Entropy Loss • Probabilities close to 0 for the correct label will cause underflow • Use range clipping. All values between 0.000001 and 0.999999.

  12. Generalization Preventing Overfitting

  13. Regularization • L2 regularization • L1 regularization • Gradient clipping (max norm constraints)

  14. Regularization • Perform layer-wise regularization • After computing the activated value of each layer, normalize with the L2 norm. • No regularization hyper-parameters • No waiting till back-propagation for weight penalties to flow in

  15. Dropout Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15

  16. Dropout

  17. Dropout • Interpret as regularization • Interpret as training an ensemble of thinned networks

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

TRAINING NEURAL TRAINING NEURAL NETWORKS ON THE NETWORKS ON THE EDGE EDGE Navjot Kukreja,

TRAINING NEURAL TRAINING NEURAL NETWORKS ON THE NETWORKS ON THE EDGE EDGE Navjot Kukreja,

TRAINING NEURAL TRAINING NEURAL NETWORKS ON THE NETWORKS ON THE EDGE EDGE Navjot Kukreja, Alena Shilova Also: Olivier Beaumont Jan Huckelheim Nicola Ferrier Paul Hovland Gerard Gorman BACKGROUND BACKGROUND Typical data ow pattern

779 views • 35 slides
Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides
CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides
Modern Systems for Neural Networks Valentin Dalibard This talk 1.Practicalities of training

Modern Systems for Neural Networks Valentin Dalibard This talk 1.Practicalities of training

Modern Systems for Neural Networks Valentin Dalibard This talk 1.Practicalities of training Neural Networks 2.Leveraging heterogeneous hardware Source: wikipedia Modern Neural Networks Applications: Image classification Modern Neural

513 views • 20 slides
Training Neural Networks CMSC 470 Marine Carpuat Neural Networks so far Powerful non-linear

Training Neural Networks CMSC 470 Marine Carpuat Neural Networks so far Powerful non-linear

Training Neural Networks CMSC 470 Marine Carpuat Neural Networks so far Powerful non-linear models for classification Predictions are made as a sequence of simple operations matrix-vector operations non-linear activation functions

417 views • 14 slides
Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Feed-forward Networks Network Training Error Backpropagation Deep Learning Feed-forward Networks Network Training Error Backpropagation Deep Learning Neural Networks Neural networks arise from attempts to model Neural Networks

380 views • 9 slides
Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural circuits David Sussillo with Valerio Mante and Bill Newsome Table of contents Introduc(on Training recurrent neural networks(RNNs) Understanding how

1.57k views • 105 slides
DL2: Training and Querying Neural Networks with Logic Ma Marc Fischer er , Mislav Balunovi ,

DL2: Training and Querying Neural Networks with Logic Ma Marc Fischer er , Mislav Balunovi ,

DL2: Training and Querying Neural Networks with Logic Ma Marc Fischer er , Mislav Balunovi , Dana Drachsler-Cohen, Timon Gehr, Ce Zhang, Martin Vechev github.com/eth-sri/dl2 differencing neural networks finding inputs that deactivates [Pei

992 views • 13 slides
Introduction to Neural Networks Slides from L. Lazebnik, B. Hariharan Outline Perceptrons

Introduction to Neural Networks Slides from L. Lazebnik, B. Hariharan Outline Perceptrons

Introduction to Neural Networks Slides from L. Lazebnik, B. Hariharan Outline Perceptrons Perceptron update rule Multi-layer neural networks Training method Best practices for training classifiers After that:

505 views • 25 slides
Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural

Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural

Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks Alexander Shevchenko Marco Mondelli Neural Network Training From theoretical perspective training of neural networks is di ffi cult

494 views • 28 slides
Neural Networks and Handwriting Recognition Background Neural Networks Neural Network Steven

Neural Networks and Handwriting Recognition Background Neural Networks Neural Network Steven

Neural Networks and Handwriting Recognition Steven Sloss Math 164 Neural Networks and Handwriting Recognition Background Neural Networks Neural Network Steven Sloss Structure Training Neural Networks Math 164 Motivation Problem

888 views • 41 slides
Population Based Training of Neural Networks M. Jaderberg, V. Dalibard, S. Osindero, W.M.

Population Based Training of Neural Networks M. Jaderberg, V. Dalibard, S. Osindero, W.M.

Presented by: Devin Taylor Population Based Training of Neural Networks M. Jaderberg, V. Dalibard, S. Osindero, W.M. Czarnecki November 14, 2018 DeepMind, London, United Kingdom Problem Statement Problem statement Neural networks suffer from

317 views • 17 slides
Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan,

FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan,

FLEET: Flexible Efficient Ensemble Training for Heterogenous Deep Neural Networks Hui Guan, Laxmikant Kishor Mokadam, Xipeng Shen, Seung-Hwan Lim, Robert Patton 1 Build an image classifier? Deep Neural Network (DNN) CPU GPU Training

957 views • 41 slides
Data Parallelism in Training Sparse Neural Networks Namhoon Lee 1 , Philip Torr 1 , Martin Jaggi 2

Data Parallelism in Training Sparse Neural Networks Namhoon Lee 1 , Philip Torr 1 , Martin Jaggi 2

Data Parallelism in Training Sparse Neural Networks Namhoon Lee 1 , Philip Torr 1 , Martin Jaggi 2 1 University of Oxford, 2 EPFL ICLR 2020 Workshop on PML4DC Motivation Compressing neural networks can save a large amount of memory and

570 views • 21 slides
What Can Neural Networks Teach us about Language? Graham Neubig a2-dlearn 11/18/2017 Supervised

What Can Neural Networks Teach us about Language? Graham Neubig a2-dlearn 11/18/2017 Supervised

What Can Neural Networks Teach us about Language? Graham Neubig a2-dlearn 11/18/2017 Supervised Training of Neural Networks for Language Training Data Training Model this is an example the cat went to the store Prediction Results

1.07k views • 62 slides
Modeling/Checking Approximate Computing Sara Achour, David Bindel, Luis Ceze, Eva Darulova,

Modeling/Checking Approximate Computing Sara Achour, David Bindel, Luis Ceze, Eva Darulova,

Modeling/Checking Approximate Computing Sara Achour, David Bindel, Luis Ceze, Eva Darulova, Andreas Gerstlauer, Karthik Pattabiraman, Ulya Karpuzcu, Subhasish Mitra Goal overview Paper 1: Modeling Approximation Errors at the ISA - Goal:

136 views • 11 slides
LARP PS2 Budget Guidance FY10 By Lab: k$ By Task: k$ LBNL 262 Sp-Ch sim. 173 FNAL 39

LARP PS2 Budget Guidance FY10 By Lab: k$ By Task: k$ LBNL 262 Sp-Ch sim. 173 FNAL 39

LARP PS2 Budget Guidance FY10 By Lab: k$ By Task: k$ LBNL 262 Sp-Ch sim. 173 FNAL 39 Instabilities 130 BNL 133 e-Cloud 199 SLAC 237 feedback 40 Total: 671 IPM: 133 Total 675 By Lab & Project FY10 PersonYear tot SLAC

462 views • 8 slides
Numerical Relativity simulations of black holes: Methodology and Computational Framework U.

Numerical Relativity simulations of black holes: Methodology and Computational Framework U.

Numerical Relativity simulations of black holes: Methodology and Computational Framework U. Sperhake CSIC-IEEC Barcelona Numerical Cosmology 2012 19 th July 2012 U. Sperhake (CSIC-IEEC) Numerical Relativity simulations of black holes:

833 views • 67 slides
Differential Equations Overview of differential equation Initial value problem

Differential Equations Overview of differential equation Initial value problem

Differential Equations Overview of differential equation Initial value problem Explicit numeric methods Implicit numeric methods Modular implementation Physics-based simulation Its an algorithm that produces a

653 views • 63 slides
ttPrt r

ttPrt r

ttPrt r rss r r s

441 views • 22 slides
Low-frequency stability analysis of periodic traveling-wave solutions of viscous conservation

Low-frequency stability analysis of periodic traveling-wave solutions of viscous conservation

Low-frequency stability analysis of periodic traveling-wave solutions of viscous conservation laws in several dimensions Myunghyun Oh Department of Mathematics University of Kansas Stability Analysis, HYP2006 p.1/16 Outline of the talk

491 views • 48 slides
Stability of quantum many-body systems with point interactions Robert Seiringer IST Austria

Stability of quantum many-body systems with point interactions Robert Seiringer IST Austria

Stability of quantum many-body systems with point interactions Robert Seiringer IST Austria Joint work with Thomas Moser arXiv:1609.08342, Commun. Math. Phys. (in press) Quantissima in the Serenissima II Venice, August 2125, 2017 R.

279 views • 13 slides
Information-theoretic Planck scale cutoff: Predictions for the CMB Achim Kempf With: A.

Information-theoretic Planck scale cutoff: Predictions for the CMB Achim Kempf With: A.

Information-theoretic Planck scale cutoff: Predictions for the CMB Achim Kempf With: A. Chatwin-Davies (CalTech), R. Martin (U. Cape Town) Departments of Applied Mathematics and Physics Institute for Quantum Computing, University of Waterloo

349 views • 33 slides

More recommend