generalisation bounds for neural networks
play

Generalisation Bounds for Neural Networks Pascale Gourdeau - PowerPoint PPT Presentation

Feb 07, 2024 •242 likes •1.05k views

Generalisation Bounds for Neural Networks Pascale Gourdeau University of Oxford 15 November 2018 Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 1 / 31 Overview Introduction 1 General Strategies to

  1. Generalisation Bounds for Neural Networks Pascale Gourdeau University of Oxford 15 November 2018 Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 1 / 31

  2. Overview Introduction 1 General Strategies to Obtain Generalisation Bounds 2 Survey of Generalisation Bounds for Neural Networks 3 A Compression Approach [Arora et al., 2018] 4 Conclusion, Research Directions 5 Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 2 / 31

  3. Overview Introduction 1 General Strategies to Obtain Generalisation Bounds 2 Survey of Generalisation Bounds for Neural Networks 3 A Compression Approach [Arora et al., 2018] 4 Conclusion, Research Directions 5 Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 3 / 31

  4. What is generalisation? The ability to perform well on unseen data. Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 4 / 31

  5. What is generalisation? The ability to perform well on unseen data. Assumption: the data (both for the training and testing) comes i.i.d. from a distribution D . Usually work in a distribution-agnostic setting. Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 4 / 31

  6. What are generalisation bounds? Classification setting : input space X and output space Y := { 1 , . . . , k } with a distribution D on X × Y . Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 5 / 31

  7. What are generalisation bounds? Classification setting : input space X and output space Y := { 1 , . . . , k } with a distribution D on X × Y . Goal: to learn a function f : X → Y from a sample S := { ( x i , y i ) } m i =1 ⊆ X × Y . Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 5 / 31

  8. What are generalisation bounds? Classification setting : input space X and output space Y := { 1 , . . . , k } with a distribution D on X × Y . Goal: to learn a function f : X → Y from a sample S := { ( x i , y i ) } m i =1 ⊆ X × Y . Generalisation bounds: bounding the difference between the expected and empirical losses of f with high probability over S . Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 5 / 31

  9. What are generalisation bounds? Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 6 / 31

  10. What are generalisation bounds? For neural networks, we use the expected classification loss: � � L 0 ( f ) := P ( x , y ) ∼ D f ( x ) y ≤ max y ′ � = y f ( x ) y ′ , Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 6 / 31

  11. What are generalisation bounds? For neural networks, we use the expected classification loss: � � L 0 ( f ) := P ( x , y ) ∼ D f ( x ) y ≤ max y ′ � = y f ( x ) y ′ , and the empirical margin loss: � � m � L γ ( f ) := 1 � f ( x ) y ≤ γ + max y ′ � = y f ( x ) y ′ . 1 m i =1 Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 6 / 31

  12. Why are generalisation bounds useful? They allow us to quantify a given model’s expected generalisation performance. Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 7 / 31

  13. Why are generalisation bounds useful? They allow us to quantify a given model’s expected generalisation performance. E.g.: With probability 95% over the training sample, the error is at most 1%. Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 7 / 31

  14. Why are generalisation bounds useful? They allow us to quantify a given model’s expected generalisation performance. E.g.: With probability 95% over the training sample, the error is at most 1%. They can also: Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 7 / 31

  15. Why are generalisation bounds useful? They allow us to quantify a given model’s expected generalisation performance. E.g.: With probability 95% over the training sample, the error is at most 1%. They can also: Provide insight on the ability of a model to generalise. Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 7 / 31

  16. Why are generalisation bounds useful? They allow us to quantify a given model’s expected generalisation performance. E.g.: With probability 95% over the training sample, the error is at most 1%. They can also: Provide insight on the ability of a model to generalise. This is of particular interest for us: neural networks have many counter-intuitive properties. Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 7 / 31

  17. Why are generalisation bounds useful? They allow us to quantify a given model’s expected generalisation performance. E.g.: With probability 95% over the training sample, the error is at most 1%. They can also: Provide insight on the ability of a model to generalise. This is of particular interest for us: neural networks have many counter-intuitive properties. Inspire new algorithms or regularisation techniques. Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 7 / 31

  18. Overview Introduction 1 General Strategies to Obtain Generalisation Bounds 2 Survey of Generalisation Bounds for Neural Networks 3 A Compression Approach [Arora et al., 2018] 4 Conclusion, Research Directions 5 Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 8 / 31

  19. General Strategies Generalisation bounds (GB) for neural networks are usually obtained by 1 Defining a class H of functions computed by neural networks with certain properties (e.g., weight matrices with bounded norms, number of layers, etc.), Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 9 / 31

  20. General Strategies Generalisation bounds (GB) for neural networks are usually obtained by 1 Defining a class H of functions computed by neural networks with certain properties (e.g., weight matrices with bounded norms, number of layers, etc.), 2 Deriving a generalisation bound in terms of a complexity measure M ( H ) (e.g. size of H , Rademacher complexity), Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 9 / 31

  21. General Strategies Generalisation bounds (GB) for neural networks are usually obtained by 1 Defining a class H of functions computed by neural networks with certain properties (e.g., weight matrices with bounded norms, number of layers, etc.), 2 Deriving a generalisation bound in terms of a complexity measure M ( H ) (e.g. size of H , Rademacher complexity), 3 Upper bounding M ( H ) in terms of model parameters (e.g., norm of weight matrices, number of layers, etc.). Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 9 / 31

  22. General Strategies: Rademacher Complexity Definition (Rademacher complexity) Let G be a family of functions from a set Z to R . Let σ 1 , . . . , σ m be Rademacher variables: P ( σ i = 1) = P ( σ i = − 1) = 1 / 2. The empirical Rademacher complexity of G w.r.t. to a sample S = { z i } m i =1 is Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 10 / 31

  23. General Strategies: Rademacher Complexity Definition (Rademacher complexity) Let G be a family of functions from a set Z to R . Let σ 1 , . . . , σ m be Rademacher variables: P ( σ i = 1) = P ( σ i = − 1) = 1 / 2. The empirical Rademacher complexity of G w.r.t. to a sample S = { z i } m i =1 is � � m � 1 R S ( G ) = E σ sup σ i g ( z i ) . m g ∈ G i =1 Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 10 / 31

  24. General Strategies: Rademacher Complexity Definition (Rademacher complexity) Let G be a family of functions from a set Z to R . Let σ 1 , . . . , σ m be Rademacher variables: P ( σ i = 1) = P ( σ i = − 1) = 1 / 2. The empirical Rademacher complexity of G w.r.t. to a sample S = { z i } m i =1 is � � m � 1 R S ( G ) = E σ sup σ i g ( z i ) . m g ∈ G i =1 Intuition: How much G correlates with random noise on S . Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 10 / 31

  25. General Strategies: Rademacher Complexity Definition (Rademacher complexity) Let G be a family of functions from a set Z to R . Let σ 1 , . . . , σ m be Rademacher variables: P ( σ i = 1) = P ( σ i = − 1) = 1 / 2. The empirical Rademacher complexity of G w.r.t. to a sample S = { z i } m i =1 is � � m � 1 R S ( G ) = E σ sup σ i g ( z i ) . m g ∈ G i =1 Intuition: How much G correlates with random noise on S . Simple examples... Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 10 / 31

  26. General Strategies: Rademacher Complexity Theorem Let G be a family of functions from Z to [0 , 1] , and let S be a sample of size m drawn from Z according to D. Let L ( g ) = E z ∼ D [ g ( z )] and � m � L ( g ) = 1 i =1 g ( z i ) . Then for any δ > 0 , with probability at least 1 − δ m over S, for all functions g ∈ G, �� � log(1 /δ ) L ( g ) ≤ � L ( g ) + 2 R S ( G ) + O . m Pascale Gourdeau (University of Oxford) Generalisation Bounds for NNs 15 November 2018 11 / 31

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Nearly-tight VC-dimension bounds for piecewise linear neural networks Nicholas J. A. Harvey,

Nearly-tight VC-dimension bounds for piecewise linear neural networks Nicholas J. A. Harvey,

Nearly-tight VC-dimension bounds for piecewise linear neural networks Nicholas J. A. Harvey, Christopher Liaw , Abbas Mehrabian University of British Columbia COLT 17 July 10, 2017 Neural networks () = max {, 0} (ReLU)

531 views • 20 slides
Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks Yuan Cao

Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks Yuan Cao

Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks Yuan Cao and Quanquan Gu Computer Science Department 1 / 14 Learning Over-parameterized DNNs Empirical observation on extremely wide deep neural networks

631 views • 14 slides
Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
Lecture 22 Interpolation, Overfitting, Ridgeless Regression, and Neural Networks Sasha Rakhlin

Lecture 22 Interpolation, Overfitting, Ridgeless Regression, and Neural Networks Sasha Rakhlin

Lecture 22 Interpolation, Overfitting, Ridgeless Regression, and Neural Networks Sasha Rakhlin Nov 21, 2019 1 / 32 In the previous lecture, we derived upper bounds on Rademacher averages of a set of neural networks in terms of norms of weight

640 views • 32 slides
Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

1.17k views • 91 slides
Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

852 views • 33 slides
CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides
Using Learning Dynamics to Understand Neural Network Generalisation Arnu Pretorius Supervisor:

Using Learning Dynamics to Understand Neural Network Generalisation Arnu Pretorius Supervisor:

Using Learning Dynamics to Understand Neural Network Generalisation Arnu Pretorius Supervisor: Dr. Steve Kroon (Computer Science ) Co-supervisor: Dr. Herman Kamper (EE Engineering + ) 20 October 2017 Maties Machine Learning, Stellenbosch

447 views • 11 slides
Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.63k views • 119 slides
Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.57k views • 124 slides
Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography Hopfield, J. J., "Neural networks and physical systems with emergent collective computational abilities," Proceedings of the National Academy

367 views • 19 slides
Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit Song Mei,

Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit Song Mei,

Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit Song Mei, Theodor Misiakiewicz, and Andrea Montanari Stanford University June 26, 2019 COLT 2019 Song Mei (Stanford University) Mean Field Dynamics for

1.12k views • 29 slides
Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides
(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning objectives What are neural networks? What are deep neural networks? How do we train neural networks? What variants of neural network

331 views • 31 slides
How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . .

How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . .

Need for Machine . . . Neural Networks . . . How to Speed Up . . . Neural Networks: A . . . How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . . . Resulting Algorithm Faster by Taking Into How to

464 views • 17 slides
Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron.

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron.

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron. A look at some current works. Adding an evolutionary strategy. Paul Nuytten CPSC 607 Why Model Neural Networks? The Neuron The nervous

697 views • 4 slides
Intro to Context-Aware Computing Matthew Lee 05-899 Special Topics in Ubiquitous Computing

Intro to Context-Aware Computing Matthew Lee 05-899 Special Topics in Ubiquitous Computing

Intro to Context-Aware Computing Matthew Lee 05-899 Special Topics in Ubiquitous Computing Readings Context-Aware Computing Applications, by Bill Schilit, Norman Adams, and Roy Want Ask not for whom the cell phone tolls: Some problems

923 views • 27 slides
Whose Bad Choices? How Policy Precludes Prosperity and What We Can Do About It February 6, 2018

Whose Bad Choices? How Policy Precludes Prosperity and What We Can Do About It February 6, 2018

Whose Bad Choices? How Policy Precludes Prosperity and What We Can Do About It February 6, 2018 Todays Speakers Kristin Lawton Kasey Wiedrich Director of Director of Applied Communications Research @krislawt Andrea Levere Jeremie

963 views • 70 slides
Inference and Representation David Sontag New York University Lecture 10, Nov. 17, 2015

Inference and Representation David Sontag New York University Lecture 10, Nov. 17, 2015

Inference and Representation David Sontag New York University Lecture 10, Nov. 17, 2015 Acknowledgements : Partially based on slides by Eric Xing at CMU and Andrew McCallum at UMass Amherst David Sontag (NYU) Inference and Representation

282 views • 24 slides
54% of workers say that financial matters 66% of employers agreed that employees are

54% of workers say that financial matters 66% of employers agreed that employees are

EMPLOYER SOLUTIONS : FROM EMERGENCY TO RESILIENCY Presented by June 25, 2020 54% of workers say that financial matters 66% of employers agreed that employees are cause them the most stress at work less productive at work when

703 views • 55 slides
Attila Szegedi, Software Engineer @asz 1 Everything I ever learned about JVM performance

Attila Szegedi, Software Engineer @asz 1 Everything I ever learned about JVM performance

Attila Szegedi, Software Engineer @asz 1 Everything I ever learned about JVM performance tuning @twitter 2 Everything More than I ever wanted to learned about JVM performance tuning @twitter 3 Twitters biggest enemy Latency 4

600 views • 59 slides
Measuring inflectional complexity: French and Mauritian Olivier Bonami 1 e 2 Fabiola Henri 3

Measuring inflectional complexity: French and Mauritian Olivier Bonami 1 e 2 Fabiola Henri 3

Measuring inflectional complexity: French and Mauritian Olivier Bonami 1 e 2 Fabiola Henri 3 Gilles Boy 1 U. Paris-Sorbonne & Institut Universitaire de France 2 U. de Bordeaux 3 U. Sorbonne Nouvelle QMMMD San Diego, January 15, 2011

740 views • 43 slides
Guide Cylinder/Compact Type Courtesy of CMA/Flodyne/Hydradyne Motion Control Hydraulic

Guide Cylinder/Compact Type Courtesy of CMA/Flodyne/Hydradyne Motion Control Hydraulic

Guide Cylinder/Compact Type Courtesy of CMA/Flodyne/Hydradyne Motion Control Hydraulic Pneumatic Electrical Mechanical (800) 426-5480 www.cmafh.com Series MGC 20, 25, 32, 40, 50 Linear Transfer Unit with

451 views • 28 slides
Assignment # 2 So You Want to Write a Physically Based Motion Which is something you may wish

Assignment # 2 So You Want to Write a Physically Based Motion Which is something you may wish

Assignment # 2 So You Want to Write a Physically Based Motion Which is something you may wish to do System since it is Assignment # 2 Assignment # 2 Assignment # 2 Write a system that will simulate a Program may be: single shot of a

57 views • 5 slides

More recommend