Neural Tangent Kernel Convergence and Generalization in Neural - PowerPoint PPT Presentation

Feb 16, 2024 •190 likes •248 views

Neural Tangent Kernel Convergence and Generalization in Neural Networks Arthur Jacot Franck Gabriel Clment Hongler arthur.jacot@epfl.ch franck.gabriel@epfl.ch clement.hongler@epfl.ch What happens during training? One step of Gradient

Neural Tangent Kernel Convergence and Generalization in Neural Networks Arthur Jacot Franck Gabriel Clément Hongler arthur.jacot@epfl.ch franck.gabriel@epfl.ch clement.hongler@epfl.ch
What happens during training? One step of Gradient Descent Neural Tangent Kernel: One datapoint x0 Describes the effect of gradient descent on the network function
Determines the trajectory of the network function during training In the Infinite width limit: - Deterministic - Fixed in time - Explicit formula
Kernel methods Neural Networks Kernel Gradient Descent Gradient Descent Positive definite NTK Convergence to a global min. Least-squares loss Kernel ridge regression
What happens inside a very wide network? - The activations of the hidden neurons become independent - The parameters and activations evolve less and less - However all layers learn: The sum of all microscopic changes yields a macroscopic effect

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Differential structure, tangent structure, and SDG Geoff Cruttwell (joint work with Robin

Introduction Tangent structure Tangent spaces Representable tangent structure Conclusion Differential structure, tangent structure, and SDG Geoff Cruttwell (joint work with Robin Cockett) In honour of Robin Cocketts 60th birthday FMCS

755 views • 58 slides

General Neural Networks Compositions of linear maps and component-wise non- linearities Neural

Pooling and Invariance in Convolutional Neural Networks General Neural Networks Compositions of linear maps and component-wise non- linearities Neural Networks Common Non-Linearities Rectifier Linear Unit Sigmoid Hyperbolic tangent

844 views • 53 slides

When do neural networks outperform kernel methods? Song Mei Stanford University June 29, 2020

When do neural networks outperform kernel methods? Song Mei Stanford University June 29, 2020 Joint work with Behrooz Ghorbani, Theodor Misiakiewicz, and Andrea Montanari Song Mei (Stanford University) Neural Networks and Kernel Methods June

645 views • 26 slides

Weil spaces and closed tangent structure June 2, 2018 1 / 30 Overview W 1 -actegories

Weil spaces and closed tangent structure June 2, 2018 1 / 30 Overview W 1 -actegories Tangent categories 1 W 1 -presheaf cats 2 Embedding theorem Closed and representable tangent categories 3 2 / 30 Tangent categories and Weil

559 views • 32 slides

Connections in Tangent Categories Robin Cockett University of Calgary (joint work with Geoff

Connections in Tangent Categories Connections in Tangent Categories Robin Cockett University of Calgary (joint work with Geoff Cruttwell) Category Theory 2016 Dalhousie Connections in Tangent Categories Tangent categories The basics

807 views • 41 slides

Recurrent Neural Networks with Flexible Gates using Kernel Activation Functions Authors : S.

2018 IEEE International Workshop on Machine Learning for Signal Processing (MLSP18) Recurrent Neural Networks with Flexible Gates using Kernel Activation Functions Authors : S. Scardapane, S. Van Vaerenbergh, D. Comminiello, S. Totaro and A.

545 views • 40 slides

A Neural Network View of Kernel Methods Shuiwang Ji Department of Computer Science &

A Neural Network View of Kernel Methods Shuiwang Ji Department of Computer Science & Engineering Texas A&M University 1 / 24 Linear Models 1 In a binary classification problem, we have training data { , y i } m i =1 ,

351 views • 24 slides

A Kernel Perspective for Regularizing Deep Neural Networks Julien Mairal Inria Grenoble Imaging

A Kernel Perspective for Regularizing Deep Neural Networks Julien Mairal Inria Grenoble Imaging and Machine Learning, IHP, 2019 Julien Mairal A Kernel Perspective for Regularizing NN 1/1 Publications Theoretical Foundations A. Bietti and

1.2k views • 105 slides

Tangent estimation along 3D digital curves l Postolski 1 , 2 , Marcin Janaszewski 2 , Yukiko

Introduction Discrete Straight Segments -Maximal Segment Tangent Direction Experimental Validation Tangent estimation along 3D digital curves l Postolski 1 , 2 , Marcin Janaszewski 2 , Yukiko Kenmochi 1 , Micha Jacques-Olivier Lachaud 3 1

613 views • 26 slides

Bird Call Recognition with Artificial Neural Networks, Support Vector Machines, and Kernel

M. Sc. Thesis Presentation Bird Call Recognition with Artificial Neural Networks, Support Vector Machines, and Kernel Density Estimation Derek Ross Advisors: Dr. H. Card, Electrical and Computer Eng. Dr. D. McNeill, Electrical and Computer

650 views • 64 slides

Differential Programming and Tangent Categories Work with Geoff Cruttwell and Ben MacAdam Wh

Differential Programming and Tangent Categories Work with Geoff Cruttwell and Ben MacAdam Wh What is s this talk lk about? Apply techniques from synthetic differential geometry and tangent categories to understand internal homs in

251 views • 24 slides

A Kernel Perspective for Regularizing Deep Neural Networks Alberto Bietti* Grgoire Mialon*

A Kernel Perspective for Regularizing Deep Neural Networks Alberto Bietti* Grgoire Mialon* Dexiong Chen Julien Mairal Inria ICML 2019, Long Beach Bietti, Mialon, Chen and Mairal Kernel regularization of deep nets ICML 2019, Long Beach 1

417 views • 17 slides

Tight Kernel Query Complexity of Kernel Ridge Regression and Kernel -means Clustering Manuel

Tight Kernel Query Complexity of Kernel Ridge Regression and Kernel -means Clustering Manuel Fernndez V, David P. Woodruff, Taisuke Yasuda Kernel Method Many machine learning tasks can be expressed as a function of the inner product

389 views • 22 slides

Tangent categories are locally Cartesian differential categories J.R.B. Cockett Department of

Tangent categories are locally Cartesian differential categories Tangent categories are locally Cartesian differential categories J.R.B. Cockett Department of Computer Science University of Calgary Alberta, Canada robin@cpsc.ucalgary.ca

825 views • 33 slides

1 Kernel methods & optimization One example of a kernel that is frequently used in practice

Machine Learning Class Notes 9-25-12 Prof. David Sontag 1 Kernel methods & optimization One example of a kernel that is frequently used in practice and which allows for highly non-linear discriminant functions is the Gaussian kernel,

240 views • 5 slides

Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit Song Mei,

Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit Song Mei, Theodor Misiakiewicz, and Andrea Montanari Stanford University June 26, 2019 COLT 2019 Song Mei (Stanford University) Mean Field Dynamics for

1.12k views • 29 slides

Is There a Demand for Reverse Mortgages in China? Evidence from Two Online Surveys Katja Hanewald

Is There a Demand for Reverse Mortgages in China? Evidence from Two Online Surveys Katja Hanewald (UNSW, CEPAR), Hazel Bateman (UNSW, CEPAR) Hanming Fang (University of Pennsylvania, ShanghaiTech, CEPAR) Shang Wu (First State Super, CEPAR)

683 views • 35 slides

What Gets Echoed? Understanding the Pointers in Explanations of Persuasive Arguments David

What Gets Echoed? Understanding the Pointers in Explanations of Persuasive Arguments David Atkinson, Kumar Bhargav Srinivasan, and Chenhao Tan {david.i.atkinson, kumar.srinivasan, chenhao.tan}@colorado.edu Explanations are important.

549 views • 37 slides

CrIS Internal Target Emissivity Check From Day in the Life Test Data NASA Sounder Science Team

CrIS Internal Target Emissivity Check From Day in the Life Test Data NASA Sounder Science Team Meeting Mark Esplin, Kevin Grant, Vladimir Zavyalov, and Chad Fish CrIS Sensor On The NPP Satellite ICT (Warm) 8-second scans 8 sec 30 Earth

234 views • 22 slides

Equivalence Class Testing Chapter 6 ECT1 Introduction What problems does boundary value

Equivalence Class Testing Chapter 6 ECT1 Introduction What problems does boundary value testing have? What are the motivations for equivalence class testing? ECT2 Introduction 2 Boundary Value Testing derives test cases

467 views • 31 slides

Foundations of algebraic surgery ANDREW RANICKI (Edinburgh) http://www.maths.ed.ac.uk/ aa

Foundations of algebraic surgery ANDREW RANICKI (Edinburgh) http://www.maths.ed.ac.uk/ aa r An n -dimensional manifold M determines an n -dimensional cellula r f.g. free ab elian group chain complex ) d ) d . . . d ( M

314 views • 20 slides

TOGETHER AT LAST Ravi Chugh, Brian Hempel, Mitchell Spradlin, Jacob Albers Program 2 Program 2

PROGRAMMATIC AND DIRECT MANIPULATION TOGETHER AT LAST Ravi Chugh, Brian Hempel, Mitchell Spradlin, Jacob Albers Program 2 Program 2 Changed Program Program 2 Changed Program Program 2 Changed Changed Program Program Program

1.81k views • 166 slides

#GenomicMed10 transcript Healthcare Social Media Transcript From: Mon May 01 0:00:00 PDT 2017

#GenomicMed10 transcript Healthcare Social Media Transcript From: Mon May 01 0:00:00 PDT 2017 To: Thu May 04 0:00:00 PDT 2017 change time period What is #GenomicMed10 ? Who were the influencers during this time period? #GenomicMed10 analytics

174 views • 5 slides

The Social Organization of the R Project John Fox McMaster University useR 2008 Dortmund John

Introduction Trajectory Development and Organization Why Did R Succeed? Can Success Continue? The Social Organization of the R Project John Fox McMaster University useR 2008 Dortmund John Fox McMaster University The Social Organization of

435 views • 18 slides