SLIDE 1
Neural Tangent Kernel
Convergence and Generalization in Neural Networks Arthur Jacot
arthur.jacot@epfl.ch
Franck Gabriel
franck.gabriel@epfl.ch
Clément Hongler
clement.hongler@epfl.ch
Neural Tangent Kernel Convergence and Generalization in Neural - - PowerPoint PPT Presentation
Neural Tangent Kernel Convergence and Generalization in Neural - - PowerPoint PPT Presentation
Neural Tangent Kernel Convergence and Generalization in Neural Networks Arthur Jacot Franck Gabriel Clment Hongler arthur.jacot@epfl.ch franck.gabriel@epfl.ch clement.hongler@epfl.ch What happens during training? One step of Gradient
SLIDE 2
SLIDE 3
In the Infinite width limit:
- Deterministic
- Fixed in time
- Explicit formula
Determines the trajectory of the network function during training
SLIDE 4
Kernel methods
Kernel Gradient Descent Positive definite NTK Kernel ridge regression
Neural Networks
Gradient Descent Convergence to a global min. Least-squares loss
SLIDE 5
The sum of all microscopic changes yields a macroscopic effect What happens inside a very wide network?
- The activations of the hidden neurons become independent
- The parameters and activations evolve less and less
- However all layers learn: