Neural Tangent Kernel: Convergence and Generalization in Neural Networks (2018-06-20T00:00:00.000000Z)

TL;DR

It is proved that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of anANN, the network function follows the so-called kernel gradient associated with a new object, which is called the Neural Tangent Kernel (NTK).

Abstract

At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit, thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network function (which maps input vectors to output vectors) follows the so-called kernel gradient associated with a new object, which we call the Neural Tangent Kernel (NTK). This kernel is central to describe the generalization features of ANNs. While the NTK is random at initialization and varies during training, in the infinite-width limit it converges to an explicit limiting kernel and stays constant during training. This makes it possible to study the training of ANNs in function space instead of parameter space. Convergence of the training can then be related to the positive-definiteness of the limiting NTK. We then focus on the setting of least-squares regression and show that in the infinite-width limit, the network function follows a linear differential equation during training. The convergence is fastest along the largest kernel principal components of the input data with respect to the NTK, hence suggesting a theoretical motivation for early stopping. Finally we study the NTK numerically, observe its behavior for wide networks, and compare it to the infinite-width limit.

Authors

Arthur Jacot

1 papers

Franck Gabriel

1 papers

Clément Hongler

1 papers

Neural Tangent Kernel: Convergence and Generalization in Neural Networks

TL;DR

Abstract

Authors

References24 items

Universal statistics of Fisher information in deep neural networks: mean field approach

A mean field view of the landscape of two-layer neural networks

Gaussian Process Behaviour in Wide Deep Neural Networks

To understand deep learning we need to understand kernel learning

Deep Neural Networks as Gaussian Processes

Geometry of Neural Network Loss Surfaces via Random Matrix Theory

Empirical Analysis of the Hessian of Over-Parametrized Neural Networks

Understanding deep learning requires rethinking generalization

Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity

The Loss Surfaces of Multilayer Networks

Generative adversarial networks

Identifying and attacking the saddle point problem in high-dimensional non-convex optimization

On the saddle point problem for non-convex optimization

Strictly and non-strictly positive definite functions on spheres

Understanding the difficulty of training deep feedforward neural networks

Kernel Methods for Deep Learning

Random Features for Large-Scale Kernel Machines

Some Gronwall Type Inequalities and Applications

Nonlinear Component Analysis as a Kernel Eigenvalue Problem

Original Contribution: Multilayer feedforward networks with a nonpolynomial activation function can approximate any function

Multilayer feedforward networks are universal approximators

Sample-then-optimize posterior sampling for Bayesian linear models

Kernel Methods for Pattern Analysis

Bayesian learning for neural networks

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names