Neural Ordinary Differential Equations

Published in

Neural Information Processing Systems(2018)

External Links:

Generate Graph

TL;DR

This work shows how to scalably backpropagate through any ODE solver, without access to its internal operations, which allows end-to-end training of ODEs within larger models.

Abstract

We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably backpropagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.

Authors

Yulia Rubanova

4 papers

D. Duvenaud

7 papers

T. Chen

2 papers

References64 items

CasADi: a software framework for nonlinear optimization and optimal control

Deep Neural Networks Motivated by Partial Differential Equations

Sylvester Normalizing Flows for Variational Inference

Optimization and uncertainty analysis of ODE models using 2nd order adjoint sensitivity analysis

Latent Space Physics: Towards Learning the Temporal Evolution of Fluid Flow

Neural Ordinary Differential Equations

Published in

Neural Information Processing Systems(2018)

External Links:

Generate Graph

TL;DR

This work shows how to scalably backpropagate through any ODE solver, without access to its internal operations, which allows end-to-end training of ODEs within larger models.

Abstract

Authors

Yulia Rubanova

4 papers

D. Duvenaud

7 papers

T. Chen

2 papers

References64 items

CasADi: a software framework for nonlinear optimization and optimal control

Deep Neural Networks Motivated by Partial Differential Equations

Sylvester Normalizing Flows for Variational Inference

Optimization and uncertainty analysis of ODE models using 2nd order adjoint sensitivity analysis

Latent Space Physics: Towards Learning the Temporal Evolution of Fluid Flow

J. Bettencourt

1 papers

Black-Box Variational Inference for Stochastic Differential Equations

Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations

Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems

Automatic differentiation in PyTorch

Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations

Multi-level Residual Networks from Dynamical Systems View

PDE-Net: Learning PDEs from Data

Reversible Architectures for Arbitrarily Deep Residual Neural Networks

Scalable Joint Models for Reliable Uncertainty-Aware Event Prediction

Hidden physics models: Machine learning of nonlinear partial differential equations

Time-Dependent Representation for Neural Event Sequence Prediction

The Reversible Residual Network: Backpropagation Without Storing Activations

Learning to Detect Sepsis with a Multitask Gaussian Process RNN Classifier

Stable architectures for deep neural networks

What-If Reasoning with Counterfactual Gaussian Processes

Reliable Decision Support using Counterfactual Models

Numerical Gaussian Processes for Time-Dependent and Nonlinear Partial Differential Equations

OptNet: Differentiable Optimization as a Layer in Neural Networks

Spatially Adaptive Computation Time for Residual Networks

Improving Variational Auto-Encoders using Householder Flow

Variable Computation in Recurrent Neural Networks

Recurrent Marked Temporal Point Processes: Embedding Event History to Vector

Improved Variational Inference with Inverse Autoregressive Flow

Fast derivatives of likelihood functionals for ODE based models using adjoint-state method

Directly Modeling Missing Data in Sequences with RNNs: Improved Classification of Clinical Time Series

Recurrent Neural Networks for Multivariate Time Series with Missing Values

Adaptive Computation Time for Recurrent Neural Networks

Identity Mappings in Deep Residual Networks

Deep Residual Learning for Image Recognition

Doctor AI: Predicting Clinical Events via Recurrent Neural Networks

The Stan Math Library: Reverse-Mode Automatic Differentiation in C++

Variational Inference with Normalizing Flows

Automatic differentiation in machine learning: a survey

Adam: A Method for Stochastic Optimization

NICE: Non-linear Independent Components Estimation

FATODE: A Library for Forward, Adjoint, and Tangent Linear Integration of ODEs

Probabilistic ODE Solvers with Runge-Kutta Means

Stochastic Backpropagation and Approximate Inference in Deep Generative Models

Auto-Encoding Variational Bayes

A General-Purpose Software Framework for Dynamic Optimization (Een algemene softwareomgeving voor dynamische optimalisatie)

Automated Derivation of the Adjoint of High-Level Transient Finite Element Programs

Computationally Efficient Convolved Multiple Output Gaussian Processes

Solving Ordinary Differential Equations I: Nonstiff Problems

Gradient calculations for dynamic recurrent neural networks: a survey

Solving ordinary differential equations I (2nd revised. ed.): nonstiff problems

The Mathematical Theory of Optimal Processes

Mathematical Theory of Optimal Processes

Ueber die numerische Auflösung von Differentialgleichungen

The Neural Hawkes Process: A Neurally Self-Modulating Multivariate Point Process

Treatment-response models for counterfactual reasoning with continuous-time, continuous-valued interventions

Theory Of Ordinary Differential Equations

Autograd: Reverse-mode differentiation of native Python

Neural networks for machine learning lecture 6a overview of mini-batch gradient descent

Gradient-based learning applied to document recognition

A Theoretical Framework for Back-Propagation

Intensitätsschwankungen im fernsprechverker. Ericsson Technics

Beitrag zur Naherungsweisen Integration Totaler Differentialgleichungen

Field of Study

Computer ScienceMathematics

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

Neural Information Processing Systems

Type

conference

URL

http://neurips.cc/

Alternate Names

Neural Inf Process Syst
NeurIPS
NIPS

TL;DR

Abstract

Authors

References64 items

CasADi: a software framework for nonlinear optimization and optimal control

Deep Neural Networks Motivated by Partial Differential Equations

Sylvester Normalizing Flows for Variational Inference

Optimization and uncertainty analysis of ODE models using 2nd order adjoint sensitivity analysis

Latent Space Physics: Towards Learning the Temporal Evolution of Fluid Flow

TL;DR

Abstract

Authors

References64 items

CasADi: a software framework for nonlinear optimization and optimal control

Deep Neural Networks Motivated by Partial Differential Equations

Sylvester Normalizing Flows for Variational Inference

Optimization and uncertainty analysis of ODE models using 2nd order adjoint sensitivity analysis

Latent Space Physics: Towards Learning the Temporal Evolution of Fluid Flow

Black-Box Variational Inference for Stochastic Differential Equations

Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations

Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems

Automatic differentiation in PyTorch

Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations

Multi-level Residual Networks from Dynamical Systems View

PDE-Net: Learning PDEs from Data

Reversible Architectures for Arbitrarily Deep Residual Neural Networks

Scalable Joint Models for Reliable Uncertainty-Aware Event Prediction

Hidden physics models: Machine learning of nonlinear partial differential equations

Time-Dependent Representation for Neural Event Sequence Prediction

The Reversible Residual Network: Backpropagation Without Storing Activations

Learning to Detect Sepsis with a Multitask Gaussian Process RNN Classifier

Stable architectures for deep neural networks

What-If Reasoning with Counterfactual Gaussian Processes

Reliable Decision Support using Counterfactual Models

Numerical Gaussian Processes for Time-Dependent and Nonlinear Partial Differential Equations

OptNet: Differentiable Optimization as a Layer in Neural Networks

Spatially Adaptive Computation Time for Residual Networks

Improving Variational Auto-Encoders using Householder Flow

Variable Computation in Recurrent Neural Networks

HyperNetworks

Recurrent Marked Temporal Point Processes: Embedding Event History to Vector

Improved Variational Inference with Inverse Autoregressive Flow

Fast derivatives of likelihood functionals for ODE based models using adjoint-state method

Directly Modeling Missing Data in Sequences with RNNs: Improved Classification of Clinical Time Series

Recurrent Neural Networks for Multivariate Time Series with Missing Values

Adaptive Computation Time for Recurrent Neural Networks

Identity Mappings in Deep Residual Networks

Deep Residual Learning for Image Recognition

Doctor AI: Predicting Clinical Events via Recurrent Neural Networks

The Stan Math Library: Reverse-Mode Automatic Differentiation in C++

Variational Inference with Normalizing Flows

Automatic differentiation in machine learning: a survey

Adam: A Method for Stochastic Optimization

NICE: Non-linear Independent Components Estimation

FATODE: A Library for Forward, Adjoint, and Tangent Linear Integration of ODEs

Probabilistic ODE Solvers with Runge-Kutta Means

Stochastic Backpropagation and Approximate Inference in Deep Generative Models

Auto-Encoding Variational Bayes

A General-Purpose Software Framework for Dynamic Optimization (Een algemene softwareomgeving voor dynamische optimalisatie)

Automated Derivation of the Adjoint of High-Level Transient Finite Element Programs

Computationally Efficient Convolved Multiple Output Gaussian Processes

Solving Ordinary Differential Equations I: Nonstiff Problems

Stable fluids

Gradient calculations for dynamic recurrent neural networks: a survey

Solving ordinary differential equations I (2nd revised. ed.): nonstiff problems

The Mathematical Theory of Optimal Processes

Mathematical Theory of Optimal Processes

Ueber die numerische Auflösung von Differentialgleichungen

The Neural Hawkes Process: A Neurally Self-Modulating Multivariate Point Process

Treatment-response models for counterfactual reasoning with continuous-time, continuous-valued interventions

Theory Of Ordinary Differential Equations

Autograd: Reverse-mode differentiation of native Python

Neural networks for machine learning lecture 6a overview of mini-batch gradient descent

Gradient-based learning applied to document recognition

A Theoretical Framework for Back-Propagation

Intensitätsschwankungen im fernsprechverker. Ericsson Technics

Beitrag zur Naherungsweisen Integration Totaler Differentialgleichungen

Field of Study

Journal Information

Name