Discretize-Optimize vs. Optimize-Discretize for Time-Series Regression and Continuous Normalizing Flows (2020-05-27T00:00:00.000000Z)

TL;DR

Disc-Opt methods can achieve similar performance as Opt-Disc at inference with drastically reduced training costs using neural ODEs for time-series regression and continuous normalizing flows (CNFs).

Abstract

We compare the discretize-optimize (Disc-Opt) and optimize-discretize (Opt-Disc) approaches for time-series regression and continuous normalizing flows (CNFs) using neural ODEs. Neural ODEs are ordinary differential equations (ODEs) with neural network components. Training a neural ODE is an optimal control problem where the weights are the controls and the hidden features are the states. Every training iteration involves solving an ODE forward and another backward in time, which can require large amounts of computation, time, and memory. Comparing the Opt-Disc and Disc-Opt approaches in image classification tasks, Gholami et al. (2019) suggest that Disc-Opt is preferable due to the guaranteed accuracy of gradients. In this paper, we extend the comparison to neural ODEs for time-series regression and CNFs. Unlike in classification, meaningful models in these tasks must also satisfy additional requirements beyond accurate final-time output, e.g., the invertibility of the CNF. Through our numerical experiments, we demonstrate that with careful numerical treatment, Disc-Opt methods can achieve similar performance as Opt-Disc at inference with drastically reduced training costs. Disc-Opt reduced costs in six out of seven separate problems with training time reduction ranging from 39% to 97%, and in one case, Disc-Opt reduced training from nine days to less than one day.

Authors

Lars Ruthotto

2 papers

Derek Onken

1 papers

TL;DR

Abstract

Authors

References45 items

How to train your neural ODE

Universal Differential Equations for Scientific Machine Learning

Discrete Processes and their Continuous Limits

Equivariant Flows: sampling configurations for multi-body systems with symmetric energies

Machine Learning for Fluid Mechanics

Deep learning as optimal control problems: models and numerical methods

ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs

DiffEqFlux.jl - A Julia Library for Neural Differential Equations

Layer-Parallel Training of Deep Residual Neural Networks

FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models

Numerical Optimization

Neural Ordinary Differential Equations

Deep Neural Networks Motivated by Partial Differential Equations

Convolutional neural networks combined with Runge–Kutta methods

Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations

Image Segmentation with Pyramid Dilated Convolution Based on ResNet and U-Net

Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations

Maximum Principle Based Algorithms for Deep Learning

Reversible Architectures for Arbitrarily Deep Residual Neural Networks

Masked Autoregressive Flow for Density Estimation

Stable architectures for deep neural networks

A Proposal on Machine Learning via Dynamical Systems

Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring

Transport Analysis of Infinitely Deep Neural Network

Deep Residual Learning for Image Recognition

Variational Inference with Normalizing Flows

Adam: A Method for Stochastic Optimization

Deep AutoRegressive Networks

A First Course in Numerical Methods

Understanding the difficulty of training deep feedforward neural networks

Identification of distributed parameter systems: A neural net based approach

DISCRETE- vs. CONTINUOUS-TIME NONLINEAR SIGNAL PROCESSING OF Cu ELECTRODISSOLUTION DATA

A Simple Weight Decay Can Improve Generalization

A Stochastic Approximation Method

AUTO-ENCODING VARIATIONAL BAYES

Student-Athlete Advisory Committee Team Representative

Awarded Peter O'Sullivan Hardest Worker Award, UGA Men's Swimming

UGA Men's Swimming 2014 -College Swim Coaches of America Association Scholar All-American Team

Qualified and competed at the 2016 U.S. Olympic Trials -NCAA Academic All-American Honorable Mention

University of Georgia Presidential Scholar 2014 University of Georgia Athletic Director's Honor Roll

Chartered Property Casualty Underwriter (CPCU) Scholarship Recipient

A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines

Perspectives in flow control and optimization

The use of adjoint systems in the problem of differential corrections for trajectories

Disc-Opt on B SDS 300

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names