Liquid Time-constant Networks (2020-06-08T00:00:00.000000Z)

TL;DR

A new class of time-continuous recurrent neural network models that exhibit stable and bounded behavior, yield superior expressivity within the family of neural ordinary differential equations, and give rise to improved performance on time-series prediction tasks are introduced.

Abstract

We introduce a new class of time-continuous recurrent neural network models. Instead of declaring a learning system's dynamics by implicit nonlinearities, we construct networks of linear first-order dynamical systems modulated via nonlinear interlinked gates. The resulting models represent dynamical systems with varying (i.e., liquid) time-constants coupled to their hidden state, with outputs being computed by numerical differential equation solvers. These neural networks exhibit stable and bounded behavior, yield superior expressivity within the family of neural ordinary differential equations, and give rise to improved performance on time-series prediction tasks. To demonstrate these properties, we first take a theoretical approach to find bounds over their dynamics, and compute their expressive power by the trajectory length measure in a latent trajectory space. We then conduct a series of time-series prediction experiments to manifest the approximation capability of Liquid Time-Constant Networks (LTCs) compared to classical and modern RNNs.

Authors

R. Grosu

3 papers

Ramin M. Hasani

2 papers

Mathias Lechner

2 papers

TL;DR

Abstract

Authors

References76 items

Neural circuit policies enabling auditable autonomy

Accurate Occupancy Detection of an Office Room From Light, Temperature, Humidity and CO2 Measurements Using Statistical Learning Models

Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE

Learning Long-Term Dependencies in Irregularly-Sampled Time Series

Gershgorin Loss Stabilizes the Recurrent Neural Network Compartment of an End-to-end Robot Learning Scheme

Learning to Control PDEs with Differentiable Physics

Causality for Machine Learning

On Robustness of Neural Ordinary Differential Equations

Latent ODEs for Irregularly-Sampled Time Series

Towards Robust, Locally Linear Deep Networks

SNODE: Spectral Discretization of Neural ODEs for System Identification

Neural Spline Flows

Neural Jump Stochastic Differential Equations

Designing Worm-inspired Neural Networks for Interpretable Robotic Control

Augmented Neural ODEs

ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs

Complexity of Linear Regions in Deep Networks

Response Characterization for Auditing Cell Dynamics in Long Short-term Memory Networks

Re-purposing Compact Neuronal Circuit Policies to Govern Reinforcement Learning Tasks

c302: a multiscale framework for modelling the nervous system of Caenorhabditis elegans

OpenWorm: overview and recent advances in integrative biological simulation of Caenorhabditis elegans

Plug-and-play supervisory control using muscle and brain signals for real-time gesture and error detection

Neural Ordinary Differential Equations

Entropy and mutual information in models of deep neural networks

How to Start Training: The Effect of Initialization and Architecture

Bounding and Counting Linear Regions of Deep Neural Networks

Discrete Event, Continuous Time RNNs

Non-Associative Learning Representation in the Nervous System of the Nematode Caenorhabditis elegans

On the Expressive Power of Deep Neural Networks

Exponential expressivity in deep neural networks through transient chaos

Recurrent Neural Networks for Multivariate Time Series with Missing Values

OpenAI Gym

Accurate occupancy detection of an office room from light, temperature, humidity and CO2 measurements using statistical learning models

The Power of Depth for Feedforward Neural Networks

Adam: A Method for Stochastic Optimization

Gesture Unit Segmentation Using Spatial-Temporal Information and Machine Learning

On the Folded Normal Distribution

On the Number of Linear Regions of Deep Neural Networks

On the number of response regions of deep feed forward networks with piece-wise linear activations

MuJoCo: A physics engine for model-based control

On the difficulty of training recurrent neural networks

Forecasting skewed biased stochastic ozone days: analyses, solutions and beyond

Numerical Recipes 3rd Edition: The Art of Scientific Computing

Recurrent Neural Networks Are Universal Approximators

Bilinear dynamical systems

Dynamic causal modelling

Long Short-Term Memory

A Dynamic Network Simulation of the Nematode Tap Withdrawal Circuit: Predictions Concerning Synaptic Function Using Behavioral Criteria

Analysis and implementation of TR-BDF2

Original Contribution: Approximation of dynamical systems by continuous time recurrent neural networks

Approximation by superpositions of a sigmoidal function

Multilayer feedforward networks are universal approximators

On the approximate realization of continuous mappings by neural networks

Learning representations by back-propagating errors

Computer Solution of Ordinary Differential Equations.

A family of embedded Runge-Kutta formulae

Computer Solution of Ordinary Differential Equations

Differential Equations, Dynamical Systems, and Linear Algebra

The Mathematical Theory of Optimal Processes

A Natural Lottery Ticket Winner: Reinforcement Learning with Ordinary Neural Circuits

Interpretable Recurrent Neural Networks in Continuous-time Control Environments

UCI machine learning repository

Norm of Projections (Raghu et al. 2017). for a W k×k with conditions of Lemma 8, and two vectors, x and y, then the following holds for x⊥

Independence of Projections (Raghu et al. 2017). Given a matrix W with iid entries drawn form N (0, σ2), then its decomposition matrices W⊥ and W‖ in respect to x, are independent random variables

Objective of this task is to predict the hourly active power consumption of a household

ForX being a zero-mean multivariate Gaussian and having a diagonal covariance matrix, and μ a vector of constants, we have: E[‖X − μ‖] ≥ E[‖X‖

Methods In Neuronal Modeling From Ions To Networks

μ and σ are the mean and variance of the normal distribution over random variable x, and Φ is a normal cumulative distribution function. For a zero-mean normal distribution with variance of σ2

A Public Domain Dataset for Human Activity Recognition using Smartphones

Half-Cheetah Kinematic modeling This task is inspired by the physics simulation experiment

A 3(2) pair of Runge - Kutta formulas

Computer solution of ordinary differential equations : the initial value problem

Hirsch and Stephen Smale

Chapter 8, section 5, Theorem). Lemma 4. For an F : R → R which is a bound C1-mapping, the differential equation

Recherches quantitatives sur l’excitation electrique des nerfs traitee comme une polarization

for dataset