Learning Scalable Deep Kernels with Recurrent Structure (2016-10-27T00:00:00.000000Z)

TL;DR

The resulting model, GP-LSTM, fully encapsulates the inductive biases of long short-term memory (L STM) recurrent networks, while retaining the non-parametric probabilistic advantages of Gaussian processes.

Abstract

Many applications in speech, robotics, finance, and biology deal with sequential data, where ordering matters and recurrent structures are common. However, this structure cannot be easily captured by standard kernel functions. To model such structure, we propose expressive closed-form kernel functions for Gaussian processes. The resulting model, GP-LSTM, fully encapsulates the inductive biases of long short-term memory (LSTM) recurrent networks, while retaining the non-parametric probabilistic advantages of Gaussian processes. We learn the properties of the proposed kernels by optimizing the Gaussian process marginal likelihood using a new provably convergent semi-stochastic gradient procedure, and exploit the structure of these kernels for scalable training and prediction. This approach provides a practical representation for Bayesian LSTMs. We demonstrate state-of-the-art performance on several benchmarks, and thoroughly investigate a consequential autonomous driving application, where the predictive uncertainties provided by GP-LSTM are uniquely valuable.

Authors

E. Xing

15 papers

A. Wilson

10 papers

Zhiting Hu

5 papers

TL;DR

Abstract

Authors

References58 items

Interpolation of Spatial Data

Bayesian GAN

Stochastic Variational Deep Kernel Learning

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

Recurrent Gaussian Processes

Deep Kernel Learning

Thoughts on Massively Scalable Gaussian Processes

Train faster, generalize better: Stability of stochastic gradient descent

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)

Distributed Gaussian Processes

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Sequence to Sequence Learning with Neural Networks

Block Stochastic Gradient Iteration for Convex and Nonconvex Optimization

Variational Gaussian Process State-Space Models

Manifold Gaussian Processes for regression

Stochastic Gradient Hamiltonian Monte Carlo

How to Construct Deep Recurrent Neural Networks

Gaussian Processes for Big Data

Speech recognition with deep recurrent neural networks

Gaussian Process Kernels for Pattern Discovery and Extrapolation

Bayesian Warped Gaussian Processes

Deep Gaussian Processes

Subspace Identification for Linear Systems: Theory ― Implementation ― Applications

Distributed delayed stochastic optimization

State-Space Inference and Learning with Gaussian Processes

A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

Copula Processes

Slow Learners are Fast

Using Deep Belief Nets to Learn Covariance Kernels for Gaussian Processes

Universal Kernels

Gaussian Process Dynamical Models

Dynamic systems identification with Gaussian processes

A Unifying View of Sparse Approximate Gaussian Process Regression

Gaussian Processes For Machine Learning

Warped Gaussian Processes

The general inefficiency of batch training for gradient descent learning

Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond

Exploiting Generative Models in Discriminative Classifiers

Long Short-Term Memory

Learning long-term dependencies in NARX recurrent neural networks

Nonlinear black-box modeling in system identification: a unified overview

Learning long-term dependencies with gradient descent is difficult

Occam's Razor

Cubic convolution interpolation for digital image processing

Occam's razor

Time series analysis, forecasting and control

2015] consider a stochastic program that minimizes the expectation of the objective w.r.t. some distribution underlying the data

2015] note that standard inducing point approaches, such as subset of regression (SoR) or fully independent

Convergence results Convergence results for the semi-stochastic alternating gradient schemes with and without delayed kernel matrix updates are based on

propose to loop over a number of coordinate blocks and apply Gauss–Seidel type gradient updates to each block

The optimization procedure

For any given kernel function, K, and a set of inducing points, U , structured kernel interpolation (SKI) procedure [Wilson and Nickisch, 2015] gives rise to the following approximate kernel

explored 5 different approximation methods known in the numerical analysis literature

Dropout: a simple way to prevent neural networks from overfitting

Nonlinear Programming

Input-output data sets for development and benchmarking in nonlinear identification

Introduction to Gaussian Processes

Field of Study

Journal Information

Name

Page

Volume

Venue Information

Name

Type

URL

Alternate Names