Recurrent Highway Networks With Grouped Auxiliary Memory (2019-12-13T00:00:00.000000Z)

TL;DR

This paper proposes a novel RNN architecture based on Recurrent Highway Network with Grouped Auxiliary Memory (GAM-RHN), which interconnects the RHN with a set of auxiliary memory units specifically for storing long-term information via reading and writing operations, analogous to Memory Augmented Neural Networks (MANNs).

Abstract

Recurrent neural networks (RNNs) are challenging to train, let alone those with deep spatial structures. Architectures built upon highway connections such as Recurrent Highway Network (RHN) were developed to allow larger step-to-step transition depth, leading to more expressive models. However, problems that require capturing long-term dependencies still cannot be well addressed by these models. Moreover, the ability to keep long-term memories tends to diminish when the spatial depth increases, since deeper structure may accelerate gradient vanishing. In this paper, we address these issues by proposing a novel RNN architecture based on RHN, namely the Recurrent Highway Network with Grouped Auxiliary Memory (GAM-RHN). The proposed architecture interconnects the RHN with a set of auxiliary memory units specifically for storing long-term information via reading and writing operations, which is analogous to Memory Augmented Neural Networks (MANNs). Experimental results on artificial long time lag tasks show that GAM-RHNs can be trained efficiently while being deep in both time and space. We also evaluate the proposed architecture on a variety of tasks, including language modeling, sequential image classification, and financial market forecasting. The potential of our approach is demonstrated by achieving state-of-the-art results on these tasks.

Authors

Wei Luo

1 papers

Feng Yu

1 papers

TL;DR

Abstract

Authors

References62 items

T-LSTM: A Long Short-Term Memory Neural Network Enhanced by Temporal Information for Traffic Flow Prediction

Transformation-gated LSTM: efficient capture of short-term mutation dependencies for multivariate time series prediction tasks

Learning Longer-term Dependencies via Grouped Distributor Unit

Using Deep Learning for price prediction by exploiting stationary limit order book features

Improved Language Modeling by Decoding the Past

DeepLOB: Deep Convolutional Neural Networks for Limit Order Books

An Analysis of Neural Language Modeling at Multiple Scales

Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN

Learning Longer-term Dependencies in RNNs with Auxiliary Losses

Temporal Attention-Augmented Bilinear Network for Financial Time-Series Data Analysis

Dilated Recurrent Neural Networks

Dynamic Evaluation of Neural Sequence Models

Using deep learning to detect price change indications in financial markets

Forecasting Stock Prices from the Limit Order Book Using Convolutional Neural Networks

Fast-Slow Recurrent Neural Networks

Benchmark dataset for mid-price forecasting of limit order book data with machine learning methods

Capacity and Trainability in Recurrent Neural Networks

Full-Capacity Unitary Recurrent Neural Networks

Hybrid computing using a neural network with dynamic external memory

HyperNetworks

Multiplicative LSTM for sequence modelling

Hierarchical Multiscale Recurrent Neural Networks

Recurrent Highway Networks

Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations

Minimal gated unit for recurrent neural networks

Recurrent Batch Normalization

Recurrent Dropout without Memory Loss

TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

Deep Residual Learning for Image Recognition

Unitary Evolution Recurrent Neural Networks

Highway long short-term memory RNNS for distant speech recognition

Grid Long Short-Term Memory

Highway Networks

A Simple Way to Initialize Recurrent Networks of Rectified Linear Units

Adam: A Method for Stochastic Optimization

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

Neural Turing Machines

Going deeper with convolutions

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

On the Complexity of Neural Network Classifiers: A Comparison Between Shallow and Deep Architectures

How to Construct Deep Recurrent Neural Networks

Generating Sequences With Recurrent Neural Networks

Speech recognition with deep recurrent neural networks

On the difficulty of training recurrent neural networks

Generating Text with Recurrent Neural Networks

Understanding the difficulty of training deep feedforward neural networks

Learning to Forget: Continual Prediction with LSTM

Recurrent nets that time and count

Long Short-Term Memory

Learning long-term dependencies with gradient descent is difficult

Building a Large Annotated Corpus of English: The Penn Treebank

On the computational power of neural nets

Learning Complex, Extended Sequences Using the Principle of History Compression

Learning representations by back-propagating errors

Character-Level Language Modeling with Recurrent Highway Hypernetworks

Dropout: a simple way to prevent neural networks from overfitting

Recurrent neural network based language model

Large Text Compression Benchmark

Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies

Gradient-based learning algorithms for recurrent networks and their computational complexity

Generalization of backpropagation with application to a recurrent gas market model

Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences

Field of Study

Journal Information

Name

Page

Volume

Venue Information

Name

Type

URL