LSTM: A Search Space Odyssey (2015-03-13T00:00:00.000000Z)

TL;DR

This paper presents the first large-scale analysis of eight LSTM variants on three representative tasks: speech recognition, handwriting recognition, and polyphonic music modeling, and observes that the studied hyperparameters are virtually independent and derive guidelines for their efficient adjustment.

Abstract

Several variants of the long short-term memory (LSTM) architecture for recurrent neural networks have been proposed since its inception in 1995. In recent years, these networks have become the state-of-the-art models for a variety of machine learning problems. This has led to a renewed interest in understanding the role and utility of various computational components of typical LSTM variants. In this paper, we present the first large-scale analysis of eight LSTM variants on three representative tasks: speech recognition, handwriting recognition, and polyphonic music modeling. The hyperparameters of all LSTM variants for each task were optimized separately using random search, and their importance was assessed using the powerful functional ANalysis Of VAriance framework. In total, we summarize the results of 5400 experimental runs ( $\approx 15$ years of CPU time), which makes our study the largest of its kind on LSTM networks. Our results show that none of the variants can improve upon the standard LSTM architecture significantly, and demonstrate the forget gate and the output activation function to be its most critical components. We further observe that the studied hyperparameters are virtually independent and derive guidelines for their efficient adjustment.

Authors

J. Schmidhuber

5 papers

Klaus Greff

4 papers

R. Srivastava

2 papers

TL;DR

Abstract

Authors

References54 items

An Empirical Exploration of Recurrent Network Architectures

Protein Secondary Structure Prediction with Long Short Term Memory Networks

Fast and Robust Training of Recurrent Neural Networks for Offline Handwriting Recognition

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

Long-term recurrent convolutional networks for visual recognition and description

Addressing the Rare Word Problem in Neural Machine Translation

Dynamic Cortex Memory: Enhancing Recurrent Neural Networks for Gradient-Based Sequence Learning

Recurrent Neural Network Regularization

An Efficient Approach for Assessing Hyperparameter Importance

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Multi-resolution linear prediction based features for audio onset detection with bidirectional LSTM neural networks

Dropout Improves Recurrent Neural Networks for Handwriting Recognition

Generating Sequences With Recurrent Neural Networks

On the importance of initialization and momentum in deep learning

Speech recognition with deep recurrent neural networks

Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription

Practical Bayesian Optimization of Machine Learning Algorithms

Supervised Sequence Labelling with Recurrent Neural Networks

Random Search for Hyper-Parameter Optimization

Sequential Model-Based Optimization for General Algorithm Configuration

Evolving Memory Cell Structures for Sequence Learning

A Novel Connectionist System for Unconstrained Handwriting Recognition

Unconstrained On-line Handwriting Recognition with Recurrent Neural Networks

Generalized Functional ANOVA Diagnostics for High-Dimensional Functions of Dependent Variables

Training Recurrent Networks by Evolino

IAM-OnDB - an on-line English sentence database acquired from handwritten text on a whiteboard

2005 Special Issue: Framewise phoneme classification with bidirectional LSTM and other neural network architectures

Harmonising Chorales by Probabilistic Inference

Learning to Forget: Continual Prediction with LSTM

Recurrent nets that time and count

Long Short-Term Memory

A dictionary of linguistics and phonetics

Learning long-term dependencies with gradient descent is difficult

DARPA TIMIT:: acoustic-phonetic continuous speech corpus CD-ROM, NIST speech disc 1-1.1

Darpa Timit Acoustic-Phonetic Continuous Speech Corpus CD-ROM {TIMIT} | NIST

Minimization by Random Search Techniques

RECENT ADVANCES IN FINDING BEST OPERATING CONDITIONS

Long short-term memory recurrent neural network architectures for large scale acoustic modeling

TTS synthesis with bidirectional LSTM based recurrent neural networks

Unconstrained Online Handwriting Recognition with Recurrent Neural Networks

Dekf-lstm

ESANN 2002, Proceedings of the 10th Eurorean Symposium on Artificial Neural Networks

Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies

Heterogeneous acoustic measurements and multiple classifiers for speech recognition

Gers , Juan Antonio Pérez - Ortiz , Douglas Eck , and Jürgen Schmidhuber . DEFK - LSTM

Untersuchungen zu dynamischen neuronalen Netzen

Generalization of backpropagation with application to a recurrent gas market model

Distance measures for speech recognition, psychological and instrumental

Association

Connectionist Temporal Classiﬁcation: Labelling Unsegmented Sequence Data with Recurrent Neural Networks

References

Connectionist Temporal Classiﬁcation: Labelling Unsegmented Sequences with Recurrent Neural Networks

The Utility Driven Dynamic Error Propagation Network

and Machine Learning – ICANN 2014

Field of Study

Journal Information

Name

Page

Volume

Venue Information

Name

Type

URL

Alternate Names