An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018-03-04T00:00:00.000000Z)

TL;DR

A systematic evaluation of generic convolutional and recurrent architectures for sequence modeling concludes that the common association between sequence modeling and recurrent networks should be reconsidered, and convolutionals should be regarded as a natural starting point for sequence modeled tasks.

Abstract

For most deep learning practitioners, sequence modeling is synonymous with recurrent networks. Yet recent results indicate that convolutional architectures can outperform recurrent networks on tasks such as audio synthesis and machine translation. Given a new sequence modeling task or dataset, which architecture should one use? We conduct a systematic evaluation of generic convolutional and recurrent architectures for sequence modeling. The models are evaluated across a broad range of standard tasks that are commonly used to benchmark recurrent networks. Our results indicate that a simple convolutional architecture outperforms canonical recurrent networks such as LSTMs across a diverse range of tasks and datasets, while demonstrating longer effective memory. We conclude that the common association between sequence modeling and recurrent networks should be reconsidered, and convolutional networks should be regarded as a natural starting point for sequence modeling tasks. To assist related work, we have made code available at this http URL .

Authors

J. Z. Kolter

11 papers

V. Koltun

29 papers

Shaojie Bai

2 papers

TL;DR

Abstract

Authors

References84 items

Breaking the Softmax Bottleneck: A High-Rank RNN Language Model

Dilated Recurrent Neural Networks

Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks

Regularizing and Optimizing LSTM Language Models

On the State of the Art of Evaluation in Neural Language Models

Deep Pyramid Convolutional Neural Networks for Text Categorization

Convolutional Sequence to Sequence Learning

Diagonal rnns in symbolic music modeling

Comparative Study of CNN and RNN for Natural Language Processing

Language Modeling with Gated Convolutional Networks

Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs

Temporal Convolutional Networks for Action Segmentation and Detection

A Convolutional Encoder Model for Neural Machine Translation

Quasi-Recurrent Neural Networks

Improving Neural Language Models with a Continuous Cache

Full-Capacity Unitary Recurrent Neural Networks

Neural Machine Translation in Linear Time

HyperNetworks

Pointer Sentinel Mixture Models

WaveNet: A Generative Model for Raw Audio

Hierarchical Multiscale Recurrent Neural Networks

Using the Output Embedding to Improve Language Models

Layer Normalization

On Multiplicative Integration with Recurrent Neural Networks

The LAMBADA dataset: Word prediction requiring a broad discourse context

Very Deep Convolutional Networks for Text Classification

Gated Word-Character Recurrent Language Model

Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations

Recurrent Batch Normalization

Architectural Complexity Measures of Recurrent Neural Networks

Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks

Deep Residual Learning for Image Recognition

Regularizing RNNs by Stabilizing Activations

Multi-Scale Context Aggregation by Dilated Convolutions

Unitary Evolution Recurrent Neural Networks

Modeling temporal dependencies in data using a DBN-LSTM

Character-level Convolutional Networks for Text Classification

An Empirical Exploration of Recurrent Network Architectures

Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting

A Simple Way to Initialize Recurrent Networks of Rectified Linear Units

LSTM: A Search Space Odyssey

Adam: A Method for Stochastic Optimization

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

Effective Use of Word Order for Text Categorization with Convolutional Neural Networks

Fully convolutional networks for semantic segmentation

Sequence to Sequence Learning with Neural Networks

On the Properties of Neural Machine Translation: Encoder–Decoder Approaches

Neural Machine Translation by Jointly Learning to Align and Translate

Convolutional Neural Networks for Sentence Classification

Learning Character-level Representations for Part-of-Speech Tagging

A Convolutional Neural Network for Modelling Sentences

A Clockwork RNN

How to Construct Deep Recurrent Neural Networks

Generating Sequences With Recurrent Neural Networks

On the difficulty of training recurrent neural networks

Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription

Supervised Sequence Labelling with Recurrent Neural Networks

Learning Recurrent Neural Networks with Hessian-Free Optimization

Generating Text with Recurrent Neural Networks

Natural Language Processing (Almost) from Scratch

Rectified Linear Units Improve Restricted Boltzmann Machines

A unified architecture for natural language processing: deep neural networks with multitask learning

Harmonising Chorales by Probabilistic Inference

Learning Precise Timing with LSTM Recurrent Networks

Bidirectional recurrent neural networks

Long Short-Term Memory

Hierarchical Recurrent Neural Networks for Long-Term Dependencies

Learning long-term dependencies with gradient descent is difficult

Building a Large Annotated Corpus of English: The Penn Treebank

Backpropagation Through Time: What It Does and How to Do It

Speaker-independent isolated digit recognition: Multilayer perceptrons vs. Dynamic time warping

Finding Structure in Time

Backpropagation Applied to Handwritten Zip Code Recognition

Connectionist Learning Procedures

Phoneme recognition using time-delay neural networks

Sequence Models (Course 5 of Deep Learning Specialization)