Residual Shuffle-Exchange Networks for Fast Processing of Long Sequences (2020-04-06T00:00:00.000000Z)

TL;DR

This paper presents a simple and lightweight variant of the Shuffle-Exchange network, which is based on a residual network employing GELU and Layer Normalization and achieves state-of-the-art performance on the MusicNet dataset for music transcription while being efficient in the number of parameters.

Abstract

Attention is a commonly used mechanism in sequence processing, but it is of O(n^2) complexity which prevents its application to long sequences. The recently introduced neural Shuffle-Exchange network offers a computation-efficient alternative, enabling the modelling of long-range dependencies in O(n log n) time. The model, however, is quite complex, involving a sophisticated gating mechanism derived from the Gated Recurrent Unit. In this paper, we present a simple and lightweight variant of the Shuffle-Exchange network, which is based on a residual network employing GELU and Layer Normalization. The proposed architecture not only scales to longer sequences but also converges faster and provides better accuracy. It surpasses the Shuffle-Exchange network on the LAMBADA language modelling task and achieves state-of-the-art performance on the MusicNet dataset for music transcription while being efficient in the number of parameters. We show how to combine the improved Shuffle-Exchange network with convolutional layers, establishing it as a useful building block in long sequence processing applications.

Authors

Andis Draguns

1 papers

Emils Ozolins

1 papers

A. Sostaks

1 papers

TL;DR

Abstract

Authors

References50 items

Big Bird: Transformers for Longer Sequences

Linformer: Self-Attention with Linear Complexity

Language Models are Few-Shot Learners

Longformer: The Long-Document Transformer

Reformer: The Efficient Transformer

Understanding and Improving Layer Normalization

Complex Transformer: A Framework for Modeling Complex-Valued Sequence

On the Variance of the Adaptive Learning Rate and Beyond

Generating Long Sequences with Sparse Transformers

Star-Transformer

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Automatic Music Transcription: An Overview

Music Transformer: Generating Music with Long-Term Structure

Character-Level Language Modeling with Deeper Self-Attention

Universal Transformers

Complex Gated Recurrent Neural Networks

Recent Advances in Neural Program Synthesis

Advances in Pre-Training Distributed Word Representations

Invariances and Data Augmentation for Supervised Music Transcription

Simple and Effective Multi-Paragraph Reading Comprehension

Attention is All you Need

Deep Complex Networks

Convolutional Sequence to Sequence Learning

Improving the Neural GPU Architecture for Algorithm Learning

Learning Features of Music from Scratch

Can Active Memory Replace Attention?

Broad Context Language Modeling as Reading Comprehension

Hybrid computing using a neural network with dynamic external memory

WaveNet: A Generative Model for Raw Audio

Layer Normalization

Gaussian Error Linear Units (GELUs)

The LAMBADA dataset: Word prediction requiring a broad discourse context

Neural GPUs Learn Algorithms

Multi-Scale Context Aggregation by Dilated Convolutions

Learning Simple Algorithms from Examples

Grid Long Short-Term Memory

Pointer Networks

Learning to Transduce with Unbounded Memory

Reinforcement Learning Neural Turing Machines - Revised

Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets

Neural Turing Machines

Neural Machine Translation by Jointly Learning to Align and Translate

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Scikit-learn: Machine Learning in Python

Long Short-Term Memory

Neural Shufﬂe-Exchange Networks − Sequence Processing in O( n log n ) Time

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

Hinton Residual Shuffle-Exchange Networks

Principles and Practices of Interconnection Networks

Field of Study

Venue Information

Name

Type

URL

Alternate Names