Efficient Attention: Attention with Linear Complexities (2018-12-04T00:00:00.000000Z)

TL;DR

A novel efficient attention mechanism equivalent to dot-product attention but with substantially less memory and computational costs is proposed, which allows more widespread and flexible integration of attention modules into a network, which leads to better accuracies.

Abstract

Dot-product attention has wide applications in computer vision and natural language processing. However, its memory and computational costs grow quadratically with the input size. Such growth prohibits its application on high- resolution inputs. To remedy this drawback, this paper proposes a novel efficient attention mechanism equivalent to dot-product attention but with substantially less memory and computational costs. Its resource efficiency allows more widespread and flexible integration of attention modules into a network, which leads to better accuracies. Empirical evaluations demonstrated the effectiveness of its advantages. Efficient attention modules brought significant performance boosts to object detectors and instance segmenters on MS-COCO 2017. Further, the resource efficiency democratizes attention to complex models, where high costs prohibit the use of dot-product attention. As an exemplar, a model with efficient attention achieved state-of- the-art accuracies for stereo depth estimation on the Scene Flow dataset. Code is available at https://github.com/cmsflash/efficient-attention.

Authors

Hongsheng Li

26 papers

Mingyuan Zhang

6 papers

Haiyu Zhao

7 papers

TL;DR

Abstract

Authors

References48 items

Hierarchical Neural Architecture Search for Deep Stereo Matching

Cross Attention Network for Few-shot Classification

SILCO: Show a Few Images, Localize the Common Object

XLNet: Generalized Autoregressive Pretraining for Language Understanding

LatentGNN: Learning Efficient Non-local Relations for Visual Recognition

Very Deep Self-Attention Networks for End-to-End Speech Recognition

GCNet: Non-Local Networks Meet Squeeze-Excitation Networks and Beyond

Compact Generalized Non-local Network

Learning Depth with Convolutional Spatial Propagation Network

Large Scale GAN Training for High Fidelity Natural Image Synthesis

Non-locally Enhanced Encoder-Decoder Network for Single Image De-raining

CBAM: Convolutional Block Attention Module

Video-based Person Re-identification via 3D Convolutional Networks and Non-local Attention

Non-Local Recurrent Network for Image Restoration

Self-Attention Generative Adversarial Networks

ConnNet: A Long-Range Relation-Aware Pixel-Connectivity Network for Salient Segmentation

Speech-Transformer: A No-Recurrence Sequence-to-Sequence Model for Speech Recognition

YOLOv3: An Incremental Improvement

Pyramid Stereo Matching Network

EdgeStereo: A Context Integrated Residual Pyramid Network for Stereo Matching

Cascade R-CNN: Delving Into High Quality Object Detection

Non-local Neural Networks

MegDet: A Large Mini-Batch Object Detector

Light-Head R-CNN: In Defense of Two-Stage Object Detector

Improving Object Localization with Fitness NMS and Bounded IoU Loss

Squeeze-and-Excitation Networks

Cascade Residual Learning: A Two-Stage Convolutional Neural Network for Stereo Matching

Focal Loss for Dense Object Detection

Attention is All you Need

Soft-NMS — Improving Object Detection with One Line of Code

R-C3D: Region Convolutional 3D Network for Temporal Activity Detection

Mask R-CNN

Deformable Convolutional Networks

YOLO9000: Better, Faster, Stronger

Beyond Skip Connections: Top-Down Modulation for Object Detection

Feature Pyramid Networks for Object Detection

R-FCN: Object Detection via Region-based Fully Convolutional Networks

Deep Residual Learning for Image Recognition

SSD: Single Shot MultiBox Detector

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Fast R-CNN

Neural Machine Translation by Jointly Learning to Align and Translate

Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

Improving Language Understanding by Generative Pre-Training

THUMOS challenge: Action recognition with a large number of classes

possesses the same representational power as the prevalent dot-product attention mechanism

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names