Is Space-Time Attention All You Need for Video Understanding? (2021-02-09T00:00:00.000000Z)

TL;DR

This work presents a convolution-free approach to video classification built exclusively on self-attention over space and time, which adapts the standard Transformer architecture to video by enabling spatiotemporal feature learning directly from a sequence of frame-level patches.

Abstract

We present a convolution-free approach to video classification built exclusively on self-attention over space and time. Our method, named"TimeSformer,"adapts the standard Transformer architecture to video by enabling spatiotemporal feature learning directly from a sequence of frame-level patches. Our experimental study compares different self-attention schemes and suggests that"divided attention,"where temporal attention and spatial attention are separately applied within each block, leads to the best video classification accuracy among the design choices considered. Despite the radically new design, TimeSformer achieves state-of-the-art results on several action recognition benchmarks, including the best reported accuracy on Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks, our model is faster to train, it can achieve dramatically higher test efficiency (at a small drop in accuracy), and it can also be applied to much longer video clips (over one minute long). Code and models are available at: https://github.com/facebookresearch/TimeSformer.

Authors

L. Torresani

6 papers

Heng Wang

4 papers

Gedas Bertasius

2 papers

TL;DR

Abstract

Authors

References73 items

Training data-efficient image transformers & distillation through attention

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

AttentionNAS: Spatiotemporal Attention Cell Search for Video Classification

MotionSqueeze: Neural Motion Feature Learning for Video Understanding

End-to-End Object Detection with Transformers

Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training

Quantifying Attention Flow in Transformers

Exploring Self-Attention for Image Recognition

X3D: Expanding Architectures for Efficient Video Recognition

TEA: Temporal Excitation and Aggregation for Action Recognition

Actor-Transformers for Group Activity Recognition

RAFT: Recurrent All-Pairs Field Transforms for Optical Flow

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation

BERT Representations for Video Question Answering

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation

More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregation

On the Relationship between Self-Attention and Convolutional Layers

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Axial Attention in Multidimensional Transformers

STM: SpatioTemporal and Motion Encoding for Action Recognition

Only Time Can Tell: Discovering Temporal Data for Temporal Modeling

Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems

Stand-Alone Self-Attention in Vision Models

Video Modeling With Correlation Networks

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips

Scaling Autoregressive Video Models

Learning Spatio-Temporal Representation With Local and Global Diffusion

Generating Long Sequences with Sparse Transformers

Attention Augmented Convolutional Networks

Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks With Octave Convolution

Video Classification With Channel-Separated Convolutional Networks

VideoBERT: A Joint Model for Video and Language Representation Learning

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

D3D: Distilled 3D Networks for Video Action Recognition

Long-Term Feature Banks for Detailed Video Understanding

SlowFast Networks for Video Recognition

Video Action Transformer Network

Graph-Based Global Reasoning Networks

CCNet: Criss-Cross Attention for Semantic Segmentation

TSM: Temporal Shift Module for Efficient Video Understanding

A2-Nets: Double Attention Networks

RESOUND: Towards Action Recognition Without Representation Bias

A Short Note about Kinetics-600

Multi-Fiber Networks for Video Recognition

Scaling Neural Machine Translation

The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation

End-to-End Dense Video Captioning with Masked Transformer

Image Transformer

Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification

A Closer Look at Spatiotemporal Convolutions for Action Recognition

Relation Networks for Object Detection

Non-local Neural Networks

Revisiting the Effectiveness of Off-the-shelf Temporal Modeling Approaches for Large-scale Video Classification

The “Something Something” Video Database for Learning and Evaluating Visual Common Sense

Attention is All you Need

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset

ActionVLAD: Learning Spatio-Temporal Aggregation for Action Classification

Layer Normalization

Going deeper with convolutions

Very Deep Convolutional Networks for Large-Scale Image Recognition

ImageNet: A large-scale hierarchical image database

Distilled 3-D (D3D) networks for video action recognition

Pyslowfast

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

Using NLG for speech synthesis of mathematical sentences

Pytorch image models

Improving Language Understanding by Generative Pre-Training

Temporal Shift Module for Efficient Video Understanding

Is Space-Time Attention All You Need for Video

Visualizing Data using t-SNE

Field of Study

Journal Information

Name