A Closer Look at Spatiotemporal Convolutions for Action Recognition (2017-11-30T00:00:00.000000Z)

TL;DR

A new spatiotemporal convolutional block "R(2+1)D" is designed which produces CNNs that achieve results comparable or superior to the state-of-the-art on Sports-1M, Kinetics, UCF101, and HMDB51.

Abstract

In this paper we discuss several forms of spatiotemporal convolutions for video analysis and study their effects on action recognition. Our motivation stems from the observation that 2D CNNs applied to individual frames of the video have remained solid performers in action recognition. In this work we empirically demonstrate the accuracy advantages of 3D CNNs over 2D CNNs within the framework of residual learning. Furthermore, we show that factorizing the 3D convolutional filters into separate spatial and temporal components yields significantly gains in accuracy. Our empirical study leads to the design of a new spatiotemporal convolutional block "R(2+1)D" which produces CNNs that achieve results comparable or superior to the state-of-the-art on Sports-1M, Kinetics, UCF101, and HMDB51.

Authors

Yann LeCun

18 papers

Du Tran

5 papers

L. Torresani

6 papers

TL;DR

Abstract

Authors

References45 items

Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset

The Kinetics Human Action Video Dataset

ActionVLAD: Learning Spatio-Temporal Aggregation for Action Classification

Spatiotemporal Residual Networks for Video Action Recognition

Densely Connected Convolutional Networks

Temporal Segment Networks: Towards Good Practices for Deep Action Recognition

Online Detection and Classification of Dynamic Hand Gestures with Recurrent 3D Convolutional Neural Networks

Convolutional Two-Stream Network Fusion for Video Action Recognition

Long-Term Temporal Convolutions for Action Recognition

Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs

Deep Residual Learning for Image Recognition

Actions ~ Transformations

Delving Deeper into Convolutional Networks for Learning Video Representations

Human Action Recognition Using Factorized Spatio-Temporal Convolutional Networks

Jointly Modeling Embedding and Translation to Bridge Video and Language

Beyond short snippets: Deep networks for video classification

Unsupervised Learning of Video Representations using LSTMs

Learning Spatiotemporal Features with 3D Convolutional Networks

Long-term recurrent convolutional networks for visual recognition and description

A discriminative CNN video representation for event detection

Going deeper with convolutions

Very Deep Convolutional Networks for Large-Scale Image Recognition

Large-Scale Video Classification with Convolutional Neural Networks

Two-Stream Convolutional Networks for Action Recognition in Videos

ImageNet classification with deep convolutional neural networks

UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

Action bank: A high-level representation of activity in video

Sequential Deep Learning for Human Action Recognition

HMDB: A large video database for human motion recognition

Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis

Convolutional Learning of Spatio-temporal Features

A Spatio-Temporal Descriptor Based on 3D-Gradients

A 3-dimensional sift descriptor and its application to action recognition

A Duality Based Approach for Realtime TV-L1 Optical Flow

Human Detection Using Oriented Histograms of Flow and Appearance

Behavior recognition via sparse spatio-temporal features

On Space-Time Interest Points

Two-Frame Motion Estimation Based on Polynomial Expansion

Quo vadis?

Ieee Transactions on Pattern Analysis and Machine Intelligence 1 3d Convolutional Neural Networks for Human Action Recognition

Author manuscript, published in "International Conference on Computer Vision (2013)" Action Recognition with Improved Trajectories

Caffe2: A new lightweight, modular, and scalable deep learning framework

Caffe2-Team

Field of Study

Journal Information

Name

Page