Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video (2020-05-04T00:00:00.000000Z)

TL;DR

Rolling-Unrolling LSTM is contributed, a learning architecture to anticipate actions form egocentric videos that achieves competitive performance on ActivityNet with respect to methods not based on unsupervised pre-training and generalizes to the tasks of early action recognition and action recognition.

Abstract

In this paper, we tackle the problem of egocentric action anticipation, i.e., predicting what actions the camera wearer will perform in the near future and which objects they will interact with. Specifically, we contribute Rolling-Unrolling LSTM, a learning architecture to anticipate actions from egocentric videos. The method is based on three components: 1) an architecture comprised of two LSTMs to model the sub-tasks of summarizing the past and inferring the future, 2) a Sequence Completion Pre-Training technique which encourages the LSTMs to focus on the different sub-tasks, and 3) a Modality ATTention (MATT) mechanism to efficiently fuse multi-modal predictions performed by processing RGB frames, optical flow fields and object-based features. The proposed approach is validated on EPIC-Kitchens, EGTEA Gaze+ and ActivityNet. The experiments show that the proposed architecture is state-of-the-art in the domain of egocentric videos, achieving top performances in the 2019 EPIC-Kitchens egocentric action anticipation challenge. The approach also achieves competitive performance on ActivityNet with respect to methods not based on unsupervised pre-training and generalizes to the tasks of early action recognition and action recognition. To encourage research on this challenging topic, we made our code, trained models, and pre-extracted features available at our web page: http://iplab.dmi.unict.it/rulstm.

Authors

Antonino Furnari

18 papers

G. Farinella

19 papers

TL;DR

Abstract

Authors

References82 items

PyTorch: An Imperative Style, High-Performance Deep Learning Library

EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition

Leveraging the Present to Anticipate the Future in Videos

Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition

Long-Term Feature Banks for Detailed Video Understanding

LSTA: Long Short-Term Attention for Egocentric Action Recognition

TSM: Temporal Shift Module for Efficient Video Understanding

TrafficPredict: Trajectory Prediction for Heterogeneous Traffic-Agents

Leveraging Uncertainty to Rethink Loss Functions and Evaluation Measures for Egocentric Action Anticipation

In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video

Attention is All We Need: Nailing Down Object-centric Attention for Egocentric Activity Recognition

Adding Attentiveness to the Neurons in Recurrent Neural Networks

Scaling Egocentric Vision: The EPIC-KITCHENS Dataset

When will you do what? - Anticipating Temporal Occurrences of Activities

Modeling Temporal Structure with LSTM for Online Action Detection

An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling

A Closer Look at Spatiotemporal Convolutions for Action Recognition

Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?

Temporal Relational Reasoning in Videos

Next-active-object prediction from egocentric videos

What will Happen Next? Forecasting Player Moves in Sports Videos

Joint Prediction of Activity Labels and Starting Times in Untrimmed Videos

Visual Forecasting by Imitating Dynamics in Natural Sequences

RED: Reinforced Encoder-Decoder Networks for Action Anticipation

Deep Future Gaze: Gaze Anticipation on Egocentric Videos Using Adversarial Networks

Spatiotemporal Multiplier Networks for Video Action Recognition

Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset

Forecasting Hand and Object Locations in Future Frames

Am I Done? Predicting Action Progress in Videos

Encouraging LSTMs to Anticipate Actions Very Early

First-Person Activity Forecasting with Online Inverse Reinforcement Learning

Spatiotemporal Residual Networks for Video Action Recognition

Choosing smartly: Adaptive multimodal fusion for object detection in changing environments

Temporal Segment Networks: Towards Good Practices for Deep Action Recognition

Learning Activity Progression in LSTMs for Activity Detection and Early Detection

First Person Action Recognition Using Deep Learned Descriptors

Egocentric Future Localization

Going Deeper into First-Person Activity Recognition

Convolutional Two-Stream Network Fusion for Video Action Recognition

Online Action Detection

Trajectory aligned features for first person action recognition

Deep Residual Learning for Image Recognition

Generating Notifications for Missing Actions: Don't Forget to Turn the Lights Off!

Recurrent Neural Networks for driver activity anticipation via sensory-fusion architecture

Delving into egocentric actions

ActivityNet: A large-scale video benchmark for human activity understanding

Anticipating Visual Representations from Unlabeled Video

Fast R-CNN

Car that Knows Before You Do: Anticipating Maneuvers via Learning Temporal Driving Models

Robot-Centric Activity Prediction from First-Person Videos: What Will They Do to Me?

Temporal Perception and Prediction in Ego-Centric Video

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Pooled motion features for first-person videos

Learning Spatiotemporal Features with 3D Convolutional Networks

Sequence to Sequence Learning with Neural Networks

A Hierarchical Representation for Future Action Prediction

Action-Reaction: Forecasting the Dynamics of Human Interaction

Sequential Max-Margin Event Detectors

Neural Machine Translation by Jointly Learning to Align and Translate

Large-Scale Video Classification with Convolutional Neural Networks

Two-Stream Convolutional Networks for Action Recognition in Videos

Anticipating Human Activities Using Object Affordances for Reactive Robotic Response

Recognize Human Activities from Partially Observed Videos

Dense Trajectories and Motion Boundary Descriptors for Action Recognition

Learning to Recognize Daily Actions Using Gaze

First-Person Vision

Detecting activities of daily living in first-person camera views

Max-Margin Early Event Detectors

Understanding egocentric activities

Human activity prediction: Early recognition of ongoing activities from streaming videos

Temporal segmentation and activity classification from first-person sensing

Learning realistic human actions from movies

A Duality Based Approach for Realtime TV-L1 Optical Flow

On Space-Time Interest Points

Learning to Forget: Continual Prediction with LSTM