What Would You Expect? Anticipating Egocentric Actions With Rolling-Unrolling LSTMs and Modality Attention (2019-05-22T00:00:00.000000Z)

TL;DR

This work tackles the problem proposing an architecture able to anticipate actions at multiple temporal scales using two LSTMs to summarize the past and formulate predictions about the future using a novel Modality ATTention mechanism which learns to weigh modalities in an adaptive fashion.

Abstract

Egocentric action anticipation consists in understanding which objects the camera wearer will interact with in the near future and which actions they will perform. We tackle the problem proposing an architecture able to anticipate actions at multiple temporal scales using two LSTMs to 1) summarize the past, and 2) formulate predictions about the future. The input video is processed considering three complimentary modalities: appearance (RGB), motion (optical flow) and objects (object-based features). Modality-specific predictions are fused using a novel Modality ATTention (MATT) mechanism which learns to weigh modalities in an adaptive fashion. Extensive evaluations on two large-scale benchmark datasets show that our method outperforms prior art by up to +7% on the challenging EPIC-Kitchens dataset including more than 2500 actions, and generalizes to EGTEA Gaze+. Our approach is also shown to generalize to the tasks of early action recognition and action recognition. Our method is ranked first in the public leaderboard of the EPIC-Kitchens egocentric action anticipation challenge 2019. Please see the project web page for code and additional details: http://iplab.dmi.unict.it/rulstm.

Authors

Antonino Furnari

18 papers

G. Farinella

19 papers

TL;DR

Abstract

Authors

References71 items

LSTA: Long Short-Term Attention for Egocentric Action Recognition

Leveraging Uncertainty to Rethink Loss Functions and Evaluation Measures for Egocentric Action Anticipation

In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video

Deepdiary: Lifelogging image captioning and summarization

Attention is All We Need: Nailing Down Object-centric Attention for Egocentric Activity Recognition

Adding Attentiveness to the Neurons in Recurrent Neural Networks

Scaling Egocentric Vision: The EPIC-KITCHENS Dataset

When will you do what? - Anticipating Temporal Occurrences of Activities

Modeling Temporal Structure with LSTM for Online Action Detection

Temporal Relational Reasoning in Videos

Next-active-object prediction from egocentric videos

Automatic differentiation in PyTorch

What will Happen Next? Forecasting Player Moves in Sports Videos

Joint Prediction of Activity Labels and Starting Times in Untrimmed Videos

Visual Forecasting by Imitating Dynamics in Natural Sequences

RED: Reinforced Encoder-Decoder Networks for Action Anticipation

Deep Future Gaze: Gaze Anticipation on Egocentric Videos Using Adversarial Networks

Spatiotemporal Multiplier Networks for Video Action Recognition

Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset

Forecasting Hand and Object Locations in Future Frames

Am I Done? Predicting Action Progress in Videos

Encouraging LSTMs to Anticipate Actions Very Early

First-Person Activity Forecasting with Online Inverse Reinforcement Learning

Long-Term Activity Forecasting Using First-Person Vision

Choosing smartly: Adaptive multimodal fusion for object detection in changing environments

Temporal Segment Networks: Towards Good Practices for Deep Action Recognition

Learning Activity Progression in LSTMs for Activity Detection and Early Detection

Egocentric Future Localization

First Person Action Recognition Using Deep Learned Descriptors

Going Deeper into First-Person Activity Recognition

Convolutional Two-Stream Network Fusion for Video Action Recognition

Online Action Detection

Trajectory aligned features for first person action recognition

KrishnaCam: Using a longitudinal, single-person, egocentric dataset for scene understanding tasks

Deep Residual Learning for Image Recognition

Generating Notifications for Missing Actions: Don't Forget to Turn the Lights Off!

Recurrent Neural Networks for driver activity anticipation via sensory-fusion architecture

Delving into egocentric actions

Anticipating Visual Representations from Unlabeled Video

Fast R-CNN

Car that Knows Before You Do: Anticipating Maneuvers via Learning Temporal Driving Models

Robot-Centric Activity Prediction from First-Person Videos: What Will They Do to Me?

Temporal Perception and Prediction in Ego-Centric Video

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Pooled motion features for first-person videos

Learning Spatiotemporal Features with 3D Convolutional Networks

Action-Reaction: Forecasting the Dynamics of Human Interaction

Sequential Max-Margin Event Detectors

A Hierarchical Representation for Future Action Prediction

Neural Machine Translation by Jointly Learning to Align and Translate

Large-Scale Video Classification with Convolutional Neural Networks

Two-Stream Convolutional Networks for Action Recognition in Videos

Anticipating Human Activities Using Object Affordances for Reactive Robotic Response

Recognize Human Activities from Partially Observed Videos

Dense Trajectories and Motion Boundary Descriptors for Action Recognition

Learning to Recognize Daily Actions Using Gaze

First-Person Vision

Detecting activities of daily living in first-person camera views

Max-Margin Early Event Detectors

Human activity prediction: Early recognition of ongoing activities from streaming videos

Understanding egocentric activities

Temporal segmentation and activity classification from first-person sensing

Learning realistic human actions from movies

A Duality Based Approach for Realtime TV-L1 Optical Flow

On Space-Time Interest Points

Quo vadis?

Research Showcase @ CMU

Author manuscript, published in "International Conference on Computer Vision (2013)" Action Recognition with Improved Trajectories

Detectron

Screenshots of the EPIC-Kitchens Egocentric Action Anticipation Challenge Leaderboards at the end of the competition

Field of Study

Journal Information

Name

Page

Venue Information