Structural-RNN: Deep Learning on Spatio-Temporal Graphs (2015-11-17T00:00:00.000000Z)

TL;DR

This paper develops a scalable method for casting an arbitrary spatio-temporal graph as a rich RNN mixture that is feedforward, fully differentiable, and jointly trainable and shows improvement over the state-of-the-art with a large margin.

Abstract

Deep Recurrent Neural Network architectures, though remarkably capable at modeling sequences, lack an intuitive high-level spatio-temporal structure. That is while many problems in computer vision inherently have an underlying high-level structure and can benefit from it. Spatiotemporal graphs are a popular tool for imposing such high-level intuitions in the formulation of real world problems. In this paper, we propose an approach for combining the power of high-level spatio-temporal graphs and sequence learning success of Recurrent Neural Networks (RNNs). We develop a scalable method for casting an arbitrary spatio-temporal graph as a rich RNN mixture that is feedforward, fully differentiable, and jointly trainable. The proposed method is generic and principled as it can be used for transforming any spatio-temporal graph through employing a certain set of well defined steps. The evaluations of the proposed approach on a diverse set of problems, ranging from modeling human motion to object interactions, shows improvement over the state-of-the-art with a large margin. We expect this method to empower new approaches to problem formulation through high-level spatio-temporal graphs and Recurrent Neural Networks.

Authors

Ashutosh Saxena

3 papers

Ashesh Jain

3 papers

Amir Zamir

6 papers

TL;DR

Abstract

Authors

References66 items

Beyond Geometric Path Planning: Learning Context-Driven Trajectory Preferences via Sub-optimal Feedback

Objects2action: Classifying and Localizing Actions without Any Video Example

Semantic Image Segmentation via Deep Parsing Network

Maximum-Margin Structured Learning with Deep Networks for 3D Human Pose Estimation

Recurrent Network Models for Human Dynamics

Scene labeling with LSTM recurrent neural networks

Mind's eye: A recurrent visual representation for image caption generation

Hierarchical recurrent neural network for skeleton based action recognition

Visualizing and Understanding Recurrent Networks

Fast R-CNN

Car that Knows Before You Do: Anticipating Maneuvers via Learning Temporal Driving Models

Efficient Piecewise Training of Deep Structured Models for Semantic Segmentation

Fully Connected Deep Structured Networks

Unsupervised Learning of Video Representations using LSTMs

Conditional Random Fields as Recurrent Neural Networks

Learning Longer Memory in Recurrent Neural Networks

Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

Translating Videos to Natural Language Using Deep Recurrent Neural Networks

Learning Spatiotemporal Features with 3D Convolutional Networks

Long-term recurrent convolutional networks for visual recognition and description

Recognizing object affordances in terms of spatio-temporal object-object relationships

Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks

Sequence to Sequence Learning with Neural Networks

Learning Deep Structured Models

Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments

Towards End-To-End Speech Recognition with Recurrent Neural Networks

Overtaking vehicle detection using a spatio-temporal CRF

ACTIVE: Activity Concept Transitions in Video Event Classification

PANDA: Pose Aligned Networks for Deep Attribute Modeling

Generating Sequences With Recurrent Neural Networks

Anticipating Human Activities Using Object Affordances for Reactive Robotic Response

Learning Spatio-Temporal Structure from RGB-D Videos for Human Activity Detection and Anticipation

Learning human activities and object affordances from RGB-D videos

Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials

Learning spatiotemporal graphs of human activities

Parsing Natural Scenes and Natural Language with Recursive Neural Networks

Track to the future: Spatio-temporal video segmentation with long-range motion cues

Human Action Segmentation and Recognition Using Discriminative Semi-Markov Models

An Introduction to Conditional Random Fields

Dynamical binary latent variable models for 3D human pose tracking

FACTORIE: Probabilistic Programming via Imperatively Defined Factor Graphs

Observing Human-Object Interactions: Using Spatial and Functional Compatibility for Recognition

Cutting-plane training of structural SVMs

Probabilistic Graphical Models - Principles and Techniques

Curriculum learning

Factored conditional restricted Boltzmann Machines for modeling motion style

The Recurrent Temporal Restricted Boltzmann Machine

Key Object Driven Multi-category Object Recognition, Localization and Tracking Using Spatio-temporal Context

Topologically-constrained latent variable models

A discriminatively trained, multiscale, deformable part model

Learning realistic human actions from movies

Gaussian Process Dynamical Models for Human Motion

Conditional random fields for activity recognition

Modeling Human Motion Using Binary Latent Variables

Markov logic networks

Conditional Random Fields for Object Recognition

Discriminative Probabilistic Models for Relational Data

Factor graphs and the sum-product algorithm

Global training of document processing systems using graph transformer networks

Globally Trained Handwritten Word Recognizer Using Spatial Representation, Convolutional Neural Networks, and Hidden Markov Models

Anticipatory planning for humanrobot teams

MoSIFT: Recognizing Human Actions in Surveillance Videos

A Spatio-Temporal Probabilistic Model for Multi-Sensor Multi-Class Object Recognition

Learning task-dependent distributed representations by backpropagation through structure

Author manuscript, published in "International Conference on Computer Vision (2013)" Action Recognition with Improved Trajectories

supplementary video and material

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names