ST-HOI: A Spatial-Temporal Baseline for Human-Object Interaction Detection in Videos (2021-05-25T00:00:00.000000Z)

TL;DR

This paper shows that a naive temporal-aware variant of a common action detection baseline does not work on video-based HOIs due to a feature-inconsistency issue, and proposes a simple yet effective architecture utilizing temporal information such as human and object trajectories, correctly-localized visual features, and spatial-temporal masking pose features.

Abstract

Detecting human-object interactions (HOI) is an important step toward a comprehensive visual understanding of machines. While detecting non-temporal HOIs (e.g., sitting on a chair) from static images is feasible, it is unlikely even for humans to guess temporal-related HOIs (e.g., opening/closing a door) from a single video frame, where the neighboring frames play an essential role. However, conventional HOI methods operating on only static images have been used to predict temporal-related interactions, which is essentially guessing without temporal contexts and may lead to sub-optimal performance. In this paper, we bridge this gap by detecting video-based HOIs with explicit temporal information. We first show that a naive temporal-aware variant of a common action detection baseline does not work on video-based HOIs due to a feature-inconsistency issue. We then propose a simple yet effective architecture named Spatial-Temporal HOI Detection (ST-HOI) utilizing temporal information such as human and object trajectories, correctly-localized visual features, and spatial-temporal masking pose features. We construct a new video HOI benchmark dubbed VidHOI where our proposed approach serves as a solid baseline.

Authors

Jiashi Feng

6 papers

Roger Zimmermann

7 papers

Meng-Jiun Chiou

2 papers

TL;DR

Abstract

Authors

References50 items

LIGHTEN: Learning Interactions with Graph and Hierarchical TEmporal Networks for HOI in videos

Visual Relationship Detection With Visual-Linguistic Knowledge From Multimodal Representations

Zero-Shot Multi-View Indoor Localization via Graph Location Networks

PaStaNet: Toward Human Activity Knowledge Engine

Learning Human-Object Interaction Detection Using Interaction Points

VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutions

Detection

PPDM: Parallel Point Detection and Matching for Real-Time Human-Object Interaction Detection

Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs

Multi-Level Fusion based Class-aware Attention Model for Weakly Labeled Audio Tagging

Video Visual Relation Detection via Multi-modal Feature Fusion

Deep Contextual Attention for Human-Object Interaction Detection

Pose-Aware Multi-Level Feature Network for Human Object Interaction Detection

Constructing an Evaluation Model for User Experience in an Unmanned Store

Annotating Objects and Relations in User-Generated Videos

Long-Term Feature Banks for Detailed Video Understanding

SlowFast Networks for Video Recognition

Transferable Interactiveness Knowledge for Human-Object Interaction Detection

No-Frills Human-Object Interaction Detection: Factorization, Layout Encodings, and Training Techniques

Learning Human-Object Interactions by Graph Parsing Neural Networks

iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection

Towards Reaching Human Performance in Pedestrian Detection

Non-local Neural Networks

Video Visual Relation Detection

AVA: A Video Dataset of Spatio-Temporally Localized Atomic Visual Actions

Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset

Detecting and Recognizing Human-Object Interactions

Tube Convolutional Neural Network (T-CNN) for Action Detection in Videos

Mask R-CNN

Learning to Detect Human-Object Interactions

Scene Graph Generation by Iterative Message Passing

RMPE: Regional Multi-person Pose Estimation

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

Visual Relationship Detection with Language Priors

Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding

Deep Residual Learning for Image Recognition

Structural-RNN: Deep Learning on Spatio-Temporal Graphs

Visual Semantic Role Labeling

VQA: Visual Question Answering

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Learning Spatiotemporal Features with 3D Convolutional Networks

Show and tell: A neural image caption generator

Gaze locking: passive eye contact detection for human-object interaction

Anticipating Human Activities Using Object Affordances for Reactive Robotic Response

Learning human activities and object affordances from RGB-D videos

Distinctive Image Features from Scale-Invariant Keypoints

Polysemy Deciphering Network for Human-Object Interaction Detection

Question about the annotations · Issue #2

Abhinav Gupta, and Kaiming He

Deep Learning

Field of Study

Journal Information

Name