NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019-05-12T00:00:00.000000Z)

TL;DR

This work introduces a large-scale dataset for RGB+D human action recognition, which is collected from 106 distinct subjects and contains more than 114 thousand video samples and 8 million frames, and investigates a novel one-shot 3D activity recognition problem on this dataset.

Abstract

Research on depth-based human activity analysis achieved outstanding performance and demonstrated the effectiveness of 3D representation for action recognition. The existing depth-based and RGB+D-based action recognition benchmarks have a number of limitations, including the lack of large-scale training samples, realistic number of distinct class categories, diversity in camera views, varied environmental conditions, and variety of human subjects. In this work, we introduce a large-scale dataset for RGB+D human action recognition, which is collected from 106 distinct subjects and contains more than 114 thousand video samples and 8 million frames. This dataset contains 120 different action classes including daily, mutual, and health-related activities. We evaluate the performance of a series of existing 3D activity analysis methods on this dataset, and show the advantage of applying deep learning methods for 3D-based human action recognition. Furthermore, we investigate a novel one-shot 3D activity recognition problem on our dataset, and a simple yet effective Action-Part Semantic Relevance-aware (APSR) framework is proposed for this task, which yields promising results for recognition of the novel action classes. We believe the introduction of this large-scale dataset will enable the community to apply, adapt, and develop various data-hungry learning techniques for depth-based and RGB+D-based human activity understanding.

Authors

G. Wang

4 papers

Amir Shahroudy

2 papers

Jun Liu

2 papers

TL;DR

Abstract

Authors

References116 items

Early Action Prediction by Soft Regression

A Real-Time and Hardware-Efficient Processor for Skeleton-Based Action Recognition With Lightweight Convolutional Neural Network

Skeleton-Based Online Action Prediction Using Scale Selection Network

Feature Boosting Network For 3D Pose Estimation

Global Regularizer and Temporal-Aware Cross-Entropy for Skeleton-Based Early Action Recognition

Deep Bilinear Learning for RGB-D Action Recognition

Recognizing Human Actions as the Evolution of Pose Estimation Maps

One-Shot Action Localization by Learning Sequence Matching Network

SSNet: Scale Selection Network for Online 3D Action Prediction

Deep Progressive Reinforcement Learning for Skeleton-Based Action Recognition

Coding Kendall's Shape Trajectories for 3D Action Recognition

Video Representation Learning Using Discriminative Pooling

Learning Clip Representations for Skeleton-Based 3D Action Recognition

2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning

Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points

Fusing Geometric Features for Skeleton-Based Action Recognition Using Multilayer LSTM Networks

Learning Action Recognition Model from Depth and Skeleton Videos

RGB-D-based Human Motion Recognition with Deep Learning: A Survey

Enhanced skeleton visualization for view invariant human action recognition

Global Context-Aware Attention LSTM Networks for 3D Action Recognition

When Kernel Methods Meet Feature Learning: Log-Covariance Network for Action Recognition From Skeletal Data

Skeleton-Based Human Action Recognition With Global Context-Aware Attention LSTM Networks

3D action recognition using data visualization and convolutional neural networks

Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates

The Kinetics Human Action Video Dataset

Relevance-based Word Embedding

SphereFace: Deep Hypersphere Embedding for Face Recognition

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks

Interpretable 3D Human Action Analysis with Temporal Convolutional Networks

Chained Multi-stream Networks Exploiting Pose, Motion, and Appearance for Action Classification and Detection

SkeletonNet: Mining Deep Part Features for 3-D Action Recognition

Pose-conditioned Spatio-Temporal Attention for Human Action Recognition

Learning and Refining of Privileged Information-Based RNNs for Action Recognition from Depth Sequences

View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data

PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding

A New Representation of Skeleton Sequences for 3D Action Recognition

Scene Flow to Action Map: A New Representation for RGB-D Based Action Recognition with Convolutional Neural Networks

Unsupervised Learning of Long-Term Motion Dynamics for Videos

Deep Learning on Lie Groups for Skeleton-Based Action Recognition

Optimization as a Model for Few-Shot Learning

Explore Efficient Local Features from RGB-D Data for One-Shot Learning Gesture Recognition

Action Recognition From Depth Maps Using Deep Convolutional Neural Networks

Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition

Representation learning for very short texts using weighted word embedding aggregation

A novel hierarchical framework for human action recognition

Hierarchical Attention Networks for Document Classification

Matching Networks for One Shot Learning

3D skeleton-based human action classification: A survey

NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis

Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos

Co-Occurrence Feature Learning for Skeleton Based Action Recognition Using Regularized Deep LSTM Networks

RGB-D-based action recognition datasets: A survey

Space-Time Representation of People Based on 3D Skeletal Data: A Review

UTD-MHAD: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor

Multi-modal & Multi-view & Interactive Benchmark Dataset for Human Action Recognition

Multimodal Multipart Learning for Action Recognition in Depth Videos

A Survey of Applications and Human Motion Recognition with Microsoft Kinect

Cross Modal Distillation for Supervision Transfer

Jointly Learning Heterogeneous Features for RGB-D Activity Recognition

ActivityNet: A large-scale video benchmark for human activity understanding

Hierarchical recurrent neural network for skeleton based action recognition

Bilinear heterogeneous information machine for RGB-D action recognition

Learning a non-linear knowledge transfer model for cross-view action recognition

Differential Recurrent Neural Networks for Action Recognition

Indoor Scene Understanding with RGB-D Images: Bottom-up Segmentation, Object Detection and Semantic Segmentation

Deep visual-semantic alignments for generating image descriptions

3D Human Activity Recognition with Reconfigurable Convolutional Neural Networks

3D Object Recognition in Cluttered Scenes with Local Surface Features: A Survey

Human activity recognition from 3D data: A review

GloVe: Global Vectors for Word Representation

Histogram of Oriented Principal Components for Cross-View Action Recognition

Very Deep Convolutional Networks for Large-Scale Image Recognition

Skeletal Quads: Human Action Recognition Using Joint Quadruples

HOPC: Histogram of Oriented Principal Components of 3D Pointclouds for Action Recognition

Super Normal Vector for Activity Recognition Using Depth Sequences