First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations (2017-04-08T00:00:00.000000Z)

TL;DR

This work collects RGB-D video sequences comprised of more than 100K frames of 45 daily hand action categories, involving 26 different objects in several hand configurations, and sees clear benefits of using hand pose as a cue for action recognition compared to other data modalities.

Abstract

In this work we study the use of 3D hand poses to recognize first-person dynamic hand actions interacting with 3D objects. Towards this goal, we collected RGB-D video sequences comprised of more than 100K frames of 45 daily hand action categories, involving 26 different objects in several hand configurations. To obtain hand pose annotations, we used our own mo-cap system that automatically infers the 3D location of each of the 21 joints of a hand model via 6 magnetic sensors and inverse kinematics. Additionally, we recorded the 6D object poses and provide 3D object models for a subset of hand-object interaction sequences. To the best of our knowledge, this is the first benchmark that enables the study of first-person hand actions with the use of 3D hand poses. We present an extensive experimental evaluation of RGB-D and pose-based action recognition by 18 baselines/state-of-the-art approaches. The impact of using appearance features, poses, and their combinations are measured, and the different training/testing protocols are evaluated. Finally, we assess how ready the 3D hand pose estimation field is when hands are severely occluded by objects in egocentric views and its influence on action recognition. From the results, we see clear benefits of using hand pose as a cue for action recognition compared to other data modalities. Our dataset and experiments can be of interest to communities of 3D hand pose estimation, 6D object pose, and robotics as well as action recognition.

Authors

Guillermo Garcia-Hernando

2 papers

Shanxin Yuan

3 papers

Seungryul Baek

1 papers

TL;DR

Abstract

Authors

References93 items

Augmented Skeleton Space Transfer for Depth-Based Hand Pose Estimation

3D Hand Pose Estimation: From Current Achievements to Future Goals

Robust Hand Pose Estimation during the Interaction with an Unknown Object

Scene Semantic Reconstruction from Egocentric RGB-D-Thermal Videos

The 2017 Hands in the Million Challenge on 3D Hand Pose Estimation

BigHand2.2M Benchmark: Hand Pose Dataset and State of the Art Analysis

Real-Time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor

Learning and Refining of Privileged Information-Based RNNs for Action Recognition from Depth Sequences

Graph Based Skeleton Motion Representation and Similarity Measurement for Action Recognition

Real-Time Joint Tracking of a Hand Manipulating an Object from RGB-D Input

Prediction of Manipulation Actions

SEMBED: Semantic Embedding of Egocentric Action Videos

Kinematic-Layout-aware Random Forests for Depth-based Action Recognition

Transition Forests: Learning Discriminative Temporal Transitions for Action Recognition and Detection

Spatio-Temporal Hough Forest for efficient detection-localisation-recognition of fingerwriting in egocentric camera

3D Action Recognition from Novel Viewpoints

Mining 3D Key-Pose-Motifs for Action Recognition

Rolling Rotations for Recognizing Human Actions from 3D Skeletal Data

Online Detection and Classification of Dynamic Hand Gestures with Recurrent 3D Convolutional Neural Networks

Efficient Temporal Sequence Comparison and Classification Using Gram Matrix Embeddings on a Riemannian Manifold

Skeleton-Based Dynamic Hand Gesture Recognition

Understanding Hand-Object Manipulation with Grasp Types and Object Attributes

First Person Action Recognition Using Deep Learned Descriptors

Going Deeper into First-Person Activity Recognition

Efficiently Creating 3D Training Data for Fine Hand Pose Estimation

Convolutional Two-Stream Network Fusion for Video Action Recognition

Spatial Attention Deep Net with Partial PSO for Hierarchical Hybrid Hand Pose Estimation

NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis

Co-Occurrence Feature Learning for Skeleton Based Action Recognition Using Regularized Deep LSTM Networks

Convolutional Pose Machines

Recognizing hand-object interactions in wearable camera videos

Training a Feedback Loop for Hand Pose Estimation

Understanding Everyday Hands in Action from RGB-D Images

Lending A Hand: Detecting Hands and Recognizing Activities in Complex Egocentric Interactions

Multimodal Multipart Learning for Action Recognition in Depth Videos

Rule of thumb: Deep derotation for improved fingertip detection

Jointly Learning Heterogeneous Features for RGB-D Activity Recognition

Hierarchical recurrent neural network for skeleton based action recognition

First-person pose recognition using egocentric workspaces

Grasp type revisited: A modern perspective on a classical feature for vision

How do we use our hands? Discovering a diverse set of common grasps

Cascaded hand pose regression

Capturing Hands in Action Using Discriminative Salient Points and Physics Simulation

A scalable approach for understanding the visual structures of hand grasps

Differential Recurrent Neural Networks for Action Recognition

3D Finger CAPE: Clicking Action and Position Estimation under Self-Occlusions in Egocentric Viewpoint

Accurate, Robust, and Flexible Real-time Hand Tracking

The Yale human grasping dataset: Grasp, object, and task data in household and machine shop environments

Hand Segmentation with Structured Convolutional Learning

Histogram of Oriented Principal Components for Cross-View Action Recognition

Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks

3D Hand Pose Detection in Egocentric RGB-D Images

Hand Gesture Recognition in Real Time for Automotive Interfaces: A Multimodal Vision-Based Approach and Evaluations

Leveraging Hierarchical Parametric Networks for Skeletal Joints Based Action Segmentation and Recognition

Latent Regression Forest: Structured Estimation of 3D Articulated Hand Posture

Iterated Second-Order Label Sensitive Pooling for 3D Human Pose Estimation

Super Normal Vector for Activity Recognition Using Depth Sequences

Experiments on an RGB-D Wearable Vision System for Egocentric Activity Recognition

Human Action Recognition by Representing 3D Skeletons as Points in a Lie Group

Realtime and Robust Hand Tracking from Depth

Parsing the Hand in Depth Images

The Moving Pose: An Efficient 3D Kinematics Descriptor for Low-Latency Action Recognition and Detection

Real-Time Articulated Hand Pose Estimation Using Semi-supervised Transductive Regression Forests

Learning Discriminative Representations from RGB-D Video Data

Non-parametric hand pose estimation with object context

Fusing Spatiotemporal Features and Joints for 3D Action Recognition

HON4D: Histogram of Oriented 4D Normals for Activity Recognition from Depth Sequences

Unconstrained Monocular 3D Human Pose Estimation by Action Detection and Cross-Modality Regression Forest

Hand Pose Estimation and Hand Shape Classification Using Multi-layered Randomized Decision Forests

Robust 3D Action Recognition with Random Occupancy Patterns

Fine-grained kitchen activity recognition using RGB-D

Detecting activities of daily living in first-person camera views

View invariant human action recognition using histograms of 3D joints

Understanding egocentric activities

Full DOF tracking of a hand interacting with an object by modeling occlusions and physical constraints

Learning to recognize objects in egocentric activities