GRAB: A Dataset of Whole-Body Human Grasping of Objects (2020-08-25T00:00:00.000000Z)

TL;DR

This work collects a new dataset, called GRAB (GRasping Actions with Bodies), of whole-body grasps, containing full 3D shape and pose sequences of 10 subjects interacting with 51 everyday objects of varying shape and size, and trains GrabNet, a conditional generative network, to predict 3D handgrasps for unseen 3D object shapes.

Abstract

Training computers to understand, model, and synthesize human grasping requires a rich dataset containing complex 3D object shapes, detailed contact information, hand pose and shape, and the 3D body motion over time. While "grasping" is commonly thought of as a single hand stably lifting an object, we capture the motion of the entire body and adopt the generalized notion of "whole-body grasps". Thus, we collect a new dataset, called GRAB (GRasping Actions with Bodies), of whole-body grasps, containing full 3D shape and pose sequences of 10 subjects interacting with 51 everyday objects of varying shape and size. Given MoCap markers, we fit the full 3D body shape and pose, including the articulated face and hands, as well as the 3D object pose. This gives detailed 3D meshes over time, from which we compute contact between the body and object. This is a unique dataset, that goes well beyond existing ones for modeling and understanding how humans grasp and manipulate objects, how their full body is involved, and how interaction varies with the task. We illustrate the practical value of GRAB with an example application; we train GrabNet, a conditional generative network, to predict 3D hand grasps for unseen 3D object shapes. The dataset and code are available for research purposes at this https URL.

Authors

Michael J. Black

39 papers

Dimitrios Tzionas

12 papers

N. Ghorbani

3 papers

TL;DR

Abstract

Authors

References88 items

3D Printing

Monocular Expressive Body Regression through Body-Driven Attention

Physics-Based Dexterous Manipulations with Estimated Hand Poses and Residual Reinforcement Learning

GanHand: Predicting Human Grasp Affordances in Multi-Object Scenes

Generating 3D People in Scenes Without People

Neural state machine for character-scene interactions

Learning Task-Oriented Grasping From Human Activity Datasets

Learning Multi-human Optical Flow

DexPilot: Vision-Based Teleoperation of Dexterous Robotic Hand-Arm System

Efficient Learning on Point Clouds With Basis Point Sets

Resolving 3D Human Pose Ambiguities With 3D Scene Constraints

InteractionFusion

HO-3D: A Multi-User, Multi-Object Dataset for Joint 3D Hand-Object Pose Estimation

6-DOF GraspNet: Variational Grasp Generation for Object Manipulation

Geometric Pose Affordance: 3D Human Pose with Scene Constraints

Capture, Learning, and Synthesis of 3D Speaking Styles

Learning the signatures of the human grasp using a scalable tactile glove

ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging

Learning Joint Reconstruction of Hands and Manipulated Objects

Expressive Body Capture: 3D Hands, Face, and Body From a Single Image

H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions

Context-Aware Human Motion Prediction

AMASS: Archive of Motion Capture As Surface Shapes

Estimating 3D Motion and Forces of Person-Object Interactions From Monocular Video

Generalized Feedback Loop for Joint Hand-Object Pose Estimation

On the Continuity of Rotation Representations in Neural Networks

Hand-Object Contact Force Estimation from Markerless Visual Tracking

Joint 3D Tracking of a Deformable Object in Interaction with a Hand

Online optical marker-based hand tracking with deep labels

Generating 3D faces using Convolutional Mesh Autoencoders

iMapper

Fingers' Range and Comfortable Area for One-Handed Smartphone Interaction Beyond the Touchscreen

End-to-End Recovery of Human Shape and Pose

Learning a model of facial shape and expression from 4D scans

Dynamic FAUST: Registering Human Bodies in Motion

Understanding and exploiting object interaction landscapes

First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations

Learning from Synthetic Humans

Real-Time Joint Tracking of a Hand Manipulating an Object from RGB-D Input

Learning manipulation actions from human demonstrations

Understanding and Exploiting Object Interaction Landscapes

PiGraphs

Haptic SLAM: An Ideal Observer Model for Bayesian Inference of Object Shape and Hand Pose from Contact Dynamics

How We Type: Movement Strategies and Performance in Everyday Typing

The GRASP Taxonomy of Human Grasp Types

ShapeNet: An Information-Rich 3D Model Repository

Understanding Everyday Hands in Action from RGB-D Images

The KIT whole-body human motion database

Capturing Hands in Action Using Discriminative Salient Points and Physics Simulation

A whole-body pose taxonomy for loco-manipulation tasks

SceneGrok

Shape2Pose

Auto-Encoding Variational Bayes

Data-Driven Grasp Synthesis—A Survey

Video-based hand manipulation capture through composite motion control

Synthesis of detailed hand manipulations using contact sampling

Maximizing parallelism in the construction of BVHs, octrees, and k-d trees

An overview of 3D object grasp synthesis algorithms

Full DOF tracking of a hand interacting with an object by modeling occlusions and physical constraints

Microgeometry capture using an elastomeric sensor

Tracking people interacting with objects

An object-dependent hand pose prior from sparse training data

Hands in action: real-time 3D reconstruction of hands in interaction with objects

Physics-Based Person Tracking Using the Anthropomorphic Walker

Markerless Motion Capture with unsynchronized moving cameras

The Columbia grasp database

Markerless motion capture of man-machine interaction

Imitation Learning of Whole-Body Grasps

Interaction capture and synthesis

Motion patches: buildings blocks for virtual environments annotated with motion data

Physically based grasping control from example

A sensor fusion approach for recognizing continuous human grasping sequences using hidden Markov models

Graspit! A versatile simulator for robotic grasping

Handrix: animating the human hand

Photoplethysmograph fingernail sensors for measuring finger forces without haptic obstruction

Scene constraints-aided tracking of human body