Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos (2021-08-16T00:00:00.000000Z)

TL;DR

This work proposes a learning-free fitting approach for hand-object reconstruction which can seamlessly handle two-hand object interactions and shows that it can be applied to datasets with varying levels of difficulty for which training data is unavailable.

Abstract

Our work aims to obtain 3D reconstruction of hands and manipulated objects from monocular videos. Reconstructing hand-object manipulations holds a great potential for robotics and learning from human demonstrations. The supervised learning approach to this problem, however, requires 3D supervision and remains limited to constrained laboratory settings and simulators for which 3D ground truth is available. In this paper we first propose a learning-free fitting approach for hand-object reconstruction which can seamlessly handle two-hand object interactions. Our method relies on cues obtained with common methods for object detection, hand pose estimation and instance segmentation. We quantitatively evaluate our approach and show that it can be applied to datasets with varying levels of difficulty for which training data is unavailable.

Authors

Gül Varol

7 papers

I. Laptev

24 papers

C. Schmid

33 papers

TL;DR

Abstract

Authors

References73 items

Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time

H2O: Two Hands Manipulating Objects for First Person Interaction Recognition

DexYCB: A Benchmark for Capturing Hand Grasping of Objects

Reconstructing Hand-Object Interactions in the Wild

GRAB: A Dataset of Whole-Body Human Grasping of Objects

InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image

FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration

CosyPose: Consistent multi-view multi-object 6D pose estimation

Grasping Field: Learning Implicit Representations for Human Grasps

Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild

ContactPose: A Dataset of Grasps with Object Contact and Hand Pose

Recent Advances in 3D Object and Hand Pose Estimation

Understanding Human Hands in Contact at Internet Scale

Coherent Reconstruction of Multiple Humans From a Single Image

GanHand: Predicting Human Grasp Affordances in Multi-Object Scenes

ManifoldPlus: A Robust and Scalable Watertight Manifold Surface Generation Method for Triangle Soups

Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction

Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild

HOPE-Net: A Graph-Based Model for Hand-Object Pose Estimation

PointRend: Image Segmentation As Rendering

Learning Task-Oriented Grasping From Human Activity Datasets

Exploiting Spatial-Temporal Relationships for 3D Pose Estimation via Graph Convolutional Networks

Learning to Reconstruct 3D Human Pose and Shape via Model-Fitting in the Loop

FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images

Resolving 3D Human Pose Ambiguities With 3D Scene Constraints

HOnnotate: A Method for 3D Annotation of Hand and Object Poses

Exploiting Temporal Context for 3D Human Pose Estimation in the Wild

ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging

Learning Joint Reconstruction of Hands and Manipulated Objects

Expressive Body Capture: 3D Hands, Face, and Body From a Single Image

H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions

3D Hand Shape and Pose From Images in the Wild

DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation

On the Continuity of Rotation Representations in Neural Networks

Occupancy Networks: Learning 3D Reconstruction in Function Space

Learning 3D Human Dynamics From Video

Hand-Object Contact Force Estimation from Markerless Visual Tracking

SFV

Hand Pose Estimation via Latent 2.5D Heatmap Regression

Scaling Egocentric Vision: The EPIC-KITCHENS Dataset

Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images

DeepIM: Deep Iterative Matching for 6D Pose Estimation

A Papier-Mache Approach to Learning 3D Surface Generation

GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB

Exploiting Temporal Information for 3D Human Pose Estimation

Neural 3D Mesh Renderer

Embodied hands

PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes

CORe50: a New Dataset and Benchmark for Continuous Object Recognition

Learning to Estimate 3D Hand Pose from Single RGB Images

Hand Keypoint Detection in Single Images Using Multiview Bootstrapping

First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations

Real-Time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor

Unite the People: Closing the Loop Between 3D and 2D Human Representations

Real-Time Joint Tracking of a Hand Manipulating an Object from RGB-D Input

Keep It SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image

ShapeNet: An Information-Rich 3D Model Repository

3D Object Reconstruction from Hand-Object Interactions

The YCB object and Model set: Towards common benchmarks for manipulation research

Capturing Hands in Action Using Discriminative Salient Points and Physics Simulation

Adam: A Method for Stochastic Optimization

Scalable 3D Tracking of Multiple Interacting Objects

Microsoft COCO: Common Objects in Context

Physically Plausible 3D Scene Tracking: The Single Actor Hypothesis

Motion Capture of Hands in Action Using Discriminative Salient Points

Full DOF tracking of a hand interacting with an object by modeling occlusions and physical constraints

Generic Remeshing of 3D Triangular Meshes with Metric-Dependent Discrete Voronoi Diagrams

Embodied Hands : Modeling and Capturing Hands and Bodies Together * * Supplementary Material * *

Kalman and bayesian ﬁlters in python

on Pattern Analysis and Machine Intelligence

segmentation and initial hand pose estimation, pre-processing 10 frames of dimension 640*480 takes less than a second on a Tesla V100 Nvidia GPU

Wiktor Muron

Zhang , Panna Felsen , and Jitendra Malik . Learning 3 D human dynamics from video

Field of Study

Journal Information

Name