AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction

Published in

European Conference on Computer Vision(2022)

External Links:

Generate Graph DownloadPDF

TL;DR

This work aims to improve SDF models using priors provided by parametric representations and proposes a joint learning framework that disentangles the pose and the shape and shows that such aligned SDFs better focus on reconstructing shape details and improve reconstruction accuracy both for hands and objects.

Abstract

Recent work achieved impressive progress towards joint reconstruction of hands and manipulated objects from monocular color images. Existing methods focus on two alternative representations in terms of either parametric meshes or signed distance fields (SDFs). On one side, parametric models can benefit from prior knowledge at the cost of limited shape deformations and mesh resolutions. Mesh models, hence, may fail to precisely reconstruct details such as contact surfaces of hands and objects. SDF-based methods, on the other side, can represent arbitrary details but are lacking explicit priors. In this work we aim to improve SDF models using priors provided by parametric representations. In particular, we propose a joint learning framework that disentangles the pose and the shape. We obtain hand and object poses from parametric models and use them to align SDFs in 3D space. We show that such aligned SDFs better focus on reconstructing shape details and improve reconstruction accuracy both for hands and objects. We evaluate our method and demonstrate significant improvements over the state of the art on the challenging ObMan and DexYCB benchmarks.

Authors

I. Laptev

24 papers

C. Schmid

33 papers

Yana Hasson

3 papers

References76 items

What's in your hands? 3D Reconstruction of Generic Objects in Hands

Self-Supervised 3D Hand Pose Estimation from monocular RGB via Contrastive Learning

A Skeleton-Driven Neural Occupancy Representation for Articulated Hands

ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesis

Single Depth View Based Real-Time Reconstruction of Hand-Object Interactions

AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction

Published in

European Conference on Computer Vision(2022)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

I. Laptev

24 papers

C. Schmid

33 papers

Yana Hasson

3 papers

References76 items

What's in your hands? 3D Reconstruction of Generic Objects in Hands

Self-Supervised 3D Hand Pose Estimation from monocular RGB via Contrastive Learning

A Skeleton-Driven Neural Occupancy Representation for Articulated Hands

ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesis

Single Depth View Based Real-Time Reconstruction of Hand-Object Interactions

Shape As Points: A Differentiable Poisson Solver

Single-view robot pose and joint angle estimation via render & compare

DexYCB: A Benchmark for Capturing Hand Grasping of Objects

SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes

SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks

Camera-Space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registration

A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose

MVHM: A Large-Scale Multi-View Hand Mesh Benchmark for Accurate 3D Hand Pose Estimation

CPF: Learning a Contact Potential Field to Model the Hand-Object Interaction

DeepHandMesh: A Weakly-supervised Deep Encoder-Decoder Framework for High-fidelity Hand Mesh Modeling

CosyPose: Consistent multi-view multi-object 6D pose estimation

Grasping Field: Learning Implicit Representations for Human Grasps

3D Hand Pose Estimation Using Synthetic Data and Weakly Labeled RGB Images

Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction

Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild

HOPE-Net: A Graph-Based Model for Hand-Object Pose Estimation

Monocular Real-Time Hand Shape and Motion Capture Using Multi-Modal Data

NASA: Neural Articulated Shape Approximation

FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images

A2J: Anchor-to-Joint Regression Network for 3D Articulated Pose Estimation From a Single Depth Image

Real-time pose and shape reconstruction of two interacting hands with a single depth camera

HOnnotate: A Method for 3D Annotation of Hand and Object Poses

DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction

PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization

Single Image 3D Hand Reconstruction with Mesh Convolutions

Learning Joint Reconstruction of Hands and Manipulated Objects

H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions

Pushing the Envelope for RGB-Based Dense 3D Hand Pose Estimation via Neural Rendering

Learning to Estimate Pose and Shape of Hand-Held Objects from RGB Images

3D Hand Shape and Pose From Images in the Wild

DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation

Occupancy Networks: Learning 3D Reconstruction in Function Space

Learning Implicit Fields for Generative Shape Modeling

Joint 3D Tracking of a Deformable Object in Interaction with a Hand

SHPR-Net: Deep Semantic Hand Pose Regression From Point Clouds

Augmented Skeleton Space Transfer for Depth-Based Hand Pose Estimation

Hand Pose Estimation via Latent 2.5D Heatmap Regression

Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images

DeepIM: Deep Iterative Matching for 6D Pose Estimation

A Papier-Mache Approach to Learning 3D Surface Generation

Using a Single RGB Frame for Real Time 3D Hand Pose Estimation in the Wild

GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB

Integral Human Pose Regression

V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map

PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes

Learning to Estimate 3D Hand Pose from Single RGB Images

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose

OctNet: Learning Deep 3D Representations at High Resolutions

Real-Time Joint Tracking of a Hand Manipulating an Object from RGB-D Input

3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction

Deep Residual Learning for Image Recognition

3D Object Reconstruction from Hand-Object Interactions

Depth-Based Hand Pose Estimation: Methods, Data, and Challenges

Adam: A Method for Stochastic Optimization

Video-based hand manipulation capture through composite motion control

Motion Capture of Hands in Action Using Discriminative Salient Points

Full DOF tracking of a hand interacting with an object by modeling occlusions and physical constraints

An object-dependent hand pose prior from sparse training data

Hands in action: real-time 3D reconstruction of hands in interaction with objects

Towards 3D hand tracking using a deformable model

Active Shape Models-Their Training and Application

Visual Tracking of High DOF Articulated Structures: an Application to Human Hand Tracking

Marching cubes: A high resolution 3D surface construction algorithm

Machine Perception of Three-Dimensional Solids

HTML: A Parametric Hand Texture Model for 3D Hand Reconstruction and Personalization

Supplemental Material for: FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images

Embodied Hands : Modeling and Capturing Hands and Bodies Together * * Supplementary Material * *