Clear Grasp: 3D Shape Estimation of Transparent Objects for Manipulation (2019-10-06T00:00:00.000000Z)

TL;DR

ClearGrasp is substantially better than monocular depth estimation baselines and is capable of generalizing to real-world images and novel objects and can be applied out-of-the-box to improve grasping algorithms’ performance on transparent objects.

Abstract

Transparent objects are a common part of everyday life, yet they possess unique visual properties that make them incredibly difficult for standard 3D sensors to produce accurate depth estimates for. In many cases, they often appear as noisy or distorted approximations of the surfaces that lie behind them. To address these challenges, we present ClearGrasp – a deep learning approach for estimating accurate 3D geometry of transparent objects from a single RGB-D image for robotic manipulation. Given a single RGB-D image of transparent objects, ClearGrasp uses deep convolutional networks to infer surface normals, masks of transparent surfaces, and occlusion boundaries. It then uses these outputs to refine the initial depth estimates for all transparent surfaces in the scene. To train and test ClearGrasp, we construct a large-scale synthetic dataset of over 50,000 RGB-D images, as well as a real-world test benchmark with 286 RGB-D images of transparent objects and their ground truth geometries. The experiments demonstrate that ClearGrasp is substantially better than monocular depth estimation baselines and is capable of generalizing to real-world images and novel objects. We also demonstrate that ClearGrasp can be applied out-of-the-box to improve grasping algorithms’ performance on transparent objects. Code, data, and benchmarks will be released. Supplementary materials: https://sites.google.com/view/cleargrasp1

Authors

Andy Zeng

10 papers

Shuran Song

13 papers

Johnny Lee

4 papers

TL;DR

Abstract

Authors

References64 items

Learning to See before Learning to Act: Visual Pre-training for Manipulation

Grasping in the Wild: Learning 6DoF Closed-Loop Grasping From Low-Cost Demonstrations

Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly

The Replica Dataset: A Digital Replica of Indoor Spaces

Material-Based Segmentation of Objects

DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactions

SharpNet: Fast and Accurate Recovery of Occluding Contours in Monocular Depth Estimation

Deep Surface Normal Estimation With Hierarchical RGB-D Fusion

TossingBot: Learning to Throw Arbitrary Objects With Residual Physics

FastDepth: Fast Monocular Depth Estimation on Embedded Systems

Transparent object detection and location based on RGB-D camera

Single-Shot Analysis of Refractive Shape Using Convolutional Neural Networks

High Quality Monocular Depth Estimation via Transfer Learning

Depth Reconstruction of Translucent Objects from a Single Time-of-Flight Camera using Deep Residual Networks

Deep Normal Estimation for Automatic Shading of Hand-Drawn Characters

Deep Ordinal Regression Network for Monocular Depth Estimation

Soccer on Your Tabletop

Transparent Object Detection Using Convolutional Neural Network

Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning

Deep Depth Completion of a Single RGB-D Image

Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps With Accurate Object Boundaries

TOM-Net: Learning Transparent Object Matting from a Single Image

Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation

Im2Pano3D: Extrapolating 360° Structure and Semantics Beyond the Field of View

Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching

Matterport3D: Learning from RGB-D Data in Indoor Environments

Face Normals "In-the-Wild" Using Fully Convolutional Networks

Dilated Residual Networks

Hand3D: Hand Pose Estimation using 3D Neural Network

Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics

ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes

Friend or foe: exploiting sensor failures for transparent object localization and classification

Looking beyond appearances: Synthetic training data for deep CNNs in re-identification

Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks

Semantic Scene Completion from a Single Depth Image

3D Reconstruction of Transparent Objects with Position-Normal Consistency

Seeing Glassware: from Edge Detection to Pose Estimation and Shape Recovery

Deeper Depth Prediction with Fully Convolutional Residual Networks

Single-Image Depth Perception in the Wild

Marr Revisited: 2D-3D Alignment via Surface Normal Prediction

Deep Residual Learning for Image Recognition

A fixed viewpoint approach for dense reconstruction of transparent objects

Designing deep networks for surface normal estimation

Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture

Depth Map Prediction from a Single Image using a Multi-Scale Deep Network

Microsoft COCO: Common Objects in Context

Guided depth enhancement via a fast marching method

Depth Map Inpainting under a Second-Order Smoothness Prior

Pose estimation of rigid transparent objects in transparent clutter

Glass object localization by joint inference of boundary and depth

Recognition and Pose Estimation of Rigid Transparent Objects with a Kinect Sensor

A novel stereoscopic cue for figure-ground segregation of semi-transparent objects

Transparent object detection and reconstruction on a mobile platform

Rectified Linear Units Improve Restricted Boltzmann Machines

An Additive Latent Feature Model for Transparent Object Recognition

A Geodesic Active Contour Framework for Finding Glass

Finding glass

Online; accessed 25-Sept-2019

Learning Visual Affordances for Robotic Manipulation

Pytorch re-implementation of deeplabv3+

Fusing Depth and Silhouette for Scanning Transparent Object with RGB-D Sensor

Using Geometry to Detect Grasp Poses in 3D Point Clouds

TRANSPARENT OBJECT DETECTION USING REGIONS WITH CONVOLUTIONAL NEURAL NETWORK

Seeing the Unseen: Simple Reconstruction of Transparent Objects from Point Cloud Data

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names