ROCA: Robust CAD Model Retrieval and Alignment from a Single Image (2021-12-03T00:00:00.000000Z)

TL;DR

Experiments on challenging, real-world imagery from ScanNet show that ROCA signif-icantly improves on state of the art, from 9.5% to 17.6% in retrieval-aware CAD alignment accuracy.

Abstract

We present ROCA 11The code is made available at https://github.com/cangurneli/ROCA., a novel end-to-end approach that re-trieves and aligns 3D CAD models from a shape database to a single input image. This enables 3D perception of an ob-served scene from a 2D RGB observation, characterized as a lightweight, compact, clean CAD representation. Core to our approach is our differentiable alignment optimization based on dense 2D-3D object correspondences and Pro-crustes alignment. ROCA can thus provide a robust CAD alignment while simultaneously informing CAD retrieval by leveraging the 2D-3D correspondences to learn geometri-cally similar CAD models. Experiments on challenging, real-world imagery from ScanNet show that ROCA signif-icantly improves on state of the art, from 9.5% to 17.6% in retrieval-aware CAD alignment accuracy.

Authors

Angela Dai

11 papers

M. Nießner

31 papers

Can Gümeli

1 papers

TL;DR

Abstract

Authors

References54 items

Voxel-based 3D Detection and Reconstruction of Multiple Objects from a Single Image

Patch2CAD: Patchwise Embedding Learning for In-the-Wild Shape Retrieval from a Single Image

From Points to Multi-Object 3D Reconstruction

Vid2CAD: CAD Model Alignment Using Multi-View Constraints From Videos

3D-FUTURE: 3D Furniture Shape with TextURE

Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve

CAD-Deform: Deformable Fitting of CAD Models to 3D Scans

Deep Global Registration

Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes From a Single Image

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Accelerating 3D deep learning with PyTorch3D

End-to-End Learnable Geometric Vision by Backpropagating PnP Optimization

Joint Embedding of 3D Scan and CAD Objects

End-to-End CAD Model Retrieval and 9DoF Alignment in 3D Scans

3D-RelNet: Joint Object and Relational Network for 3D Prediction

CenterNet: Keypoint Triplets for Object Detection

ShapeMask: Learning to Segment Novel Objects by Refining Shape Priors

Fast Graph Representation Learning with PyTorch Geometric

DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation

Occupancy Networks: Learning 3D Reconstruction in Function Space

Scan2CAD: Learning CAD Model Alignment in RGB-D Scans

Scan2Mesh: From Unstructured Range Scans to 3D Meshes

Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image

Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images

Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps With Accurate Object Boundaries

A Papier-Mache Approach to Learning 3D Surface Generation

Open3D: A Modern Library for 3D Data Processing

Focal Loss for Dense Object Detection

Focal Loss for Dense Object Detection

Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs

Mask R-CNN

ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes

Feature Pyramid Networks for Object Detection

A Point Set Generation Network for 3D Object Reconstruction from a Single Image

IM2CAD

Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network

Deeper Depth Prediction with Fully Convolutional Residual Networks

3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction

Deep Residual Learning for Image Recognition

ShapeNet: An Information-Rich 3D Model Repository

ImageNet Large Scale Visual Recognition Challenge

3D ShapeNets: A deep representation for volumetric shapes

Depth Map Prediction from a Single Image using a Multi-Scale Deep Network

Microsoft COCO: Common Objects in Context

Model-based recognition in robot vision

Survey of Model-Based Image Analysis Systems

Machine Perception of Three-Dimensional Solids

non-rigid tracking

Mesh R-CNN

Detectron2. https://github. 10 com/facebookresearch/detectron2, 2019

Single Image

A Deep Representation for Volumetric Shapes

Vid 2 cad : Cad model alignment using multiview constraints from videos Occupancy networks : Learning 3 d reconstruction in function space

Rasterization: a practical implementation

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names