Automatically Annotating Indoor Images with CAD Models via RGB-D Scans (2022-12-22T00:00:00.000000Z)

TL;DR

An automatic method for annotating images of indoor scenes with the CAD models of the objects by relying on RGB-D scans is presented, and a ’cloning procedure’ is introduced that identifies objects that have the same geometry, to annotate these objects with the same CAD models.

Abstract

We present an automatic method for annotating images of indoor scenes with the CAD models of the objects by relying on RGB-D scans. Through a visual evaluation by 3D experts, we show that our method retrieves annotations that are at least as accurate as manual annotations, and can thus be used as ground truth without the burden of manually annotating 3D data. We do this using an analysis-by-synthesis approach, which compares renderings of the CAD models with the captured scene. We introduce a ’cloning procedure’ that identifies objects that have the same geometry, to annotate these objects with the same CAD models. This allows us to obtain complete annotations for the Scan-Net dataset and the recent ARKitScenes dataset. We will release these annotations publicly, as we believe they will be very useful for the computer vision community.

Authors

F. Fraundorfer

5 papers

V. Lepetit

10 papers

Stefan Ainetter

2 papers

TL;DR

Abstract

Authors

References28 items

ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data

Monte Carlo Scene Search for 3D Scene Understanding

Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding

Accelerating 3D deep learning with PyTorch3D

Joint Embedding of 3D Scan and CAD Objects

End-to-End CAD Model Retrieval and 9DoF Alignment in 3D Scans

Scan2CAD: Learning CAD Model Alignment in RGB-D Scans

Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image

3D-RCNN: Instance-Level 3D Object Reconstruction via Render-and-Compare

Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling

DeepIM: Deep Iterative Matching for 6D Pose Estimation

3D Pose Estimation and 3D Model Retrieval for Objects in the Wild

Complete 3D Scene Parsing from an RGBD Image

Matterport3D: Learning from RGB-D Data in Indoor Environments

ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes

SceneNN: A Scene Meshes Dataset with aNNotations

ShapeNet: An Information-Rich 3D Model Repository

Adam: A Method for Stochastic Optimization

Analysis by Synthesis: 3D Object Recognition by Object Reconstruction

Scene Collaging: Analysis and Synthesis of Natural Images with Semantic Layers

Estimating Spatial Layout of Rooms using Volumetric Reasoning about Objects and Surfaces

, Friedrich Fraundorfer , and Vincent Lepetit . Monte Carlo Scene Search for 3 D Scene Understanding Analysis by Synthesis : 3 D Object Recognition by Object Reconstruction

Opinion TRENDS in Cognitive Sciences Vol.10 No.7 July 2006 Special Issue: Probabilistic models of cognition Vision as Bayesian inference: analysis by synthesis?

We show how to identify the objects that share the same geometry, and retrieve a common CAD model for these objects—we call this procedure ’cloning’

Chang , and Matthias Nießner . Scan 2 cad : Learning CAD Model Alignment in RGB - D Scans End - To - End CAD Model Retrieval and 9 dof Alignment in 3 D Scans

As no supervision is needed

Our method can be seen as a tool to automatically annotate images with CAD models and poses for the objects in the images, and we will make our annotations for

We show that the quality of our results is comparable or better than human annotations

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names