Neural RGB-D Surface Reconstruction (2021-04-09T00:00:00.000000Z)

TL;DR

This work proposes to represent the surface using an implicit function (truncated signed distance function), and shows how to incorporate this representation in the NeRF framework, and extend it to use depth measurements from a commodity RGB-D sensor, such as a Kinect.

Abstract

Obtaining high-quality 3D reconstructions of room-scale scenes is of paramount importance for upcoming applications in AR or VR. These range from mixed reality applications for teleconferencing, virtual measuring, virtual room planing, to robotic applications. While current volume-based view synthesis methods that use neural radiance fields (NeRFs) show promising results in reproducing the appearance of an object or scene, they do not reconstruct an actual surface. The volumetric representation of the surface based on densities leads to artifacts when a surface is extracted using Marching Cubes, since during optimization, densities are accumulated along the ray and are not used at a single sample point in isolation. Instead of this volumetric representation of the surface, we propose to represent the surface using an implicit function (truncated signed distance function). We show how to incorporate this representation in the NeRF framework, and extend it to use depth measurements from a commodity RGB-D sensor, such as a Kinect. In addition, we propose a pose and camera re-finement technique which improves the overall reconstruction quality. In contrast to concurrent work on integrating depth priors in NeRF which concentrates on novel view synthesis, our approach is able to reconstruct high-quality, metrical 3D reconstructions.

Authors

M. Nießner

31 papers

Ricardo Martin-Brualla

3 papers

Dan B. Goldman

5 papers

TL;DR

Abstract

Authors

References96 items

Plenoxels: Radiance Fields without Neural Networks

Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction

Panoptic 3D Scene Reconstruction From a Single RGB Image

NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo

Depth-supervised NeRF: Fewer Views and Faster Training for Free

TransformerFusion: Monocular RGB Scene Reconstruction using Transformers

Volume Rendering of Neural Implicit Surfaces

NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction

UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction

BARF: Bundle-Adjusting Neural Radiance Fields

NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video

DONeRF: Towards Real‐Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks

IBRNet: Learning Multi-View Image-Based Rendering

iNeRF: Inverting Neural Radiance Fields for Pose Estimation

Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction

pixelNeRF: Neural Radiance Fields from One or Few Images

pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis

NeuralFusion: Online Depth Fusion in Latent Space

Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes

A Class of C2 Interpolating Splines

NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections

GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis

SPSG: Self-Supervised Photometric Scene Generation from RGB-D Scans

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains

Implicit Neural Representations with Periodic Activation Functions

State of the Art on Neural Rendering

PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization

Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance

Convolutional Occupancy Networks

Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion

Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes From a Single Image

RoutedFusion: Learning Real-Time Depth Map Fusion

DeepFactors: Real-Time Probabilistic Dense Monocular SLAM

Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision

SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans

Neural volumes

Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations

Texture Fields: Learning Texture Representations in Function Space

PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization

Deferred neural rendering

SceneCode: Monocular Dense Semantic Reconstruction Using Learned Encoded Scene Representations

DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation

Occupancy Networks: Learning 3D Reconstruction in Function Space

DeepVoxels: Learning Persistent 3D Feature Embeddings

PSDF Fusion: Probabilistic Signed Distance Function for On-the-fly 3D Data Fusion and Scene Reconstruction

Deep Ordinal Regression Network for Monocular Depth Estimation

State of the Art on 3D Reconstruction with RGB‐D Cameras

MVSNet: Depth Inference for Unstructured Multi-view Stereo

Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images

CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM

Deep Depth Completion of a Single RGB-D Image

Matterport3D: Learning from RGB-D Data in Indoor Environments

Learning a Multi-View Stereo Machine

ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes

Image-to-Image Translation with Conditional Adversarial Networks

A Vote-and-Verify Strategy for Fast Spatial Verification in Image Retrieval

Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Pixelwise View Selection for Unstructured Multi-View Stereo

Unsupervised Monocular Depth Estimation with Left-Right Consistency

Structure-from-Motion Revisited

Deeper Depth Prediction with Fully Convolutional Residual Networks

Semantic 3D Reconstruction with Continuous Regularization and Ray Potentials Using a Visibility Consistency Constraint

BundleFusion

ShapeNet: An Information-Rich 3D Model Repository

Multi-view 3D Models from Single Images with a Convolutional Network

Deep Stereo: Learning to Predict New Views from the World's Imagery

Adam: A Method for Stochastic Optimization

A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM

SVO: Fast semi-direct monocular visual odometry

Robot arm pose estimation through pixel-wise part classification

LSD-SLAM: Large-Scale Direct Monocular SLAM

Color map optimization for 3D reconstruction with consumer depth cameras

Auto-Encoding Variational Bayes

Semi-dense Visual Odometry for a Monocular Camera

Real-time 3D reconstruction at scale using voxel hashing

Intrinsic Scene Properties from a Single RGB-D Image