SCFusion: Real-time Incremental Scene Reconstruction with Semantic Completion

Published in

International Conference on 3D Vision(2020)

External Links:

Generate Graph DownloadPDF

TL;DR

This work proposes a framework that ameliorates scene reconstruction and semantic scene completion jointly in an incremental and real-time manner, based on an input sequence of depth maps and relies on a novel neural architecture designed to process occupancy maps and leverages voxel states to accurately and efficiently fuse semantic completion with the 3D global model.

Abstract

Real-time scene reconstruction from depth data inevitably suffers from occlusion, thus leading to incomplete 3D models. Partial reconstructions, in turn, limit the performance of algorithms that leverage them for applications in the context of, e.g., augmented reality, robotic navigation, and 3D mapping. Most methods address this issue by predicting the missing geometry as an offline optimization, thus being incompatible with real-time applications. We propose a framework that ameliorates this issue by performing scene reconstruction and semantic scene completion jointly in an incremental and real-time manner, based on an input sequence of depth maps. Our framework relies on a novel neural architecture designed to process occupancy maps and leverages voxel states to accurately and efficiently fuse semantic completion with the 3D global model. We evaluate the proposed approach quantitatively and qualitatively, demonstrating that our method can obtain accurate 3D semantic scene completion in real-time.

Authors

N. Navab

26 papers

Federico Tombari

15 papers

Shun-cheng Wu

3 papers

References54 items

SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans

ForkNet: Multi-Branch Volumetric Semantic Completion From a Single Depth Image

RIO: 3D Object Instance Re-Localization in Changing Indoor Environments

VITAMIN-E: VIsual Tracking and MappINg With Extremely Dense Feature Points

PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things

SCFusion: Real-time Incremental Scene Reconstruction with Semantic Completion

Published in

International Conference on 3D Vision(2020)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

N. Navab

26 papers

Federico Tombari

15 papers

Shun-cheng Wu

3 papers

References54 items

SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans

ForkNet: Multi-Branch Volumetric Semantic Completion From a Single Depth Image

RIO: 3D Object Instance Re-Localization in Changing Indoor Environments

VITAMIN-E: VIsual Tracking and MappINg With Extremely Dense Feature Points

PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things

Keisuke Tateno

2 papers

TL;DR

Abstract

Authors

References54 items

SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans

ForkNet: Multi-Branch Volumetric Semantic Completion From a Single Depth Image

RIO: 3D Object Instance Re-Localization in Changing Indoor Environments

VITAMIN-E: VIsual Tracking and MappINg With Extremely Dense Feature Points

PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things

TL;DR

Abstract

Authors

References54 items

SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans

ForkNet: Multi-Branch Volumetric Semantic Completion From a Single Depth Image

RIO: 3D Object Instance Re-Localization in Changing Indoor Environments

VITAMIN-E: VIsual Tracking and MappINg With Extremely Dense Feature Points

PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things

RGBD Based Dimensional Decomposition Residual Network for 3D Semantic Scene Completion

Scan2CAD: Learning CAD Model Alignment in RGB-D Scans

Learning Priors for Semantic 3D Reconstruction

InteriorNet: Mega-scale Multi-sensor Photo-realistic Indoor Scenes Dataset

Adversarial Semantic Scene Completion from a Single Depth Image

View-Volume Network for Semantic Scene Completion from a Single Depth Image

Free-Form Image Inpainting With Gated Convolution

Image Inpainting for Irregular Holes Using Partial Convolutions

Spectral Normalization for Generative Adversarial Networks

Efficient Octree-Based Volumetric SLAM Supporting Signed-Distance and Occupancy Mapping

ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans

Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?

Matterport3D: Learning from RGB-D Data in Indoor Environments

Dense Semantic 3D Reconstruction

3D Object Reconstruction from a Single Depth View with Adversarial Learning

ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes

Semantic Scene Completion from a Single Depth Image

A Symmetry Prior for Convex Variational 3D Reconstruction

A Closed-Form Bayesian Fusion Equation Using Occupancy Probabilities

SemanticFusion: Dense 3D semantic mapping with convolutional neural networks

Structured Prediction of Unobserved Voxels from a Single Depth Image

BundleFusion

ElasticFusion: Dense SLAM Without A Pose Graph

Real-time and scalable incremental segmentation on dense SLAM

Very High Frame Rate Volumetric Integration of Depth Images on Mobile Devices

3D Modeling on the Go: Interactive 3D Reconstruction of Large-Scale Scenes on Mobile Devices

Segment based 3D object shape priors

Mask-Specific Inpainting with Deep Neural Networks

Class Specific 3D Object Shape Priors Using Surface Normals

3D ShapeNets: A deep representation for volumetric shapes

3D Scene Understanding by Voxel-CRF

Real-time 3D reconstruction at scale using voxel hashing

Real-Time 3D Reconstruction in Dynamic Scenes Using Point-Based Fusion

Joint 3D Scene Reconstruction and Class Segmentation

OctoMap: an efficient probabilistic 3D mapping framework based on octrees

A search-classify approach for cluttered indoor scene understanding

Acquiring 3D indoor environments with variability and repetition

Indoor Segmentation and Support Inference from RGBD Images

DTAM: Dense tracking and mapping in real-time

KinectFusion: Real-time dense surface mapping and tracking

Joint Optimization for Object Class Segmentation and Dense Stereo Reconstruction

A robust hole-filling algorithm for triangular mesh

Laplacian mesh optimization

Partial and approximate symmetry detection for 3D geometry

Shape from symmetry

Least-squares meshes

High resolution maps from wide angle sonar

European

Derek Hoiem and R

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names