NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Space (2023-09-26T00:00:00.000000Z)

TL;DR

A novel Normalized Device Coordinates scene completion network (NDC-Scene) is devised that directly extends the 2D feature map to a Normalized device coordinates space, rather than to the world space directly, through progressive restoration of the dimension of depth with deconvolution operations.

Abstract

Monocular 3D Semantic Scene Completion (SSC) has garnered significant attention in recent years due to its potential to predict complex semantics and geometry shapes from a single image, requiring no 3D inputs. In this paper, we identify several critical issues in current state-of-the-art methods, including the Feature Ambiguity of projected 2D features in the ray to the 3D space, the Pose Ambiguity of the 3D convolution, and the Computation Imbalance in the 3D convolution across different depth levels. To address these problems, we devise a novel Normalized Device Coordinates scene completion network (NDC-Scene) that directly extends the 2D feature map to a Normalized Device Coordinates (NDC) space, rather than to the world space directly, through progressive restoration of the dimension of depth with deconvolution operations. Experiment results demonstrate that transferring the majority of computation from the target 3D space to the proposed normalized device coordinates space benefits monocular SSC tasks. Additionally, we design a Depth-Adaptive Dual Decoder to simultaneously upsample and fuse the 2D and 3D feature maps, further improving overall performance. Our extensive experiments confirm that the proposed method consistently outperforms state-of-the-art methods on both outdoor SemanticKITTI and indoor NYUv2 datasets. Our code are available at https://github.com/Jiawei-Yao0812/NDCScene.

Authors

Hongsheng Li

5 papers

Jiawei Yao

2 papers

Chuming Li

1 papers

TL;DR

Abstract

Authors

References52 items

VoxFormer: Sparse Voxel Transformer for Camera-Based 3D Semantic Scene Completion

CGOF++: Controllable 3D Face Synthesis With Conditional Generative Occupancy Fields

PaLM: Scaling Language Modeling with Pathways

MonoScene: Monocular 3D Semantic Scene Completion

Panoptic 3D Scene Reconstruction From a Single RGB Image

Inverting Generative Adversarial Renderer for Face Reconstruction

Semantic Scene Completion via Integrating Instances and Scene in-the-Loop

3D Semantic Scene Completion: A Survey

S3CNet: A Sparse Semantic Scene Completion Network for LiDAR Point Clouds

Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion

AdaBins: Depth Estimation Using Adaptive Bins

Semantic Scene Completion Using Local Deep Implicit Functions on LiDAR Data

LMSCNet: Lightweight Multiscale 3D Semantic Completion

Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild

Pix2Vox++: Multi-scale Context-aware 3D Object Reconstruction from Single and Multiple Images

Anisotropic Convolutional Networks for 3D Semantic Scene Completion

Context Prior for Scene Segmentation

3D Sketch-Aware Semantic Scene Completion via Semi-Supervised Structure Prior

Detection in Crowded Scenes: One Proposal, Multiple Predictions

Convolutional Occupancy Networks

Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes From a Single Image

BSP-Net: Generating Compact Meshes via Binary Space Partitioning

Multi-View PointNet for 3D Scene Understanding

Location Field Descriptors: Single Image 3D Model Retrieval in the Wild

Cascaded Context Pyramid for Full-Resolution 3D Semantic Scene Completion

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences

RGBD Based Dimensional Decomposition Residual Network for 3D Semantic Scene Completion

DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation

HorizonNet: Learning Room Layout With 1D Representation and Pano Stretch Data Augmentation

High Quality Monocular Depth Estimation via Transfer Learning

Occupancy Networks: Learning 3D Reconstruction in Function Space

Efficient Semantic Scene Completion Network with Spatial Group Convolution

Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image

Deep Marching Cubes: Learning Explicit Surface Representations

Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images

Focal Loss for Dense Object Detection

Learning Representations and Generative Models for 3D Point Clouds

RoomNet: End-to-End Room Layout Estimation

A Point Set Generation Network for 3D Object Reconstruction from a Single Image

Semantic Scene Completion from a Single Depth Image

Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

IM2CAD

3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction

Learning a Predictable and Generative Vector Representation for Objects

Deep Residual Learning for Image Recognition

U-Net: Convolutional Networks for Biomedical Image Segmentation

Indoor Segmentation and Support Inference from RGBD Images

Monocular 3D scene understanding with explicit occlusion reasoning

Mesh R-CNN

See and Think: Disentangling Semantic Scene Completion

9430 the applicable license agreement with IEEE. Restrictions apply

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names