VoxFormer: Sparse Voxel Transformer for Camera-Based 3D Semantic Scene Completion (2023-02-23T00:00:00.000000Z)

TL;DR

VoxFormer, a Transformer-based semantic scene completion framework that can output complete 3D volumetric semantics from only 2D images that outperforms the state of the art with a relative improvement of 20.0% in geometry and 18.1% in semantics.

Abstract

Humans can easily imagine the complete 3D geometry of occluded objects and scenes. This appealing ability is vital for recognition and understanding. To enable such capability in AI systems, we propose VoxFormer, a Transformer-based semantic scene completion framework that can output complete 3D volumetric semantics from only 2D images. Our framework adopts a two-stage design where we start from a sparse set of visible and occupied voxel queries from depth estimation, followed by a densification stage that generates dense 3D voxels from the sparse ones. A key idea of this design is that the visual features on 2D images correspond only to the visible scene structures rather than the occluded or empty spaces. Therefore, starting with the fea-turization and prediction of the visible structures is more reliable. Once we obtain the set of sparse queries, we apply a masked autoencoder design to propagate the information to all the voxels by self-attention. Experiments on SemanticKITTI show that VoxFormer outperforms the state of the art with a relative improvement of 20.0% in geometry and 18.1% in semantics and reduces GPU memory during training to less than 16GB. Our code is available on https://github.com/NV1abs/VoxFormer.

Authors

Chaowei Xiao

4 papers

S. Fidler

22 papers

J. Álvarez

8 papers

TL;DR

Abstract

Authors

References77 items

Uncertainty Quantification of Collaborative Detection for Self-Driving

CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers

Make it Dense: Self-Supervised Geometric Scan Completion of Sparse 3D LiDAR Scans in Large Outdoor Environments

Revealing the Dark Secrets of Masked Image Modeling

M2BEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representation

BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers

Rethinking Semantic Segmentation: A Prototype View

V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer

V2X-Sim: Multi-Agent Collaborative Perception Dataset and Benchmark for Autonomous Driving

ShapeFormer: Transformer-based Shape Completion via Sparse Representation

NICE-SLAM: Neural Implicit Scalable Encoding for SLAM

MonoScene: Monocular 3D Semantic Scene Completion

Semi-supervised Implicit Scene Completion from Sparse LiDAR

Masked Autoencoders Are Scalable Vision Learners

Learning Distilled Collaboration Graph for Multi-Agent Perception

DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries

Voxel-based Network for Shape Completion by Leveraging Edge Generation

MobileStereoNet: Towards Lightweight Deep Networks for Stereo Matching

Per-Pixel Classification is Not All You Need for Semantic Segmentation

SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers

3D Shape Generation and Completion through Point-Voxel Diffusion

Semantic Scene Completion via Integrating Instances and Scene in-the-Loop

3D Semantic Scene Completion: A Survey

S3CNet: A Sparse Semantic Scene Completion Network for LiDAR Point Clouds

Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion

AdaBins: Depth Estimation Using Adaptive Bins

Semantic Scene Completion Using Local Deep Implicit Functions on LiDAR Data

Deformable DETR: Deformable Transformers for End-to-End Object Detection

LMSCNet: Lightweight Multiscale 3D Semantic Completion

Weakly-supervised 3D Shape Completion in the Wild

V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Prediction

HITNet: Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matching

A Survey on Deep Learning Techniques for Stereo-Based Depth Estimation

End-to-End Object Detection with Transformers

CoReNet: Coherent 3D scene reconstruction from a single RGB image

Anisotropic Convolutional Networks for 3D Semantic Scene Completion

3D Sketch-Aware Semantic Scene Completion via Semi-Supervised Structure Prior

Convolutional Occupancy Networks

Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion

Image Segmentation Using Deep Learning: A Survey

Depth Based Semantic Scene Completion With Position Importance Aware Loss

Deep Learning for 3D Point Clouds: A Survey

SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans

RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds

PANet: Few-Shot Image Semantic Segmentation With Prototype Alignment

Cascaded Context Pyramid for Full-Resolution 3D Semantic Scene Completion

Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era

DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction

SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences

RGBD Based Dimensional Decomposition Residual Network for 3D Semantic Scene Completion

Occupancy Networks: Learning 3D Reconstruction in Function Space

Efficient Semantic Scene Completion Network with Spatial Group Convolution

PCN: Point Completion Network

Attention is All you Need

Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency

Feature Pyramid Networks for Object Detection

A Point Set Generation Network for 3D Object Reconstruction from a Single Image

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Shape Completion Using 3D-Encoder-Predictor CNNs and Shape Synthesis

Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision

Semantic Scene Completion from a Single Depth Image

3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction

Deep Residual Learning for Image Recognition

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Learning Deconvolution Network for Semantic Segmentation

Fully convolutional networks for semantic segmentation

Perceptual Organization and Recognition of Indoor Scenes from RGB-D Images

Are we ready for autonomous driving? The KITTI vision benchmark suite

KinectFusion: Real-time dense surface mapping and tracking

Shape from symmetry

Multi-Robot Scene Completion: Towards Task-Agnostic Collaborative Perception

Weakly supervised semantic point cloud segmentation: Towards 10x fewer labels

See and Think: Disentangling Semantic Scene Completion

DeMoN: Depth and Motion Network for Learning Monocular Stereo

Multiple View Geometry in Computer Vision

[ 9098 applicable license agreement with IEEE. Restrictions apply