OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction

Published in

IEEE International Conference on Computer Vision(2023)

External Links:

Generate Graph DownloadPDF

TL;DR

OccFormer, a dual-path transformer network to effectively process the 3D volume for semantic occupancy prediction, achieves a long-range, dynamic, and efficient encoding of the camera-generated 3D voxel features.

Abstract

The vision-based perception for autonomous driving has undergone a transformation from the bird-eye-view (BEV) representations to the 3D semantic occupancy. Compared with the BEV planes, the 3D semantic occupancy further provides structural information along the vertical direction. This paper presents OccFormer, a dual-path transformer network to effectively process the 3D volume for semantic occupancy prediction. OccFormer achieves a long-range, dynamic, and efficient encoding of the camera-generated 3D voxel features. It is obtained by decomposing the heavy 3D processing into the local and global transformer pathways along the horizontal plane. For the occupancy decoder, we adapt the vanilla Mask2Former for 3D semantic occupancy by proposing preserve-pooling and class-guided sampling, which notably mitigate the sparsity and class imbalance. Experimental results demonstrate that OccFormer significantly outperforms existing methods for semantic scene completion on SemanticKITTI dataset and for LiDAR semantic segmentation on nuScenes dataset. Code is available at https://github.com/zhangyp15/OccFormer.

Authors

Dalong Du

4 papers

Yunpeng Zhang

3 papers

Zhengbiao Zhu

1 papers

References65 items

Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction

LidarMultiNet: Towards a Unified Multi-task Network for LiDAR Perception

BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection

BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving

Cross-view Transformers for real-time Map-view Semantic Segmentation

OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction

Published in

IEEE International Conference on Computer Vision(2023)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Dalong Du

4 papers

Yunpeng Zhang

3 papers

Zhengbiao Zhu

1 papers

References65 items

Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction

LidarMultiNet: Towards a Unified Multi-task Network for LiDAR Perception

BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection

BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving

Cross-view Transformers for real-time Map-view Semantic Segmentation

BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers

Exploring Plain Vision Transformer Backbones for Object Detection

StretchBEV: Stretching Future Instance Prediction Spatially and Temporally

PETR: Position Embedding Transformation for Multi-View 3D Object Detection

BEVSegFormer: Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigs

ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond

BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View

MPViT: Multi-Path Vision Transformer for Dense Prediction

Masked-attention Mask Transformer for Universal Image Segmentation

MonoScene: Monocular 3D Semantic Scene Completion

DRINet++: Efficient Voxel-as-point Point Cloud Segmentation

Panoptic nuScenes: A Large-Scale Benchmark for LiDAR Panoptic Segmentation and Tracking

PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers

Per-Pixel Classification is Not All You Need for Semantic Segmentation

Video Swin Transformer

ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias

FIERY: Future Instance Prediction in Bird’s-Eye View from Surround Monocular Cameras

Categorical Depth Distribution Network for Monocular 3D Object Detection

(AF)2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Network

Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

S3CNet: A Sparse Semantic Scene Completion Network for LiDAR Point Clouds

AMVNet: Assertion-based Multi-View Fusion Network for LiDAR Semantic Segmentation

Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion

Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation

Semantic Scene Completion Using Local Deep Implicit Functions on LiDAR Data

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Deformable DETR: Deformable Transformers for End-to-End Object Detection

LMSCNet: Lightweight Multiscale 3D Semantic Completion

Multi-Scale Interaction for Real-Time LiDAR Data Segmentation on an Embedded Platform

Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D

Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution

Center-based 3D Object Detection and Tracking

End-to-End Object Detection with Transformers

Anisotropic Convolutional Networks for 3D Semantic Scene Completion

PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentation

3D Sketch-Aware Semantic Scene Completion via Semi-Supervised Structure Prior

Predicting Semantic Map Representations From Images Using Pyramid Occupancy Networks

SalsaNext: Fast, Uncertainty-Aware Semantic Segmentation of LiDAR Point Clouds

PointRend: Image Segmentation As Rendering

Scalability in Perception for Autonomous Driving: Waymo Open Dataset

RangeNet ++: Fast and Accurate LiDAR Semantic Segmentation

SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences

nuScenes: A Multimodal Dataset for Autonomous Driving

RGBD Based Dimensional Decomposition Residual Network for 3D Semantic Scene Completion

PointPillars: Fast Encoders for Object Detection From Point Clouds

PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud

Video Action Transformer Network

Efficient Semantic Scene Completion Network with Spatial Group Convolution

Frustum PointNets for 3D Object Detection from RGB-D Data

Decoupled Weight Decay Regularization

Attention is All you Need

Feature Pyramid Networks for Object Detection

Semantic Scene Completion from a Single Depth Image

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

Deep Residual Learning for Image Recognition

Are we ready for autonomous driving? The KITTI vision benchmark suite

The Hungarian method for the assignment problem

PolarStream: Streaming Object Detection and Segmentation with Polar Pillars

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

agreement with IEEE. Restrictions apply

Field of Study

Computer Science

Journal Information

Name

2023 IEEE/CVF International Conference on Computer Vision (ICCV)

Page

20313-20325

Venue Information

Name

IEEE International Conference on Computer Vision

Type

conference

URL

https://ieeexplore.ieee.org/xpl/conhome/1000149/all-proceedings

Alternate Names

ICCV
IEEE Int Conf Comput Vis
ICCV Workshops
ICCV Work