Virtual Multi-view Fusion for 3D Semantic Segmentation (2020-07-26T00:00:00.000000Z)

TL;DR

This paper revisits the classic multiview representation of 3D meshes and study several techniques that make them effective for 3D semantic segmentation of meshes and shows that the virtual views enable more effective training of 2D semantic segmentsation networks than previous multIView approaches.

Abstract

Semantic segmentation of 3D meshes is an important problem for 3D scene understanding. In this paper we revisit the classic multiview representation of 3D meshes and study several techniques that make them effective for 3D semantic segmentation of meshes. Given a 3D mesh reconstructed from RGBD sensors, our method effectively chooses different virtual views of the 3D mesh and renders multiple 2D channels for training an effective 2D semantic segmentation model. Features from multiple per view predictions are finally fused on 3D mesh vertices to predict mesh semantic segmentation labels. Using the large scale indoor 3D semantic segmentation benchmark of ScanNet, we show that our virtual views enable more effective training of 2D semantic segmentation networks than previous multiview approaches. When the 2D per pixel predictions are aggregated on 3D surfaces, our virtual multiview fusion method is able to achieve significantly better 3D semantic segmentation results compared to all prior multiview approaches and competitive with recent 3D convolution approaches.

Authors

T. Funkhouser

13 papers

C. Pantofaru

5 papers

David A. Ross

3 papers

TL;DR

Abstract

Authors

References49 items

JSENet: Joint Semantic Segmentation and Edge Detection Network for 3D Point Clouds

OccuSeg: Occupancy-Aware 3D Instance Segmentation

PointASNL: Robust Point Clouds Processing Using Nonlocal Neural Networks With Adaptive Sampling

PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection

Fully Convolutional Geometric Features

Multi-View PointNet for 3D Scene Understanding

A Unified Point-Based Framework for 3D Segmentation

Large-Scale 3D Semantic Mapping Using Monocular Vision

4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks

KPConv: Flexible and Deformable Convolution for Point Clouds

JSIS3D: Joint Semantic-Instance Segmentation of 3D Point Clouds With Multi-Task Pointwise Networks and Multi-Value Conditional Random Fields

PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things

TextureNet: Consistent Local Parametrizations for Learning From High-Resolution Signals on Meshes

PointConv: Deep Convolutional Networks on 3D Point Clouds

3D Recurrent Neural Networks with Context Fusion for Point Cloud Semantic Segmentation

InteriorNet: Mega-scale Multi-sensor Photo-realistic Indoor Scenes Dataset

Self-Supervised Model Adaptation for Multimodal Semantic Segmentation

Deep Parametric Continuous Convolutional Neural Networks

Tangent Convolutions for Dense Prediction in 3D

3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation

Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation

PointCNN: Convolution On X-Transformed Points

SnapNet: 3D point cloud semantic labeling with 2D deep segmentation networks

3D Semantic Segmentation with Submanifold Sparse Convolutional Networks

Large-Scale Point Cloud Semantic Segmentation with Superpoint Graphs

SnapNet-R: Consistent 3D Multi-view Semantic Labeling for Robotics

SEGCloud: Semantic Segmentation of 3D Point Clouds

SceneNet RGB-D: Can 5M Synthetic Images Beat Generic ImageNet Pre-training on Indoor Segmentation?

Scene Parsing through ADE20K Dataset

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

Deep Projective 3D Semantic Segmentation

Multi-view deep learning for consistent semantic mapping with RGB-D cameras

ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes

Joint 2D-3D-Semantic Data for Indoor Scene Understanding

Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Semantic Scene Completion from a Single Depth Image

OctNet: Learning Deep 3D Representations at High Resolutions

Xception: Deep Learning with Depthwise Separable Convolutions

SemanticFusion: Dense 3D semantic mapping with convolutional neural networks

3D Semantic Parsing of Large-Scale Indoor Spaces

SemanticPaint

Incremental dense semantic stereo fusion for large-scale semantic scene reconstruction

Multi-view Convolutional Neural Networks for 3D Shape Recognition

Unsupervised feature learning for 3D scene labeling

Dense 3D semantic mapping of indoor scenes from RGB-D images

The Role of Context for Object Detection and Semantic Segmentation in the Wild

MeshCNN: a network with an edge

Virtual Multiview Fusion

Field of Study

Venue Information

Name

Type

URL

Alternate Names