PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding (2020-07-21T00:00:00.000000Z)

TL;DR

This work aims at facilitating research on 3D representation learning by selecting a suite of diverse datasets and tasks to measure the effect of unsupervised pre-training on a large source set of 3D scenes and achieving improvement over recent best results in segmentation and detection across 6 different benchmarks.

Abstract

Arguably one of the top success stories of deep learning is transfer learning. The finding that pre-training a network on a rich source set (eg., ImageNet) can help boost performance once fine-tuned on a usually much smaller target set, has been instrumental to many applications in language and vision. Yet, very little is known about its usefulness in 3D point cloud understanding. We see this as an opportunity considering the effort required for annotating data in 3D. In this work, we aim at facilitating research on 3D representation learning. Different from previous works, we focus on high-level scene understanding tasks. To this end, we select a suite of diverse datasets and tasks to measure the effect of unsupervised pre-training on a large source set of 3D scenes. Our findings are extremely encouraging: using a unified triplet of architecture, source dataset, and contrastive loss for pre-training, we achieve improvement over recent best results in segmentation and detection across 6 different benchmarks for indoor and outdoor, real and synthetic datasets -- demonstrating that the learned representation can generalize across domains. Furthermore, the improvement was similar to supervised pre-training, suggesting that future efforts should favor scaling data collection over more detailed annotation. We hope these findings will encourage more research on unsupervised pretext task design for 3D deep learning.

Authors

C. Qi

15 papers

L. Guibas

28 papers

Saining Xie

10 papers

TL;DR

Abstract

Authors

References87 items

On Learning Sets of Symmetric Elements

A Simple Framework for Contrastive Learning of Visual Representations

Continuous Geodesic Convolutions for Learning on 3D Shapes

ImVoteNet: Boosting 3D Object Detection in Point Clouds With Image Votes

Self-Supervised Learning of Pretext-Invariant Representations

Momentum Contrast for Unsupervised Visual Representation Learning

Fully Convolutional Geometric Features

Unsupervised Multi-Task Feature Learning on Point Clouds

ShellNet: Efficient Point Cloud Convolutional Neural Networks Using Concentric Shells Statistics

Contrastive Multiview Coding

Learning Representations by Maximizing Mutual Information Across Views

Data-Efficient Image Recognition with Contrastive Predictive Coding

Deep Closest Point: Learning Representations for Point Cloud Registration

Scaling and Benchmarking Self-Supervised Visual Representation Learning

Deep Hough Voting for 3D Object Detection in Point Clouds

4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks

ConvPoint: Continuous convolutions for point cloud processing

Local Aggregation for Unsupervised Learning of Visual Embeddings

Self-Supervised Deep Learning on Point Clouds by Reconstructing Space

3D Point Capsule Networks

3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans

GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloud

Rethinking ImageNet Pre-Training

3D Recurrent Neural Networks with Context Fusion for Point Cloud Semantic Segmentation

PPF-FoldNet: Unsupervised Learning of Rotation Invariant 3D Local Descriptors

Learning deep representations by mutual information estimation and maximization

Deep Clustering for Unsupervised Learning of Visual Features

Multiresolution Tree Networks for 3D Point Cloud Processing

Representation Learning with Contrastive Predictive Coding

RGCNN: Regularized Graph CNN for Point Cloud Segmentation

Deep Parametric Continuous Convolutional Neural Networks

Tangent Convolutions for Dense Prediction in 3D

Attentional ShapeContextNet for Point Cloud Recognition

Spherical Convolutional Neural Network for 3D Point Clouds

Unsupervised Feature Learning via Non-parametric Instance Discrimination

SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters

Local Spectral Graph Convolution for Point Set Feature Learning

SO-Net: Self-Organizing Network for Point Cloud Analysis

SPLATNet: Sparse Lattice Networks for Point Cloud Processing

A Papier-Mache Approach to Learning 3D Surface Generation

Unsupervised Representation Learning by Predicting Image Rotations

Dynamic Graph CNN for Learning on Point Clouds

PointCNN: Convolution On X-Transformed Points

FoldingNet: Point Cloud Auto-Encoder via Deep Grid Deformation

Mining Point Cloud Local Structures by Kernel Correlation and Graph Pooling

Pointwise Convolutional Neural Networks

3DContextNet: K-d Tree Guided Hierarchical Learning of Point Clouds Using Local Contextual Cues

3D Semantic Segmentation with Submanifold Sparse Convolutional Networks

Large-Scale Point Cloud Semantic Segmentation with Superpoint Graphs

Frustum PointNets for 3D Object Detection from RGB-D Data

SEGCloud: Semantic Segmentation of 3D Point Clouds

3D Point Cloud Registration for Localization Using a Deep Neural Network Auto-Encoder

Learning Representations and Generative Models for 3D Point Clouds

FeaStNet: Feature-Steered Graph Convolutions for 3D Shape Analysis

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset

Escape from Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Models

Deep Sets

ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes

SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction

Aggregated Residual Transformations for Deep Neural Networks

A scalable active framework for region annotation in 3D shape collections

3D Semantic Parsing of Large-Scale Indoor Spaces

The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes

Context Encoders: Feature Learning by Inpainting

Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles

Colorful Image Colorization

3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions

Deep Residual Learning for Image Recognition

ShapeNet: An Information-Rich 3D Model Repository

Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images

SUN RGB-D: A RGB-D scene understanding benchmark suite

Unsupervised Visual Representation Learning by Context Prediction

U-Net: Convolutional Networks for Biomedical Image Segmentation