POS-BERT: Point Cloud One-Stage BERT Pre-Training (2022-04-03T00:00:00.000000Z)

TL;DR

POS-BERT is proposed, a one-stage BERT pre-training method for point clouds that achieves the state-of-the-art classification accuracy and has significantly improved many downstream tasks, such as fine-tuned classification, few-shot classification, part segmentation.

Abstract

Recently, the pre-training paradigm combining Transformer and masked language modeling has achieved tremendous success in NLP, images, and point clouds, such as BERT. However, directly extending BERT from NLP to point clouds requires training a fixed discrete Variational AutoEncoder (dVAE) before pre-training, which results in a complex two-stage method called Point-BERT. Inspired by BERT and MoCo, we propose POS-BERT, a one-stage BERT pre-training method for point clouds. Specifically, we use the mask patch modeling (MPM) task to perform point cloud pre-training, which aims to recover masked patches information under the supervision of the corresponding tokenizer output. Unlike Point-BERT, its tokenizer is extra-trained and frozen. We propose to use the dynamically updated momentum encoder as the tokenizer, which is updated and outputs the dynamic supervision signal along with the training process. Further, in order to learn high-level semantic representation, we combine contrastive learning to maximize the class token consistency between different transformation point clouds. Extensive experiments have demonstrated that POS-BERT can extract high-quality pre-training features and promote downstream tasks to improve performance. Using the pre-training model without any fine-tuning to extract features and train linear SVM on ModelNet40, POS-BERT achieves the state-of-the-art classification accuracy, which exceeds Point-BERT by 3.5\%. In addition, our approach has significantly improved many downstream tasks, such as fine-tuned classification, few-shot classification, part segmentation. The code and trained-models will be available at: \url{https://github.com/fukexue/POS-BERT}.

Authors

Renrui Zhang

15 papers

Peng Gao

14 papers

Y. Qiao

8 papers

TL;DR

Abstract

Authors

References70 items

CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding

Generative Adversarial Networks

Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling

Masked Autoencoders Are Scalable Vision Learners

ViT-YOLO:Transformer-Based YOLO for Object Detection

Progressive Seed Generation Auto-encoder for Unsupervised Point Cloud Learning

Unsupervised Learning of Geometric Sampling Invariant Representations for 3D Point Clouds

Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds

BEiT: BERT Pre-Training of Image Transformers

Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline

Self-Supervised Learning on 3D Point Clouds by Learning Discrete Generative Models

Segmenter: Transformer for Semantic Segmentation

Walk in the Cloud: Learning Curves for Point Clouds Shape Analysis

Pre-Training by Completing Point Clouds

An Empirical Study of Training Self-Supervised Vision Transformers

Robust Point Cloud Registration Framework Based on Deep Graph Matching

Barlow Twins: Self-Supervised Learning via Redundancy Reduction

Fast Convergence of DETR with Spatially Modulated Co-Attention

PCT: Point cloud transformer

Point Transformer

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Unsupervised Point Cloud Pre-training via Occlusion Completion

Unsupervised 3D Learning for Shape Analysis via Multiresolution Instance Discrimination

Self-Supervised Learning of Point Clouds via Orientation Estimation

PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Grasping pose estimation for SCARA robot based on deep learning of point cloud

Self-supervised Feature Learning by Cross-modality and Cross-view Correspondences

Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review

Global-Local Bidirectional Reasoning for Unsupervised Representation Learning of 3D Point Clouds

Improved Baselines with Momentum Contrastive Learning

Learning to Match 2D Images and 3D LiDAR Point Clouds for Outdoor Augmented Reality

A Simple Framework for Contrastive Learning of Visual Representations

Momentum Contrast for Unsupervised Visual Representation Learning

DensePoint: Learning Densely Contextual Representation for Efficient Point Cloud Processing

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

Multi-Angle Point Cloud-VAE: Unsupervised Feature Learning for 3D Point Clouds From Multiple Angles by Joint Self-Reconstruction and Half-to-Half Prediction

Relation-Shape Convolutional Neural Network for Point Cloud Analysis

Self-Supervised Deep Learning on Point Clouds by Reconstructing Space

3D Point Capsule Networks

Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering

View Inter-Prediction GAN: Unsupervised Representation Learning for 3D Shapes by Learning Global Shape Memories to Support Local View Predictions

Multiresolution Tree Networks for 3D Point Cloud Processing

SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters

SO-Net: Self-Organizing Network for Point Cloud Analysis

Dynamic Graph CNN for Learning on Point Clouds

PointCNN: Convolution On X-Transformed Points

FoldingNet: Point Cloud Auto-Encoder via Deep Grid Deformation

Fixing Weight Decay Regularization in Adam

Learning Representations and Generative Models for 3D Point Clouds

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Discrete Variational Autoencoders

3D Semantic Parsing of Large-Scale Indoor Spaces

VConv-DAE: Deep Volumetric Shape Learning Without Object Labels

Learning a Predictable and Generative Vector Representation for Objects

ShapeNet: An Information-Rich 3D Model Repository

Neural Machine Translation of Rare Words with Subword Units

3D ShapeNets: A deep representation for volumetric shapes

Auto-Encoding Variational Bayes

Generalized Denoising Auto-Encoders as Generative Models

On Visual Similarity Based 3D Model Retrieval

Rotation Invariant Spherical Harmonic Representation of 3D Shape Descriptors

Competition and cooperation in neural nets

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

“Container: Context aggregation networks,”

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Visual Pattern Recognition

mask patch modeling pretext task. The Momentum Encoder is updated dynamically during the pre-training stage and does not require extra pre-training processing

Field of Study

Journal Information

Name

Volume

Venue Information

Name