Masked Autoencoders for Point Cloud Self-supervised Learning (2022-03-13T00:00:00.000000Z)

TL;DR

A simple architecture entirely based on standard Transformers can surpass dedicated Transformer models from supervised learning and inspires the feasibility of applying unified architectures from languages and images to the point cloud.

Abstract

As a promising scheme of self-supervised learning, masked autoencoding has significantly advanced natural language processing and computer vision. Inspired by this, we propose a neat scheme of masked autoencoders for point cloud self-supervised learning, addressing the challenges posed by point cloud's properties, including leakage of location information and uneven information density. Concretely, we divide the input point cloud into irregular point patches and randomly mask them at a high ratio. Then, a standard Transformer based autoencoder, with an asymmetric design and a shifting mask tokens operation, learns high-level latent features from unmasked point patches, aiming to reconstruct the masked point patches. Extensive experiments show that our approach is efficient during pre-training and generalizes well on various downstream tasks. Specifically, our pre-trained models achieve 85.18% accuracy on ScanObjectNN and 94.04% accuracy on ModelNet40, outperforming all the other self-supervised learning methods. We show with our scheme, a simple architecture entirely based on standard Transformers can surpass dedicated Transformer models from supervised learning. Our approach also advances state-of-the-art accuracies by 1.5%-2.3% in the few-shot object classification. Furthermore, our work inspires the feasibility of applying unified architectures from languages and images to the point cloud.

Authors

W. Liu

10 papers

Yonghong Tian

2 papers

Yatian Pang

2 papers

TL;DR

Abstract

Authors

References63 items

Hierarchical Perceiver

data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

Implicit Autoencoder for Point-Cloud Self-Supervised Representation Learning

Full Transformer Framework for Robust Point Cloud Registration With Deep Information Interaction

Masked Feature Prediction for Self-Supervised Visual Pre-Training

Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling

SimMIM: a Simple Framework for Masked Image Modeling

Masked Autoencoders Are Scalable Vision Learners

Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds

CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention

Perceiver IO: A General Architecture for Structured Inputs & Outputs

VOLO: Vision Outlooker for Visual Recognition

BEiT: BERT Pre-Training of Image Transformers

Self-Supervised Learning on 3D Point Clouds by Learning Discrete Generative Models

PRA-Net: Point Relation-Aware Network for 3D Point Cloud Analysis

Perceiver: General Perception with Iterative Attention

Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet

Self-Supervised Pretraining of 3D Features on any Point-Cloud

PCT: Point cloud transformer

Point Transformer

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Unsupervised Point Cloud Pre-training via Occlusion Completion

Self-Supervised Few-Shot Learning on Point Clouds

PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding

Generative Pretraining From Pixels

Self-Supervised Learning: Generative or Contrastive

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Language Models are Few-Shot Learners

Global-Local Bidirectional Reasoning for Unsupervised Representation Learning of 3D Point Clouds

Improved Baselines with Momentum Contrastive Learning

A Simple Framework for Contrastive Learning of Visual Representations

Geometric Back-Projection Network for Point Cloud Classification

Momentum Contrast for Unsupervised Visual Representation Learning

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

RoBERTa: A Robustly Optimized BERT Pretraining Approach

KPConv: Flexible and Deformable Convolution for Point Clouds

Relation-Shape Convolutional Neural Network for Point Cloud Analysis

Self-Supervised Deep Learning on Point Clouds by Reconstructing Space

Unsupervised Feature Learning via Non-parametric Instance Discrimination

SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters

SO-Net: Self-Organizing Network for Point Cloud Analysis

Dynamic Graph CNN for Learning on Point Clouds

PointCNN: Convolution On X-Transformed Points

FoldingNet: Point Cloud Auto-Encoder via Deep Grid Deformation

Decoupled Weight Decay Regularization

Learning Representations and Generative Models for 3D Point Clouds

Attention is All you Need

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

A Point Set Generation Network for 3D Object Reconstruction from a Single Image

A scalable active framework for region annotation in 3D shape collections

Discrete Variational Autoencoders

SGDR: Stochastic Gradient Descent with Warm Restarts

Context Encoders: Feature Learning by Inpainting

ShapeNet: An Information-Rich 3D Model Repository

3D ShapeNets: A deep representation for volumetric shapes

PVT: Point-Voxel Transformer for 3D Deep Learning

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

Set transformer: A framework for attention-based permutation-invariant neural networks

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names