Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning? (2022-12-16T00:00:00.000000Z)

TL;DR

This paper revisits masked modeling in a unified fashion of knowledge distillation, and shows that foundational Transformers pretrained with 2D images or natural languages can help self-supervised 3D representation learning through training Autoencoders as Cross-Modal Teachers (ACT).

Abstract

The success of deep learning heavily relies on large-scale data with comprehensive labels, which is more expensive and time-consuming to fetch in 3D compared to 2D images or natural languages. This promotes the potential of utilizing models pretrained with data more than 3D as teachers for cross-modal knowledge transferring. In this paper, we revisit masked modeling in a unified fashion of knowledge distillation, and we show that foundational Transformers pretrained with 2D images or natural languages can help self-supervised 3D representation learning through training Autoencoders as Cross-Modal Teachers (ACT). The pretrained Transformers are transferred as cross-modal 3D teachers using discrete variational autoencoding self-supervision, during which the Transformers are frozen with prompt tuning for better knowledge inheritance. The latent features encoded by the 3D teachers are used as the target of masked point modeling, wherein the dark knowledge is distilled to the 3D Transformer students as foundational geometry understanding. Our ACT pretrained 3D learner achieves state-of-the-art generalization capacity across various downstream benchmarks, e.g., 88.21% overall accuracy on ScanObjectNN. Codes have been released at https://github.com/RunpeiDong/ACT.

Authors

Zekun Qi

5 papers

Runpei Dong

5 papers

Zheng Ge

5 papers

TL;DR

Abstract

Authors

References122 items

Language-Assisted 3D Feature Learning for Semantic Scene Understanding

Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting

HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions

Contrastive Deep Supervision

Siamese Image Modeling for Self-Supervised Vision Representation Learning

Towards Efficient 3D Object Detection with Knowledge Distillation

PointDistiller: Structured Knowledge Distillation Towards Efficient and Compact 3D Detection

Masked Image Modeling with Denoising Contrast

Language-Grounded Indoor 3D Semantic Segmentation in the Wild

Beyond Masking: Demystifying Token-Based Pre-Training for Vision Transformers

Visual Prompt Tuning

Masked Discrimination for Self-Supervised Learning on Point Clouds

Masked Autoencoders for Point Cloud Self-supervised Learning

HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction

CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding

Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework

Masked Feature Prediction for Self-Supervised Visual Pre-Training

SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representations

4DContrast: Contrastive Learning with Dynamic Correspondences for 3D Scene Understanding

PointCLIP: Point Cloud Understanding by CLIP

Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling

iBOT: Image BERT Pre-Training with Online Tokenizer

Masked Autoencoders Are Scalable Vision Learners

Towards a Unified View of Parameter-Efficient Transfer Learning

An End-to-End Transformer Model for 3D Object Detection

Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds

PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers

RandomRooms: Unsupervised Pre-training from Synthetic Shapes and Randomized Layouts for 3D Object Detection

On the Opportunities and Risks of Foundation Models

BEiT: BERT Pre-Training of Image Transformers

Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline

Cross-Modal Center Loss for 3D Cross-Modal Retrieval

ResMLP: Feedforward Networks for Image Classification With Data-Efficient Training

MLP-Mixer: An all-MLP Architecture for Vision

PRA-Net: Point Relation-Aware Network for 3D Point Cloud Analysis

Learning Transferable Visual Models From Natural Language Supervision

Zero-Shot Text-to-Image Generation

Self-Supervised Pretraining of 3D Features on any Point-Cloud

Training data-efficient image transformers & distillation through attention

PCT: Point cloud transformer

Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts

MVTN: Multi-View Transformation Network for 3D Shape Recognition

Exploring Simple Siamese Representation Learning

Point Transformer

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Unsupervised Point Cloud Pre-training via Occlusion Completion

Self-Supervised Few-Shot Learning on Point Clouds

Self-Supervised Learning of Point Clouds via Orientation Estimation

PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding

Generative Pretraining From Pixels

A Closer Look at Local Aggregation Operators in Point Cloud Analysis

Debiased Contrastive Learning

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Language Models are Few-Shot Learners

Dense-Resolution Network for Point Cloud Classification and Segmentation

Self-Supervised Learning for Domain Adaptation on Point Clouds

Global-Local Bidirectional Reasoning for Unsupervised Representation Learning of 3D Point Clouds

PointASNL: Robust Point Clouds Processing Using Nonlocal Neural Networks With Adaptive Sampling

A Simple Framework for Contrastive Learning of Visual Representations

Just Go With the Flow: Self-Supervised Scene Flow Estimation

Geometric Back-Projection Network for Point Cloud Classification

Momentum Contrast for Unsupervised Visual Representation Learning

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

TinyBERT: Distilling BERT for Natural Language Understanding

DensePoint: Learning Densely Contextual Representation for Efficient Point Cloud Processing

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

openTSNE: a modular Python library for t-SNE dimensionality reduction and embedding

Selfie: Self-supervised Pretraining for Image Embedding

Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation

Deep Hough Voting for 3D Object Detection in Point Clouds

KPConv: Flexible and Deformable Convolution for Point Clouds

Relation-Shape Convolutional Neural Network for Point Cloud Analysis

Unsupervised Embedding Learning via Invariant and Spreading Instance Feature

Self-Supervised Deep Learning on Point Clouds by Reconstructing Space