Instance-aware Dynamic Prompt Tuning for Pre-trained Point Cloud Models (2023-04-14T00:00:00.000000Z)

TL;DR

The essence of IDPT is to develop a dynamic prompt generation module to perceive semantic prior features of each point cloud instance and generate adaptive prompt tokens to enhance the model's robustness, providing a promising solution to parameter-efficient learning for pre-trained point cloud models.

Abstract

Pre-trained point cloud models have found extensive applications in 3D understanding tasks like object classification and part segmentation. However, the prevailing strategy of full fine-tuning in downstream tasks leads to large per-task storage overhead for model parameters, which limits the efficiency when applying large-scale pre-trained models. Inspired by the recent success of visual prompt tuning (VPT), this paper attempts to explore prompt tuning on pre-trained point cloud models, to pursue an elegant balance between performance and parameter efficiency. We find while instance-agnostic static prompting, e.g. VPT, shows some efficacy in downstream transfer, it is vulnerable to the distribution diversity caused by various types of noises in real-world point cloud data. To conquer this limitation, we propose a novel Instance-aware Dynamic Prompt Tuning (IDPT) strategy for pre-trained point cloud models. The essence of IDPT is to develop a dynamic prompt generation module to perceive semantic prior features of each point cloud instance and generate adaptive prompt tokens to enhance the model's robustness. Notably, extensive experiments demonstrate that IDPT outperforms full finetuning in most tasks with a mere 7% of the trainable parameters, providing a promising solution to parameter-efficient learning for pre-trained point cloud models. Code is available at https://github.com/zyh16143998882/ICCV23-IDPT.

Authors

Shutao Xia

4 papers

Yaohua Zha

1 papers

Jinpeng Wang

3 papers

TL;DR

Abstract

Authors

References58 items

Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining

Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?

Point-DAE: Denoising Autoencoders for Self-Supervised Point Cloud Learning

On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey

CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-Training

P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting

Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training

Visual Prompt Tuning

Masked Discrimination for Self-Supervised Learning on Point Clouds

Masked Autoencoders for Point Cloud Self-supervised Learning

Conditional Prompt Learning for Vision-Language Models

CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding

Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework

data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

PointCLIP: Point Cloud Understanding by CLIP

DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting

Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling

Masked Autoencoders Are Scalable Vision Learners

P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks

Learning to Prompt for Vision-Language Models

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

Multimodal Few-Shot Learning with Frozen Language Models

Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline

The Power of Scale for Parameter-Efficient Prompt Tuning

PRA-Net: Point Relation-Aware Network for 3D Point Cloud Analysis

GPT Understands, Too

Learning Transferable Visual Models From Natural Language Supervision

Making Pre-trained Language Models Better Few-shot Learners

Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning

PCT: Point cloud transformer

End-to-End Human Pose and Mesh Reconstruction with Transformers

MVTN: Multi-View Transformation Network for 3D Shape Recognition

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Unsupervised Point Cloud Pre-training via Occlusion Completion

PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding

Language Models are Few-Shot Learners

Dense-Resolution Network for Point Cloud Classification and Segmentation

A Simple Framework for Contrastive Learning of Visual Representations

Geometric Back-Projection Network for Point Cloud Classification

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

Relation-Shape Convolutional Neural Network for Point Cloud Analysis

Parameter-Efficient Transfer Learning for NLP

Unsupervised Feature Learning via Non-parametric Instance Discrimination

Unsupervised Representation Learning by Predicting Image Rotations

Dynamic Graph CNN for Learning on Point Clouds

PointCNN: Convolution On X-Transformed Points

Attention is All you Need

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

A scalable active framework for region annotation in 3D shape collections

ShapeNet: An Information-Rich 3D Model Repository

3D ShapeNets: A deep representation for volumetric shapes

A theory of learning from different domains

Analysis of Representations for Domain Adaptation

Self-organizing neural network that discovers surfaces in random-dot stereograms

Frozen CLIP Model is An Efficient Point Cloud Backbone

PVT: Point-Voxel Transformer for 3D Deep Learning

Visualizing Data using t-SNE

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names