PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning (2022-11-21T00:00:00.000000Z)

TL;DR

This paper first collaborate CLIP and GPT to be a unified 3D open-world learner, named as Point-CLIP V2, which fully unleashes their potential for zero-shot 3D classification, segmentation, and detection, demonstrating the generalization ability for unified 3D open-world learning.

Abstract

Large-scale pre-trained models have shown promising open-world performance for both vision and language tasks. However, their transferred capacity on 3D point clouds is still limited and only constrained to the classification task. In this paper, we first collaborate CLIP and GPT to be a unified 3D open-world learner, named as Point-CLIP V2, which fully unleashes their potential for zero-shot 3D classification, segmentation, and detection. To better align 3D data with the pre-trained language knowledge, Point-CLIP V2 contains two key designs. For the visual end, we prompt CLIP via a shape projection module to generate more realistic depth maps, narrowing the domain gap between projected point clouds with natural images. For the textual end, we prompt the GPT model to generate 3D-specific text as the input of CLIP’s textual encoder. Without any training in 3D domains, our approach significantly surpasses PointCLIP by +42.90%, +40.44%, and +28.75% accuracy on three datasets for zero-shot 3D classification. On top of that, V2 can be extended to few-shot 3D classification, zero-shot 3D part segmentation, and 3D object detection in a simple manner, demonstrating our generalization ability for unified 3D open-world learning. Code is available at https://github.com/yangyangyang127/PointCLIP_V2.

Authors

Shanghang Zhang

3 papers

Renrui Zhang

15 papers

Ziyu Guo

4 papers

TL;DR

Abstract

Authors

References79 items

Personalize Segment Anything Model with One Shot

ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding

Parameter is Not All You Need: Starting from Non-Parametric Networks for 3D Point Cloud Analysis

Prompt, Generate, Then Cache: Cascade of Foundation Models Makes Strong Few-Shot Learners

Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-training

Learning 3D Representations from 2D Pre-Trained Models via Image-to-Point Masked Autoencoders

Decorate the Newcomers: Visual Domain Prompt for Continual Test Time Adaptation

Visual Prompting for Adversarial Robustness

CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-Training

What does a platypus look like? Generating customized prompts for zero-shot image classification

Frozen CLIP Models are Efficient Video Learners

P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting

Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learning

Open-Vocabulary 3D Detection via Image-level Class and Debiased Cross-modal Contrastive Learning

Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training

Multimodal Knowledge Alignment with Reinforcement Learning

Large Language Models are Zero-Shot Reasoners

Unsupervised Prompt Learning for Vision-Language Models

POS-BERT: Point Cloud One-Stage BERT Pre-Training

Visual Prompt Tuning

Conditional Prompt Learning for Vision-Language Models

Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework

Chain of Thought Prompting Elicits Reasoning in Large Language Models

RegionCLIP: Region-based Language-Image Pretraining

PointCLIP: Point Cloud Understanding by CLIP

3D Compositional Zero-shot Learning with DeCompositional Consensus

Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling

Generated Knowledge Prompting for Commonsense Reasoning

CLIP-Adapter: Better Vision-Language Models with Feature Adapters

An End-to-End Transformer Model for 3D Object Detection

An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA

Learning to Prompt for Vision-Language Models

Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Clouds

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline

Walk in the Cloud: Learning Curves for Point Clouds Shape Analysis

Open-vocabulary Object Detection via Vision and Language Knowledge Distillation

The Power of Scale for Parameter-Efficient Prompt Tuning

Zero-Shot Learning on 3D Point Cloud Objects and Beyond

Towards a Weakly Supervised Framework for 3D Point Cloud Object Detection and Annotation

Learning Transferable Visual Models From Natural Language Supervision

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Center-based 3D Object Detection and Tracking

View-GCN: View-Based Graph Convolutional Network for 3D Shape Analysis

Language Models are Few-Shot Learners

How Can We Know What Language Models Know?

EPN: Edge-Aware PointNet for Object Recognition from Multi-View 2.5D Point Clouds

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Learning Relationships for Multi-View 3D Object Recognition

Universal Adversarial Triggers for Attacking and Analyzing NLP

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Mitigating the Hubness Problem for Zero-Shot Learning of 3D Objects

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Dominant Set Clustering and Pooling for Multi-View 3D Object Recognition

nuScenes: A Multimodal Dataset for Autonomous Driving

Zero-shot Learning of 3D Point Cloud Objects

PointConv: Deep Convolutional Networks on 3D Point Clouds

A Deeper Look at 3D Shape Classifiers

Dynamic Graph CNN for Learning on Point Clouds

Attention is All you Need

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

A scalable active framework for region annotation in 3D shape collections

Deep Residual Learning for Image Recognition

ShapeNet: An Information-Rich 3D Model Repository

DeepPano: Deep Panoramic Representation for 3-D Shape Recognition

Multi-view Convolutional Neural Networks for 3D Shape Recognition

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

3D ShapeNets: A deep representation for volumetric shapes

Rectified Linear Units Improve Restricted Boltzmann Machines

EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual and Language Learning

Visual Prompting: Modifying Pixel Space to Adapt Pre-trained Models

Segmenting 3D Hybrid Scenes via Zero-Shot Learning

Prefix-Tuning: Optimizing Continuous Prompts for Generation