ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding (2022-12-10T00:00:00.000000Z)

TL;DR

ULIP is introduced to learn a unified representation of image, text, and 3D point cloud by pre-training with object triplets from the three modalities, using a pre-trained vision-language model that has already learned a common visual and textual space by training with massive image-text pairs.

Abstract

The recognition capabilities of current state-of-the-art 3D models are limited by datasets with a small number of annotated data and a pre-defined set of categories. In its 2D counterpart, recent advances have shown that similar problems can be significantly alleviated by employing knowledge from other modalities, such as language. Inspired by this, leveraging multimodal information for 3D modality could be promising to improve 3D understanding under the restricted data regime, but this line of research is not well studied. Therefore, we introduce ULIP to learn a unified representation of image, text, and 3D point cloud by pre-training with object triplets from the three modalities. To overcome the shortage of training triplets, ULIP leverages a pre-trained vision-language model that has already learned a common visual and textual space by training with massive image-text pairs. Then, ULIP learns a 3D representation space aligned with the common image-text space, using a small number of automatically synthesized triplets. ULIP is agnostic to 3D backbone networks and can easily be integrated into any 3D architecture. Experiments show that ULIP effectively improves the performance of multiple recent 3D backbones by simply pre-training them on ShapeNet55 using our framework, achieving state-of-the-art performance in both standard 3D classification and zero-shot 3D classification on ModelNet40 and ScanObjectNN. ULIP also improves the performance of PointMLP by around 3% in 3D classification on ScanObjectNN, and outperforms PointCLIP by 28.8% on top-1 accuracy for zero-shot 3D classification on ModelNet40. Our code and pre-trained models will be released.

References62 items

Let Images Give You More: Point Cloud Cross-Modal Training for Shape Analysis

PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies

Surface Representation for Point Clouds

DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection

Masked Autoencoders for Point Cloud Self-supervised Learning

SoftGroup for 3D Instance Segmentation on Point Clouds

Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework

SLIP: Self-supervision meets Language-Image Pre-training

Grounded Language-Image Pre-training

PointCLIP: Point Cloud Understanding by CLIP

Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling

Open Vocabulary Object Detection with Pseudo Bounding-Box Labels

Multimodal Semi-Supervised Learning for 3D Objects

An End-to-End Transformer Model for 3D Object Detection

Learning Inner-Group Relations on Point Clouds

Align before Fuse: Vision and Language Representation Learning with Momentum Distillation

Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline

Walk in the Cloud: Learning Curves for Point Clouds Shape Analysis

Open-vocabulary Object Detection via Vision and Language Knowledge Distillation

Group-Free 3D Object Detection via Transformers

StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery

LiDAR R-CNN: An Efficient and Universal 3D Object Detector

Learning Transferable Visual Models From Natural Language Supervision

Learning Geometry-Disentangled Representation for Complementary Understanding of 3D Object Point Cloud

PCT: Point cloud transformer

MVTN: Multi-View Transformation Network for 3D Shape Recognition

Point Transformer

Center-based 3D Object Detection and Tracking

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection

Geometric Back-Projection Network for Point Cloud Classification

RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds

UNITER: UNiversal Image-TExt Representation Learning

DensePoint: Learning Densely Contextual Representation for Efficient Point Cloud Processing

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

VisualBERT: A Simple and Performant Baseline for Vision and Language

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

KPConv: Flexible and Deformable Convolution for Point Clouds

Relation-Shape Convolutional Neural Network for Point Cloud Analysis

Adaptive Cross-Modal Few-Shot Learning

PointConv: Deep Convolutional Networks on 3D Point Clouds

SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters

Dynamic Graph CNN for Learning on Point Clouds

PointCNN: Convolution On X-Transformed Points

DGCNN: Disordered Graph Convolutional Neural Network Based on the Gaussian Mixture Model

3D Semantic Segmentation with Submanifold Sparse Convolutional Networks

Large-Scale Point Cloud Semantic Segmentation with Superpoint Graphs

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

3D Semantic Parsing of Large-Scale Indoor Spaces

Multi-modal Auto-Encoders as Joint Estimators for Robotics Scene Understanding

3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction

ShapeNet: An Information-Rich 3D Model Repository

VoxNet: A 3D Convolutional Neural Network for real-time object recognition

3D ShapeNets: A deep representation for volumetric shapes

Processing

Monocular 3D scene understanding with explicit occlusion reasoning

ImageNet: A large-scale hierarchical image database

Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories

Knowledge distillation

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding