Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following

Published in

arXiv.org(2023)

External Links:

Generate Graph DownloadPDF

TL;DR

Point-LLM is presented, the first 3D large language model (LLM) following 3D multi-modal instructions, which injects the semantics of Point-Bind into pre-trained LLMs, e.g., LLaMA, which requires no 3D instruction data, but exhibits superior 3D and multi- modal question-answering capacity.

Abstract

We introduce Point-Bind, a 3D multi-modality model aligning point clouds with 2D image, language, audio, and video. Guided by ImageBind, we construct a joint embedding space between 3D and multi-modalities, enabling many promising applications, e.g., any-to-3D generation, 3D embedding arithmetic, and 3D open-world understanding. On top of this, we further present Point-LLM, the first 3D large language model (LLM) following 3D multi-modal instructions. By parameter-efficient fine-tuning techniques, Point-LLM injects the semantics of Point-Bind into pre-trained LLMs, e.g., LLaMA, which requires no 3D instruction data, but exhibits superior 3D and multi-modal question-answering capacity. We hope our work may cast a light on the community for extending 3D point clouds to multi-modality applications. Code is available at https://github.com/ZiyuGuo99/Point-Bind_Point-LLM.

Authors

P. Heng

25 papers

Hongsheng Li

5 papers

Renrui Zhang

15 papers

References99 items

Meta-Transformer: A Unified Framework for Multimodal Learning

VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models

OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding

ULIP-2: Towards Scalable Multimodal Pre-Training for 3D Understanding

ImageBind One Embedding Space to Bind Them All

Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following

Published in

arXiv.org(2023)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

P. Heng

25 papers

Hongsheng Li

5 papers

Renrui Zhang

15 papers

References99 items

Meta-Transformer: A Unified Framework for Multimodal Learning

VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models

OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding

ULIP-2: Towards Scalable Multimodal Pre-Training for 3D Understanding

ImageBind One Embedding Space to Bind Them All

Jiaming Han

6 papers

Xiangyang Zhu

1 papers

Xianzheng Ma

3 papers

Instance-Variant Loss with Gaussian RBF Kernel for 3D Cross-modal Retriveal

LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model

MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuning

ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding

LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention

CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition

3D Concept Learning and Reasoning from Multi-View Images

PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detection

Parameter is Not All You Need: Starting from Non-Parametric Networks for 3D Point Cloud Analysis

LLaMA: Open and Efficient Foundation Language Models

Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-training

Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining

End-to-End 3D Dense Captioning with Vote2Cap-DETR

Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models

Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?

Point-E: A System for Generating 3D Point Clouds from Complex Prompts

Learning 3D Representations from 2D Pre-Trained Models via Image-to-Point Masked Autoencoders

PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning

Magic3D: High-Resolution Text-to-3D Content Creation

LAION-5B: An open large-scale dataset for training next generation image-text models

CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-Training

DreamFusion: Text-to-3D using 2D Diffusion

EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Grounding

CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention

ISS: Image as Stepping Stone for Text-Guided 3D Shape Generation

P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting

Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer

Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification

PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies

Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

Hierarchical Text-Conditional Image Generation with CLIP Latents

Learning Audio-Video Modalities from Image Captions

CLIP-Mesh: Generating textured meshes from text using pretrained image-text models

Masked Autoencoders for Point Cloud Self-supervised Learning

SoftGroup for 3D Instance Segmentation on Point Clouds

X -Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioning

CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding

ScanQA: 3D Question Answering for Spatial Scene Understanding

High-Resolution Image Synthesis with Latent Diffusion Models

Text2Mesh: Text-Driven Neural Stylization for Meshes

PointCLIP: Point Cloud Understanding by CLIP

Zero-Shot Text-Guided Object Generation with Dream Fields

Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling

End-to-End Referring Video Object Segmentation with Multimodal Transformers

Masked Autoencoders Are Scalable Vision Learners

CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation

An End-to-End Transformer Model for 3D Object Detection

CLIP2Video: Mastering Video-Text Retrieval via Image CLIP

BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models

LoRA: Low-Rank Adaptation of Large Language Models

Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Models

Vset: A Multimodal Transformer for Visual Speech Enhancement

Cross-Modal Center Loss for 3D Cross-Modal Retrieval

Group-Free 3D Object Detection via Transformers

Learning Transferable Visual Models From Natural Language Supervision

Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts

VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs

Contrastive Learning of Medical Visual Representations from Paired Images and Text

Unsupervised Point Cloud Pre-training via Occlusion Completion

Self-Supervised Learning of Point Clouds via Orientation Estimation

VirTex: Learning Visual Representations from Textual Annotations

Language Models are Few-Shot Learners

Global-Local Bidirectional Reasoning for Unsupervised Representation Learning of 3D Point Clouds

3D Point Cloud Processing and Learning for Autonomous Driving: Impacting Map Creation, Localization, and Perception

Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs

PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection

読書行動識別のためのSelf-supervised Learningの実験的検討

Vision-Language Navigation With Self-Supervised Auxiliary Reasoning Tasks

PU-GAN: A Point Cloud Upsampling Adversarial Network

Embodied Question Answering in Photorealistic Environments With Point Cloud Perception

Self-Supervised Deep Learning on Point Clouds by Reconstructing Space

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning

Decoupled Weight Decay Regularization

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Layer Normalization

3D Semantic Parsing of Large-Scale Indoor Spaces

ShapeNet: An Information-Rich 3D Model Repository

ESC: Dataset for Environmental Sound Classification

Microsoft COCO Captions: Data Collection and Evaluation Server

3D ShapeNets: A deep representation for volumetric shapes

ImageNet: A large-scale hierarchical image database

Exploring 3D navigation: combining speed-coupled flying with orbiting

Visual instruction tuning, 2023

The Expressive Power of Tuning Only the Norm Layers

ULIP: Learning Unified Representation of Language, Image and Point Cloud for 3D Understanding

Pix4Point: Image Pretrained Transformers for 3D Point Cloud Understanding

Exploring Cross-Video and Cross-Modality Signals for Weakly-Supervised Audio-Visual Video Parsing

Learning from 2D: Contrastive Pixel-to-point Knowledge Transfer for 3D Pretraining

TextANIMAR Challenge

Less is more: Towards efficient few-shot 3d semantic segmentation via training-free networks

Field of Study

Computer Science

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

arXiv.org

Type

URL

https://arxiv.org

Alternate Names

ArXiv