Uni3D: Exploring Unified 3D Representation at Scale (2023-10-10T00:00:00.000000Z)

TL;DR

Uni3D, a 3D foundation model to explore the unified 3D representation at scale, efficiently scale up Uni3D to one billion parameters, and set new records on a broad range of 3D tasks, such as zero-shot classification, few- shot classification, open-world understanding and part segmentation.

Abstract

Scaling up representations for images or text has been extensively investigated in the past few years and has led to revolutions in learning vision and language. However, scalable representation for 3D objects and scenes is relatively unexplored. In this work, we present Uni3D, a 3D foundation model to explore the unified 3D representation at scale. Uni3D uses a 2D initialized ViT end-to-end pretrained to align the 3D point cloud features with the image-text aligned features. Via the simple architecture and pretext task, Uni3D can leverage abundant 2D pretrained models as initialization and image-text aligned models as the target, unlocking the great potential of 2D models and scaling-up strategies to the 3D world. We efficiently scale up Uni3D to one billion parameters, and set new records on a broad range of 3D tasks, such as zero-shot classification, few-shot classification, open-world understanding and part segmentation. We show that the strong Uni3D representation also enables applications such as 3D painting and retrieval in the wild. We believe that Uni3D provides a new direction for exploring both scaling up and efficiency of the representation in 3D domain.

Authors

Junsheng Zhou

2 papers

Jinsheng Wang

2 papers

Baorui Ma

2 papers

TL;DR

Abstract

Authors

References55 items

VIT-LENS: Towards Omni-modal Representations

Meta-Transformer: A Unified Framework for Multimodal Learning

Objaverse-XL: A Universe of 10M+ 3D Objects

OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding

ULIP-2: Towards Scalable Multimodal Pre-Training for 3D Understanding

EVA-CLIP: Improved Training Techniques for CLIP at Scale

CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data

EVA-02: A Visual Representation for Neon Genesis

CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition

Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining

Objaverse: A Universe of Annotated 3D Objects

Reproducible Scaling Laws for Contrastive Language-Image Learning

ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding

Scaling Language-Image Pre-Training via Masking

EVA: Exploring the Limits of Masked Visual Representation Learning at Scale

CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-Training

Hierarchical Text-Conditional Image Generation with CLIP Latents

Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework

Grounded Language-Image Pre-training

PointCLIP: Point Cloud Understanding by CLIP

LAVT: Language-Aware Vision Transformer for Referring Image Segmentation

DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting

Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling

Masked Autoencoders Are Scalable Vision Learners

ABO: Dataset and Benchmarks for Real-World 3D Object Understanding

PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers

BEiT: BERT Pre-Training of Image Transformers

Emerging Properties in Self-Supervised Vision Transformers

Learning Transferable Visual Models From Natural Language Supervision

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

Point Transformer

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Unsupervised Point Cloud Pre-training via Occlusion Completion

3D-FUTURE: 3D Furniture Shape with TextURE

Cylinder3D: An Effective 3D Framework for Driving-scene LiDAR Semantic Segmentation

DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters

Center-based 3D Object Detection and Tracking

Language Models are Few-Shot Learners

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

ZeRO: Memory optimizations Toward Training Trillion Parameter Models

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Dynamic Graph CNN for Learning on Point Clouds

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

A scalable active framework for region annotation in 3D shape collections

Deep Networks with Stochastic Depth

ShapeNet: An Information-Rich 3D Model Repository

Adam: A Method for Stochastic Optimization

3D ShapeNets: A deep representation for volumetric shapes

PointCLIP V2: Adapting CLIP for Powerful 3D Open-world Learning

Frozen CLIP Model is An Efficient Point Cloud Backbone

EW - SHOT RESULTS We conduct few-shot linear probing under the difficult Objaverse-LVIS dataset with labeled training samples per class from 1, 2, 4, 8 to 16

Language and cognition as distributed process interactions

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names