MVDream: Multi-view Diffusion for 3D Generation (2023-08-31T00:00:00.000000Z)

TL;DR

MVDream, a diffusion model that is able to generate consistent multi-view images from a given text prompt, is introduced and it is demonstrated that such a multi-view diffusion model is implicitly a generalizable 3D prior agnostic to 3D representations.

Abstract

We introduce MVDream, a diffusion model that is able to generate consistent multi-view images from a given text prompt. Learning from both 2D and 3D data, a multi-view diffusion model can achieve the generalizability of 2D diffusion models and the consistency of 3D renderings. We demonstrate that such a multi-view diffusion model is implicitly a generalizable 3D prior agnostic to 3D representations. It can be applied to 3D generation via Score Distillation Sampling, significantly enhancing the consistency and stability of existing 2D-lifting methods. It can also learn new concepts from a few 2D examples, akin to DreamBooth, but for 3D generation.

Authors

X. Yang

5 papers

Kejie Li

2 papers

Yichun Shi

4 papers

TL;DR

Abstract

Authors

References55 items

SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis

MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusion

DreamTime: An Improved Optimization Strategy for Diffusion-Guided 3D Generation

Viewset Diffusion: (0-)Image-Conditioned 3D Generative Models from 2D Data

ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation

Common Diffusion Noise Schedules and Sample Steps are Flawed

Shap-E: Generating Conditional 3D Implicit Functions

TextMesh: Generation of Realistic 3D Meshes From Text Prompts

Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models

Generative Novel View Synthesis with 3D-Aware Diffusion Models

HOLODIFFUSION: Training a 3D Diffusion Model Using 2D Images

Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation

Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior

DreamBooth3D: Subject-Driven Text-to-3D Generation

Zero-1-to-3: Zero-shot One Image to 3D Object

Multiview Compressive Coding for 3D Reconstruction

Point-E: A System for Generating 3D Point Clouds from Complex Prompts

Objaverse: A Universe of Annotated 3D Objects

RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion

Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation

SparseFusion: Distilling View-Conditioned Diffusion for 3D Reconstruction

3D Neural Field Generation Using Triplane Diffusion

MagicVideo: Efficient Video Generation With Latent Diffusion Models

Magic3D: High-Resolution Text-to-3D Content Creation

eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers

LAION-5B: An open large-scale dataset for training next generation image-text models

Novel View Synthesis with Diffusion Models

Imagen Video: High Definition Video Generation with Diffusion Models

DreamFusion: Text-to-3D using 2D Diffusion

Make-A-Video: Text-to-Video Generation without Text-Video Data

GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images

DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

Instant neural graphics primitives with a multiresolution hash encoding

High-Resolution Image Synthesis with Latent Diffusion Models

GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation

Efficient Geometry-aware 3D Generative Adversarial Networks

Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields

Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis

RoFormer: Enhanced Transformer with Rotary Position Embedding

Learning Transferable Visual Models From Natural Language Supervision

GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields

Leveraging 2D Data to Learn Textured 3D Mesh Generation

BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images

Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations

HoloGAN: Unsupervised Learning of 3D Representations From Natural Images

Learning Single-Image 3D Reconstruction by Generative Modelling of Shape, Pose and Shading

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Improved Techniques for Training GANs

Adam: A Method for Stochastic Optimization

AUTO-ENCODING VARIATIONAL BAYES

Representing Scenes as Neural Radiance Fields for View Synthesis

Threestudio project

huggingface

Stable diffusion image variation

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names