LION: Latent Point Diffusion Models for 3D Shape Generation (2022-10-12T00:00:00.000000Z)

TL;DR

The hierarchical Latent Point Diffusion Model (LION) is introduced, set up as a variational autoencoder (VAE) with a hierarchical latent space that combines a global shape latent representation with a point-structured latent space for 3D shape generation.

Abstract

Denoising diffusion models (DDMs) have shown promising results in 3D point cloud synthesis. To advance 3D DDMs and make them useful for digital artists, we require (i) high generation quality, (ii) flexibility for manipulation and applications such as conditional synthesis and shape interpolation, and (iii) the ability to output smooth surfaces or meshes. To this end, we introduce the hierarchical Latent Point Diffusion Model (LION) for 3D shape generation. LION is set up as a variational autoencoder (VAE) with a hierarchical latent space that combines a global shape latent representation with a point-structured latent space. For generation, we train two hierarchical DDMs in these latent spaces. The hierarchical VAE approach boosts performance compared to DDMs that operate on point clouds directly, while the point-structured latents are still ideally suited for DDM-based modeling. Experimentally, LION achieves state-of-the-art generation performance on multiple ShapeNet benchmarks. Furthermore, our VAE framework allows us to easily use LION for different relevant tasks: LION excels at multimodal shape denoising and voxel-conditioned synthesis, and it can be adapted for text- and image-driven 3D generation. We also demonstrate shape autoencoding and latent shape interpolation, and we augment LION with modern surface reconstruction techniques to generate smooth 3D meshes. We hope that LION provides a powerful tool for artists working with 3D shapes due to its high-quality generation, flexibility, and surface reconstruction. Project page and code: https://nv-tlabs.github.io/LION.

Authors

S. Fidler

22 papers

Arash Vahdat

3 papers

Karsten Kreis

4 papers

TL;DR

Abstract

Authors

References147 items

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

Hierarchical Text-Conditional Image Generation with CLIP Latents

AUV-Net: Learning Aligned UV Maps for Texture Transfer and Synthesis

Texturify: Generating Textures on 3D Shape Surfaces

AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation

Pseudo Numerical Methods for Diffusion Models on Manifolds

Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality

DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs

Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models

DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents

StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation

High-Resolution Image Synthesis with Latent Diffusion Models

GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models

Tackling the Generative Learning Trilemma with Denoising Diffusion GANs

Efficient Geometry-aware 3D Generative Adversarial Networks

Score-Based Generative Modeling with Critically-Damped Langevin Diffusion

A Conditional Point Diffusion-Refinement Paradigm for 3D Point Cloud Completion

Text2Mesh: Text-Driven Neural Stylization for Meshes

Zero-Shot Text-Guided Object Generation with Dream Fields

Diffusion Autoencoders: Toward a Meaningful and Decodable Representation

Neural Fields as Learnable Kernels for 3D Reconstruction

Octree Transformer: Autoregressive 3D Shape Generation on Hierarchically Structured Sequences

Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis

DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer

CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis

StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis

CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation

SurfGen: Adversarial 3D Shape Synthesis with Explicit Surface Discriminators

3DStyleNet: Creating 3D Shapes with Geometric and Texture Style Variations

Unsupervised Learning of Fine Structure Generation for 3D Point Clouds by 2D Projection Matching

SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations

Advances in Neural Rendering

Variational Diffusion Models

WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis

Score-based Generative Modeling in Latent Space

Shape As Points: A Differentiable Poisson Solver

Learning to Efficiently Sample from Diffusion Probabilistic Models

3D Shape Generation with Grid-based Implicit Functions

Learning Progressive Point Embeddings for 3D Point Cloud Generation

On Fast Sampling of Diffusion Probabilistic Models

Cascaded Diffusion Models for High Fidelity Image Generation

RPG: Learning Recursive Point Cloud Generation

Gotta Go Fast When Generating Data with Score-Based Models

Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech

Diffusion Models Beat GANs on Image Synthesis

GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds

3D Shape Generation and Completion through Point-Voxel Diffusion

Diff-TTS: A Denoising Diffusion Model for Text-to-Speech

Symbolic Music Generation with Diffusion Models

SetVAE: Learning Hierarchical Composition for Generative Modeling of Set-Structured Data

Learning Generative Models of Textured 3D Meshes from Real-World Images

Learning to Generate 3D Shapes with Generative Cellular Automata

Diffusion Probabilistic Models for 3D Point Cloud Generation

Learning Transferable Visual Models From Natural Language Supervision

Improved Denoising Diffusion Probabilistic Models

DECOR-GAN: 3D Shape Detailization by Conditional Refinement

pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis

Deformed Implicit Field: Modeling 3D Shapes with Learned Dense Correspondence

Score-Based Generative Modeling through Stochastic Differential Equations

GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields

Modular primitives for high-performance differentiable rendering

Point Transformer

Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering

Denoising Diffusion Implicit Models

NCP-VAE: Variational Autoencoders with Noise Contrastive Priors

DiffWave: A Versatile Diffusion Model for Audio Synthesis

WaveGrad: Estimating Gradients for Waveform Generation

Learning Gradient Fields for Shape Generation

Discrete Point Flow Networks for Efficient Point Cloud Generation

Progressive Point Cloud Deconvolution Generation Network

NVAE: A Deep Hierarchical Variational Autoencoder

GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis

Neural Splines: Fitting 3D Surfaces with Infinitely-Wide Neural Networks

Denoising Diffusion Probabilistic Models

SoftFlow: Probabilistic Framework for Normalizing Flow on Manifolds

The Creation and Detection of Deepfakes