LION: Latent Point Diffusion Models for 3D Shape Generation

Published in

Neural Information Processing Systems(2022)

External Links:

Generate Graph DownloadPDF

TL;DR

The hierarchical Latent Point Diffusion Model (LION) is introduced, set up as a variational autoencoder (VAE) with a hierarchical latent space that combines a global shape latent representation with a point-structured latent space for 3D shape generation.

Abstract

Denoising diffusion models (DDMs) have shown promising results in 3D point cloud synthesis. To advance 3D DDMs and make them useful for digital artists, we require (i) high generation quality, (ii) flexibility for manipulation and applications such as conditional synthesis and shape interpolation, and (iii) the ability to output smooth surfaces or meshes. To this end, we introduce the hierarchical Latent Point Diffusion Model (LION) for 3D shape generation. LION is set up as a variational autoencoder (VAE) with a hierarchical latent space that combines a global shape latent representation with a point-structured latent space. For generation, we train two hierarchical DDMs in these latent spaces. The hierarchical VAE approach boosts performance compared to DDMs that operate on point clouds directly, while the point-structured latents are still ideally suited for DDM-based modeling. Experimentally, LION achieves state-of-the-art generation performance on multiple ShapeNet benchmarks. Furthermore, our VAE framework allows us to easily use LION for different relevant tasks: LION excels at multimodal shape denoising and voxel-conditioned synthesis, and it can be adapted for text- and image-driven 3D generation. We also demonstrate shape autoencoding and latent shape interpolation, and we augment LION with modern surface reconstruction techniques to generate smooth 3D meshes. We hope that LION provides a powerful tool for artists working with 3D shapes due to its high-quality generation, flexibility, and surface reconstruction. Project page and code: https://nv-tlabs.github.io/LION.

Authors

S. Fidler

22 papers

Arash Vahdat

3 papers

Karsten Kreis

4 papers

References147 items

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

Hierarchical Text-Conditional Image Generation with CLIP Latents

AUV-Net: Learning Aligned UV Maps for Texture Transfer and Synthesis

Texturify: Generating Textures on 3D Shape Surfaces

AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation

LION: Latent Point Diffusion Models for 3D Shape Generation

Published in

Neural Information Processing Systems(2022)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

S. Fidler

22 papers

Arash Vahdat

3 papers

Karsten Kreis

4 papers

References147 items

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

Hierarchical Text-Conditional Image Generation with CLIP Latents

AUV-Net: Learning Aligned UV Maps for Texture Transfer and Synthesis

Texturify: Generating Textures on 3D Shape Surfaces

AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation

Francis Williams

2 papers

Xiaohui Zeng

2 papers

Pseudo Numerical Methods for Diffusion Models on Manifolds

Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality

DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs

Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models

DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents

StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation

High-Resolution Image Synthesis with Latent Diffusion Models

GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models

Tackling the Generative Learning Trilemma with Denoising Diffusion GANs

Efficient Geometry-aware 3D Generative Adversarial Networks

Score-Based Generative Modeling with Critically-Damped Langevin Diffusion

A Conditional Point Diffusion-Refinement Paradigm for 3D Point Cloud Completion

Text2Mesh: Text-Driven Neural Stylization for Meshes

Zero-Shot Text-Guided Object Generation with Dream Fields

Diffusion Autoencoders: Toward a Meaningful and Decodable Representation

Neural Fields as Learnable Kernels for 3D Reconstruction

Octree Transformer: Autoregressive 3D Shape Generation on Hierarchically Structured Sequences

Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis

DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer

CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis

StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis

CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation

SurfGen: Adversarial 3D Shape Synthesis with Explicit Surface Discriminators

3DStyleNet: Creating 3D Shapes with Geometric and Texture Style Variations

Unsupervised Learning of Fine Structure Generation for 3D Point Clouds by 2D Projection Matching

SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations

Advances in Neural Rendering

Variational Diffusion Models

WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis

Score-based Generative Modeling in Latent Space

Shape As Points: A Differentiable Poisson Solver

Learning to Efficiently Sample from Diffusion Probabilistic Models

3D Shape Generation with Grid-based Implicit Functions

Learning Progressive Point Embeddings for 3D Point Cloud Generation

On Fast Sampling of Diffusion Probabilistic Models

Cascaded Diffusion Models for High Fidelity Image Generation

RPG: Learning Recursive Point Cloud Generation

Gotta Go Fast When Generating Data with Score-Based Models

Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech

Diffusion Models Beat GANs on Image Synthesis

GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds

3D Shape Generation and Completion through Point-Voxel Diffusion

Diff-TTS: A Denoising Diffusion Model for Text-to-Speech

Symbolic Music Generation with Diffusion Models

SetVAE: Learning Hierarchical Composition for Generative Modeling of Set-Structured Data

Learning Generative Models of Textured 3D Meshes from Real-World Images

Learning to Generate 3D Shapes with Generative Cellular Automata

Diffusion Probabilistic Models for 3D Point Cloud Generation

Learning Transferable Visual Models From Natural Language Supervision

Improved Denoising Diffusion Probabilistic Models

DECOR-GAN: 3D Shape Detailization by Conditional Refinement

pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis

Deformed Implicit Field: Modeling 3D Shapes with Learned Dense Correspondence

Score-Based Generative Modeling through Stochastic Differential Equations

GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields

Modular primitives for high-performance differentiable rendering

Point Transformer

Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering

Denoising Diffusion Implicit Models

NCP-VAE: Variational Autoencoders with Noise Contrastive Priors

DiffWave: A Versatile Diffusion Model for Audio Synthesis

WaveGrad: Estimating Gradients for Waveform Generation

Learning Gradient Fields for Shape Generation

Discrete Point Flow Networks for Efficient Point Cloud Generation

Progressive Point Cloud Deconvolution Generation Network

NVAE: A Deep Hierarchical Variational Autoencoder

GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis

Neural Splines: Fitting 3D Surfaces with Infinitely-Wide Neural Networks

Denoising Diffusion Probabilistic Models

SoftFlow: Probabilistic Framework for Normalizing Flow on Manifolds

The Creation and Detection of Deepfakes

Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classification

Convolutional Occupancy Networks

PolyGen: An Autoregressive Generative Model of 3D Meshes

Deepfakes and Disinformation: Exploring the Impact of Synthetic Political Video on Deception, Uncertainty, and Trust in News

Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis

Deep learning for deepfakes creation and detection: A survey

SDM-NET: Deep Generative Network for Structured Deformable Mesh

Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer

3D Volumetric Modeling with Introspective Neural Networks

Generative Modeling by Estimating Gradients of the Data Distribution

Point-Voxel CNN for Efficient 3D Deep Learning

PointFlow: 3D Point Cloud Generation With Continuous Normalizing Flows

An Introduction to Variational Autoencoders

Texture Fields: Learning Texture Representations in Function Space

3D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutions

Occupancy Networks: Learning 3D Reconstruction in Function Space

Learning Implicit Fields for Generative Shape Modeling

Resampled Priors for Variational Autoencoders

PointGrow: Autoregressively Learned Point Cloud Generation with Self-Attention

FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models

Learning Localized Generative Models for 3D Point Clouds via Graph Convolution

Variational Autoencoder with Implicit Optimal Priors

100

Neural Ordinary Differential Equations

101

Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling

102

Learning Descriptor Networks for 3D Shape Synthesis and Analysis

103

Group Normalization

104

Distribution Matching in Variational Inference

105

Spectral Normalization for Generative Adversarial Networks

106

A Papier-Mache Approach to Learning 3D Surface Generation

107

Dynamic Graph CNN for Learning on Point Clouds

108

Deformable Shape Completion with Graph Convolutional Autoencoders

109

Variational Autoencoders for Deforming 3D Mesh Models

110

Squeeze-and-Excitation Networks

111

Learning Representations and Generative Models for 3D Point Clouds

112

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

113

VAE with a VampPrior

114

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

115

Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

116

Density estimation using Real NVP

117

A Large Dataset of Object Scans

118

Deep Residual Learning for Image Recognition

119

ShapeNet: An Information-Rich 3D Model Repository

120

Variational Inference with Normalizing Flows

121

Deep Unsupervised Learning using Nonequilibrium Thermodynamics

122

Stochastic Backpropagation and Approximate Inference in Deep Generative Models

123

3D Object Representations for Fine-Grained Categorization

124

A Connection Between Score Matching and Denoising Autoencoders

125

Poisson surface reconstruction

126

A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants

127

Marching cubes: A high resolution 3D surface construction algorithm

128

TIME REVERSAL OF DIFFUSIONS

129

Reverse-time diffusion equation models

130

A family of embedded Runge-Kutta formulae

131

Text to Mesh Without 3D Supervision Using Limit Subdivision

132

Progressive distillation for fast sampling of diffusion models

133

D2C: Diffusion-Denoising Models for Few-shot Conditional Generation

134

SP-GAN:sphere-guided 3d shape generation and manipulation

135

AUTO-ENCODING VARIATIONAL BAYES

136

Mitsuba 2: a retargetable forward and inverse renderer

137

Elbo surgery: yet another way to carve up the variational evidence lower bound

138

Visualizing Data using t-SNE

139

Structurenet: Hierarchical graph networks for 3d shape generation

140

Representing Scenes as Neural Radiance Fields for View Synthesis

141

b) Did you mention the license of the assets? [Yes]

142

c) Did you include any new assets either in the supplemental material or as a URL?

143

e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

144

Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

145

c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

146

b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

147

Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Field of Study

Computer ScienceMathematics

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

Neural Information Processing Systems

Type

conference

URL

http://neurips.cc/

Alternate Names

Neural Inf Process Syst
NeurIPS
NIPS