GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis (2020-07-05T00:00:00.000000Z)

TL;DR

This paper proposes a generative model for radiance fields which have recently proven successful for novel view synthesis of a single scene, and introduces a multi-scale patch-based discriminator to demonstrate synthesis of high-resolution images while training the model from unposed 2D images alone.

Abstract

While 2D generative adversarial networks have enabled high-resolution image synthesis, they largely lack an understanding of the 3D world and the image formation process. Thus, they do not provide precise control over camera viewpoint or object pose. To address this problem, several recent approaches leverage intermediate voxel-based representations in combination with differentiable rendering. However, existing methods either produce low image resolution or fall short in disentangling camera and scene properties, e.g., the object identity may vary with the viewpoint. In this paper, we propose a generative model for radiance fields which have recently proven successful for novel view synthesis of a single scene. In contrast to voxel-based representations, radiance fields are not confined to a coarse discretization of the 3D space, yet allow for disentangling camera and scene properties while degrading gracefully in the presence of reconstruction ambiguity. By introducing a multi-scale patch-based discriminator, we demonstrate synthesis of high-resolution images while training our model from unposed 2D images alone. We systematically analyze our approach on several challenging synthetic and real-world datasets. Our experiments reveal that radiance fields are a powerful representation for generative image synthesis, leading to 3D consistent models that render with high fidelity.

Authors

Andreas Geiger

18 papers

Michael Niemeyer

4 papers

Yiyi Liao

4 papers

TL;DR

Abstract

Authors

References78 items

Learning Implicit Surface Light Fields

Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction

Universal Differentiable Renderer for Implicit Neural Representations

Local Implicit Grid Representations for 3D Scenes

Convolutional Occupancy Networks

Semi-Supervised StyleGAN for Disentanglement Learning

BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images

High-Fidelity Synthesis with Disentangled Representation

Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision

Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis

Analyzing and Improving the Image Quality of StyleGAN

Fourier Spectrum Discrepancies in Deep Network Generated Images

Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics

RGBD-GAN: Unsupervised 3D Representation Learning From Natural Image Datasets via RGBD Image Synthesis

Deep Mesh Reconstruction From Single RGB Images via Topology Modification Networks

Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations

Generating Diverse High-Fidelity Images with VQ-VAE-2

DeepView: View Synthesis With Learned Gradient Descent

Texture Fields: Learning Texture Representations in Function Space

PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization

Source Generator Attribution via Inversion

Pushing the Boundaries of View Extrapolation With Multiplane Images

Learning Shape Templates With Structured Implicit Functions

HoloGAN: Unsupervised Learning of 3D Representations From Natural Images

BAE-NET: Branched Autoencoder for Shape Co-Segmentation

Pix2Vox: Context-Aware 3D Reconstruction From Single and Multi-View Images

FaceForensics++: Learning to Detect Manipulated Facial Images

DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation

Detecting GAN generated Fake Images using Co-occurrence Matrices

A Style-Based Generator Architecture for Generative Adversarial Networks

Occupancy Networks: Learning 3D Reconstruction in Function Space

Learning Implicit Fields for Generative Shape Modeling

DeepVoxels: Learning Persistent 3D Feature Embeddings

Visual Object Networks: Image Generation with Disentangled 3D Representations

Escaping Plato’s Cave: 3D Shape From Adversarial Rendering

Large Scale GAN Training for High Fidelity Natural Image Synthesis

Geometric Image Synthesis

On the Spectral Bias of Neural Networks

Deep Marching Cubes: Learning Explicit Surface Representations

Stereo magnification

Detection of GAN-Generated Fake Images over Social Networks

Modular Generative Adversarial Networks

Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images

Unsupervised Geometry-Aware Representation for 3D Human Pose Estimation

Spectral Normalization for Generative Adversarial Networks

A Papier-Mache Approach to Learning 3D Surface Generation

Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation

Which Training Methods for GANs do actually Converge?

Demystifying MMD GANs

Progressive Growing of GANs for Improved Quality, Stability, and Variation

CARLA: An Open Urban Driving Simulator

Interpretable Transformations with Encoder-Decoder Networks

Learning Representations and Generative Models for 3D Point Clouds

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Image-to-Image Translation with Conditional Adversarial Networks

OctNet: Learning Deep 3D Representations at High Resolutions

Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Generative and Discriminative Voxel Modeling with Convolutional Neural Networks

Instance Normalization: The Missing Ingredient for Fast Stylization

Unsupervised Learning of 3D Structure from Images

Structure-from-Motion Revisited

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets

3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction

Generative Image Modeling Using Style and Structure Adversarial Networks

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Semantic Image Segmentation via Deep Parsing Network

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Learning to Disentangle Factors of Variation with Manifold Interaction

The Caltech-UCSD Birds-200-2011 Dataset

Cat Head Detection - How to Effectively Exploit Shape and Texture Features

Optical Models for Direct Volume Rendering

Ray tracing volume densities

AUTO-ENCODING VARIATIONAL BAYES

Protecting World Leaders Against Deep Fakes

Supplementary: Monocular Neural Image Based Rendering with Continuous View Control

GENERATIVE ADVERSARIAL NETS