pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis (2020-12-02T00:00:00.000000Z)

Abstract

We have witnessed rapid progress on 3D-aware image synthesis, leveraging recent advances in generative visual models and neural rendering. Existing approaches how-ever fall short in two ways: first, they may lack an under-lying 3D representation or rely on view-inconsistent rendering, hence synthesizing images that are not multi-view consistent; second, they often depend upon representation network architectures that are not expressive enough, and their results thus lack in image quality. We propose a novel generative model, named Periodic Implicit Generative Adversarial Networks (π-GAN or pi-GAN), for high-quality 3D-aware image synthesis. π-GAN leverages neural representations with periodic activation functions and volumetric rendering to represent scenes as view-consistent radiance fields. The proposed approach obtains state-of-the-art results for 3D-aware image synthesis with multiple real and synthetic datasets.

Authors

Jiajun Wu

18 papers

Gordon Wetzstein

8 papers

Petr Kellnhofer

4 papers

Abstract

Authors

References75 items

Generative Adversarial Networks

Neural Lumigraph Rendering

Semantic Implicit Neural Scene Representations With Semi-Supervised Training

Overfit Neural Networks as a Compact Shape Representation

Neural Sparse Voxel Fields

Shape and Viewpoint without Keypoints

Implicit Mesh Reconstruction from Unannotated Image Collections

GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis

MetaSDF: Meta-learning Signed Distance Functions

Implicit Neural Representations with Periodic Activation Functions

Leveraging 2D Data to Learn Textured 3D Mesh Generation

State of the Art on Neural Rendering

Self-Supervised Viewpoint Learning From Image Collections

Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction

Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance

Pix2Shape: Towards Unsupervised Learning of 3D Scenes from Images Using a View-Based Representation

Local Implicit Grid Representations for 3D Scenes

Convolutional Occupancy Networks

Inverse Graphics GAN: Learning to Generate 3D Shapes from Unstructured 2D Data

Implicit Geometric Regularization for Learning Shapes

BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images

Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision

SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization

Local Deep Implicit Functions for 3D Shape

Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis

Analyzing and Improving the Image Quality of StyleGAN

DIST: Rendering Deep Implicit Signed Distance Function With Differentiable Sphere Tracing

Unsupervised Learning of Probably Symmetric Deformable 3D Objects From Images in the Wild

SAL: Sign Agnostic Learning of Shapes From Raw Data

Implicit Surface Representations As Layers in Neural Networks

Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics

Unsupervised Generative 3D Shape Learning from Natural Images

Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations

Texture Fields: Learning Texture Representations in Function Space

PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization

Improved Precision and Recall Metric for Assessing Generative Models

Learning Shape Templates With Structured Implicit Functions

HoloGAN: Unsupervised Learning of 3D Representations From Natural Images

DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation

A Style-Based Generator Architecture for Generative Adversarial Networks

Occupancy Networks: Learning 3D Reconstruction in Function Space

Learning Implicit Fields for Generative Shape Modeling

Visual Object Networks: Image Generation with Disentangled 3D Representations

Escaping Plato’s Cave: 3D Shape From Adversarial Rendering

Large Scale GAN Training for High Fidelity Natural Image Synthesis

Feature-wise transformations

An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution

Neural scene representation and rendering

Assessing Generative Models via Precision and Recall

Learning Category-Specific Mesh Reconstruction from Image Collections

AmbientGAN: Generative models from lossy measurements

On the convergence properties of GAN training

Which Training Methods for GANs do actually Converge?

Demystifying MMD GANs

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

Progressive Growing of GANs for Improved Quality, Stability, and Variation

CARLA: An Open Urban Driving Simulator

FiLM: Visual Reasoning with a General Conditioning Layer

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium

3D Shape Induction from 2D Views of Multiple Objects

Learning Shape Abstractions by Assembling Volumetric Primitives

Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Improved Techniques for Training GANs

Deep Residual Learning for Image Recognition

Multi-view 3D Models from Single Images with a Convolutional Network

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Deep Learning Face Attributes in the Wild

Cat Head Detection - How to Effectively Exploit Shape and Texture Features

Optical Models for Direct Volume Rendering

Marching cubes: A high resolution 3D surface construction algorithm

Ray tracing volume densities

, Andrew Jones , Ryan Spicer , Kari Pulli , and Gordon Wetzstein . Neural lumigraph render

Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks

Representing Scenes as Neural Radiance Fields for View Synthesis

Field of Study

Journal Information