3D-aware Image Synthesis via Learning Structural and Textural Representations (2021-12-20T00:00:00.000000Z)

TL;DR

This work proposes a novel framework, termed as VolumeGAN, for high-fidelity 3D-aware image synthesis, through explicitly learning a structural representation and a textural representation in a Generative Adversarial Network.

Abstract

Making generative models 3D-aware bridges the 2D image space and the 3D physical world yet remains challenging. Recent attempts equip a Generative Adversarial Network (GAN) with a Neural Radiance Field (NeRF), which maps 3D coordinates to pixel values, as a 3D prior. However, the implicit function in NeRF has a very local receptive field, making the generator hard to become aware of the global structure. Meanwhile, NeRF is built on volume rendering which can be too costly to produce high-resolution results, increasing the optimization difficulty. To alleviate these two problems, we propose a novel framework, termed as VolumeGAN, for high-fidelity 3D-aware image synthesis, through explicitly learning a structural representation and a textural representation. We first learn a feature volume to represent the underlying structure, which is then converted to a feature field using a NeRF-like model. The feature field is further accumulated into a 2D feature map as the textural representation, followed by a neural renderer for appearance synthesis. Such a design enables independent control of the shape and the appearance. Project page is at https://genforce.github.io/volumegan.

Authors

Bolei Zhou

20 papers

Ceyuan Yang

5 papers

Sida Peng

3 papers

TL;DR

Abstract

Authors

References55 items

Generative Occupancy Fields for 3D Surface-Aware Image Synthesis

A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis

StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis

Alias-Free Generative Adversarial Networks

NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction

Low-Rank Subspaces in GANs

EigenGAN: Layer-Wise Eigen-Learning for GANs

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis

GNeRF: GAN-based Neural Radiance Field without Posed Camera

pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis

Image Generators with Conditionally-Independent Pixel Synthesis

GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields

The Hessian Penalty: A Weak Prior for Unsupervised Disentanglement

NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections

Neural Sparse Voxel Fields

Generative Hierarchical Features from Synthesizing Images

Closed-Form Factorization of Latent Semantics in GANs

GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis

Implicit Neural Representations with Periodic Activation Functions

WHENet: Real-time Fine-Grained Estimation for Wide Range Head Pose

InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs

In-Domain GAN Inversion for Real Image Editing

Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance

Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion

Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision

Image Processing Using Multi-Code GAN Prior

Analyzing and Improving the Image Quality of StyleGAN

DIST: Rendering Deep Implicit Signed Distance Function With Differentiable Sphere Tracing

Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis

Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations

Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?

HoloGAN: Unsupervised Learning of 3D Representations From Natural Images

HoloGAN: Unsupervised Learning of 3D Representations From Natural Images

DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation

A Style-Based Generator Architecture for Generative Adversarial Networks

Occupancy Networks: Learning 3D Reconstruction in Function Space

Visual Object Networks: Image Generation with Disentangled 3D Representations

Progressive Growing of GANs for Improved Quality, Stability, and Variation

CARLA: An Open Urban Driving Simulator

FiLM: Visual Reasoning with a General Conditioning Layer

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Attention is All you Need

Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets

LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop

A large-scale car dataset for fine-grained categorization and verification

Adam: A Method for Stochastic Optimization

Deep Learning Face Attributes in the Wild

Principal Component Analysis (PCA)

Cat Head Detection - How to Effectively Exploit Shape and Texture Features

Marching cubes: a high resolution 3D surface construction algorithm

Nerf: 9 Representing scenes as neural radiance fields for view synthesis

GENERATIVE ADVERSARIAL NETS

Rectifier Nonlinearities Improve Neural Network Acoustic Models

Figure 1. More Synthesized results with the front camera view by π-GAN [1] and our VolumeGAN, where the faces proposed by VolumeGAN are more consistent with the given view

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names