CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis (2021-10-19T00:00:00.000000Z)

TL;DR

CIPS-3D is presented, a style-based, 3D-aware generator that is composed of a shallow NeRF network and a deep implicit neural representation (INR) network that synthesizes each pixel value independently without any spatial convolution or upsampling operation.

Abstract

The style-based GAN (StyleGAN) architecture achieved state-of-the-art results for generating high-quality images, but it lacks explicit and precise control over camera poses. The recently proposed NeRF-based GANs made great progress towards 3D-aware generators, but they are unable to generate high-quality images yet. This paper presents CIPS-3D, a style-based, 3D-aware generator that is composed of a shallow NeRF network and a deep implicit neural representation (INR) network. The generator synthesizes each pixel value independently without any spatial convolution or upsampling operation. In addition, we diagnose the problem of mirror symmetry that implies a suboptimal solution and solve it by introducing an auxiliary discriminator. Trained on raw, single-view images, CIPS-3D sets new records for 3D-aware image synthesis with an impressive FID of 6.97 for images at the $256\times256$ resolution on FFHQ. We also demonstrate several interesting directions for CIPS-3D such as transfer learning and 3D-aware face stylization. The synthesis results are best viewed as videos, so we recommend the readers to check our github project at https://github.com/PeterouZh/CIPS-3D

Authors

Bingbing Ni

8 papers

Lingxi Xie

6 papers

Peng Zhou

2 papers

TL;DR

Abstract

Authors

References62 items

StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis

Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering

CodeNeRF: Disentangled Neural Radiance Fields for Object Categories

Alias-Free Generative Adversarial Networks

On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation

Unconstrained Scene Generation with Locally Conditioned Radiance Fields

CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields

MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo

Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields

UltraSR: Spatial Encoding is a Missing Key for Implicit Image Function-based Arbitrary-Scale Super-Resolution

Learning Continuous Image Representation with Local Implicit Image Function

pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis

Image Generators with Conditionally-Independent Pixel Synthesis

GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields

Adversarial Generation of Continuous Images

Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs

NeRF++: Analyzing and Improving Neural Radiance Fields

Unsupervised Image-to-Image Translation via Pre-Trained StyleGAN2 Network

Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains

NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections

Closed-Form Factorization of Latent Semantics in GANs

GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis

Implicit Neural Representations with Periodic Activation Functions

Training Generative Adversarial Networks with Limited Data

GANSpace: Discovering Interpretable GAN Controls

Local Deep Implicit Functions for 3D Shape

StarGAN v2: Diverse Image Synthesis for Multiple Domains

Analyzing and Improving the Image Quality of StyleGAN

Unsupervised Learning of Probably Symmetric Deformable 3D Objects From Images in the Wild

Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis

Deep Meta Functionals for Shape Representation

Interpreting the Latent Space of GANs for Semantic Face Editing

On the "steerability" of generative adversarial networks

PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization

Learning Shape Templates With Structured Implicit Functions

HoloGAN: Unsupervised Learning of 3D Representations From Natural Images

DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation

A Style-Based Generator Architecture for Generative Adversarial Networks

Occupancy Networks: Learning 3D Reconstruction in Function Space

Learning Implicit Fields for Generative Shape Modeling

Visual Object Networks: Image Generation with Disentangled 3D Representations

Escaping Plato’s Cave: 3D Shape From Adversarial Rendering

Large Scale GAN Training for High Fidelity Natural Image Synthesis

GAN Dissection: Visualizing and Understanding Generative Adversarial Networks

Feature-wise transformations

Which Training Methods for GANs do actually Converge?

Demystifying MMD GANs

Progressive Growing of GANs for Improved Quality, Stability, and Variation

FiLM: Visual Reasoning with a General Conditioning Layer

Wasserstein Generative Adversarial Networks

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Improved Training of Wasserstein GANs

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Adam: A Method for Stochastic Optimization

Ray tracing volume densities

NeRF-: Neural Radiance Fields Without Known Camera Parameters

GENERATIVE ADVERSARIAL NETS

Representing Scenes as Neural Radiance Fields for View Synthesis

Alexey Dosovitskiy, and Daniel Duckworth

FreezeG

CartoonFaces

High - fidelity performance metrics for generative models in Py - Torch

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names