3D-Aware Semantic-Guided Generative Model for Human Synthesis (2021-12-02T00:00:00.000000Z)

TL;DR

A 3D-aware Semantic-Guided Generative Model (3D-SGAN) for human image synthesis, which combines a GNeRF with a texture generator that can learn 3D human representations with a photo-realistic, controllable generation.

Abstract

Generative Neural Radiance Field (GNeRF) models, which extract implicit 3D representations from 2D images, have recently been shown to produce realistic images representing rigid/semi-rigid objects, such as human faces or cars. However, they usually struggle to generate high-quality images representing non-rigid objects, such as the human body, which is of a great interest for many computer graphics applications. This paper proposes a 3D-aware Semantic-Guided Generative Model (3D-SGAN) for human image synthesis, which combines a GNeRF with a texture generator. The former learns an implicit 3D representation of the human body and outputs a set of 2D semantic segmentation masks. The latter transforms these semantic masks into a real image, adding a realistic texture to the human appearance. Without requiring additional 3D information, our model can learn 3D human representations with a photo-realistic, controllable generation. Our experiments on the DeepFashion dataset show that 3D-SGAN significantly outperforms the most recent baselines. The code is available at https://github.com/zhangqianhui/3DSGAN

Authors

N. Sebe

21 papers

Aliaksandr Siarohin

5 papers

H. Tang

3 papers

TL;DR

Abstract

Authors

References89 items

SMPL: A Skinned Multi-Person Linear Model

VoLux-GAN: A Generative Model for 3D Face Synthesis with HDRI Relighting

StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation

3D-aware Image Synthesis via Learning Structural and Textural Representations

GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation

Generative Adversarial Networks

Efficient Geometry-aware 3D Generative Adversarial Networks

FENeRF: Face Editing in Neural Radiance Fields

Generative Occupancy Fields for 3D Surface-Aware Image Synthesis

A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis

CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis

StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis

Learning Realistic Human Reposing using Cyclic Self-Supervision with 3D Shape, Pose, and Appearance Consistency

Alias-Free Generative Adversarial Networks

CoCosNet v2: Full-Resolution Correspondence Learning for Image Translation

Controllable Person Image Synthesis with Spatially-Adaptive Warped Normalization

Towards a Neural Graphics Pipeline for Controllable Image Generation

EigenGAN: Layer-Wise Eigen-Learning for GANs

StylePeople: A Generative Model of Fullbody Human Avatars

Learning Semantic Person Image Generation by Region-Adaptive Normalization

Unconstrained Scene Generation with Locally Conditioned Radiance Fields

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis

CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields

KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs

HumanGAN: A Generative Model of Human Images

PISE: Person Image Synthesis and Editing with Decoupled GAN

Style and Pose Control for Image Synthesis of Humans from a Single Monocular View

Designing an encoder for StyleGAN image manipulation

Neural Re-rendering of Humans from a Single Image

Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans

Patch2Pix: Epipolar-Guided Pixel-Level Correspondences

pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis

GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields

Liquid Warping GAN With Attention: A Unified Framework for Human Image Synthesis

Bipartite Graph Reasoning GANs for Person Image Generation

StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows

XingGAN for Person Image Generation

Closed-Form Factorization of Latent Semantics in GANs

GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis

Generating Person Images with Appearance-aware Pose Stylizer

Training Generative Adversarial Networks with Limited Data

Controllable Person Image Synthesis With Attribute-Decomposed GAN

Deep Image Spatial Transformation for Person Image Generation

First Order Motion Model for Image Animation

Inverse Graphics GAN: Learning to Generate 3D Shapes from Unstructured 2D Data

BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images

Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis

Analyzing and Improving the Image Quality of StyleGAN

Generating High-Resolution Fashion Model Images Wearing Custom Outfits

Interpreting the Latent Space of GANs for Semantic Face Editing

Appearance and Pose-Conditioned Human Image Generation Using Deformable GANs

Progressive Pose Attention Transfer for Person Image Generation

Unsupervised Person Image Generation With Semantic Parsing Transformation

Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?

HoloGAN: Unsupervised Learning of 3D Representations From Natural Images

Semantic Image Synthesis With Spatially-Adaptive Normalization

Learning Single-Image 3D Reconstruction by Generative Modelling of Shape, Pose and Shading

A Style-Based Generator Architecture for Generative Adversarial Networks

Visual Object Networks: Image Generation with Disentangled 3D Representations

Escaping Plato’s Cave: 3D Shape From Adversarial Rendering

Coordinate-based Texture Inpainting for Pose-Guided Image Generation

Large Scale GAN Training for High Fidelity Natural Image Synthesis

Dense Pose Transfer

Synthesizing Images of Humans in Unseen Poses

Multimodal Unsupervised Image-to-Image Translation

DensePose: Dense Human Pose Estimation in the Wild

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

Disentangled Person Image Generation

StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation

VITON: An Image-Based Virtual Try-on Network

Progressive Growing of GANs for Improved Quality, Stability, and Variation

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Attention is All you Need

Pose Guided Person Image Generation

A Generative Model of People in Clothing

Improved Training of Wasserstein GANs