StyleAvatar3D: Leveraging Image-Text Diffusion Models for High-Fidelity 3D Avatar Generation (2023-05-30T00:00:00.000000Z)

TL;DR

This paper presents a novel method for generating high-quality, stylized 3D avatars that utilizes pre-trained image-text diffusion models for data generation and a Generative Adversarial Network (GAN)-based 3D generation network for training.

Abstract

The recent advancements in image-text diffusion models have stimulated research interest in large-scale 3D generative models. Nevertheless, the limited availability of diverse 3D resources presents significant challenges to learning. In this paper, we present a novel method for generating high-quality, stylized 3D avatars that utilizes pre-trained image-text diffusion models for data generation and a Generative Adversarial Network (GAN)-based 3D generation network for training. Our method leverages the comprehensive priors of appearance and geometry offered by image-text diffusion models to generate multi-view images of avatars in various styles. During data generation, we employ poses extracted from existing 3D models to guide the generation of multi-view images. To address the misalignment between poses and images in data, we investigate view-specific prompts and develop a coarse-to-fine discriminator for GAN training. We also delve into attribute-related prompts to increase the diversity of the generated avatars. Additionally, we develop a latent diffusion model within the style space of StyleGAN to enable the generation of avatars based on image inputs. Our approach demonstrates superior performance over current state-of-the-art methods in terms of visual quality and diversity of the produced avatars.

Authors

Chunhua Shen

21 papers

Tao Chen

5 papers

YU Gang

2 papers

TL;DR

Abstract

Authors

References55 items

DreamBooth3D: Subject-Driven Text-to-3D Generation

T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models

Adding Conditional Control to Text-to-Image Diffusion Models

Learning 3D-Aware Image Synthesis with Unknown Pose Distribution

3DAvatarGAN: Bridging Domains for Personalized Editable Avatars

Point-E: A System for Generating 3D Point Clouds from Complex Prompts

RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion

Diffusion Guided Domain Adaptation of Image Generators

Magic3D: High-Resolution Text-to-3D Content Creation

Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures

Hierarchical Normalization for Robust Monocular Depth Estimation

LION: Latent Point Diffusion Models for 3D Shape Generation

DreamFusion: Text-to-3D using 2D Diffusion

GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images

Diffusion Models: A Comprehensive Survey of Methods and Applications

DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation

Classifier-Free Diffusion Guidance

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

Hierarchical Text-Conditional Image Generation with CLIP Latents

High-Resolution Image Synthesis with Latent Diffusion Models

GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models

Efficient Geometry-aware 3D Generative Adversarial Networks

Zero-Shot Text-Guided Object Generation with Dream Fields

Diffusion Autoencoders: Toward a Meaningful and Decodable Representation

Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields

StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis

Fake it till you make it: face analysis in the wild using synthetic data alone

ADNet: Leveraging Error-Bias Towards Normal Direction in Face Alignment

LoRA: Low-Rank Adaptation of Large Language Models

Pivotal Tuning for Latent-based Editing of Real Images

Diffusion Models Beat GANs on Image Synthesis

3D Shape Generation and Completion through Point-Voxel Diffusion

OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association

Diffusion Probabilistic Models for 3D Point Cloud Generation

Learning Transferable Visual Models From Natural Language Supervision

pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis

Score-Based Generative Modeling through Stochastic Differential Equations

GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields

Denoising Diffusion Implicit Models

Denoising Diffusion Probabilistic Models

Analyzing and Improving the Image Quality of StyleGAN

Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer

OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields

A Style-Based Generator Architecture for Generative Adversarial Networks

Deep appearance models for face rendering

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

How Far are We from Solving the 2D & 3D Face Alignment Problem? (and a Dataset of 230,000 3D Facial Landmarks)

Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Facial Landmark Detection by Deep Multi-task Learning

Marching cubes: A high resolution 3D surface construction algorithm

A Computational Approach to Edge Detection

Nerf: Representing scenes as neural radiance fields for view synthesis

The face of art: landmark detection and geometric style in portraits

The relightables: Volumetric performance capture of humans with realistic relighting

radiant light, Shade, darkness, silhouette, dimness, obscurity, shadow, blur ” Table 3: Overview of the prompts

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names