FaceChain: A Playground for Human-centric Artificial Intelligence Generated Content (2023-08-28T00:00:00.000000Z)

TL;DR

FaceChain is presented, a personalized portrait generation framework that combines a series of customized image-generation model and a rich set of face-related perceptual understanding models, to tackle aforementioned challenges and to generate truthful personalized portraits, with only a handful of portrait images as input.

Abstract

Recent advancement in personalized image generation have unveiled the intriguing capability of pre-trained text-to-image models on learning identity information from a collection of portrait images. However, existing solutions are vulnerable in producing truthful details, and usually suffer from several defects such as (i) The generated face exhibit its own unique characteristics, \ie facial shape and facial feature positioning may not resemble key characteristics of the input, and (ii) The synthesized face may contain warped, blurred or corrupted regions. In this paper, we present FaceChain, a personalized portrait generation framework that combines a series of customized image-generation model and a rich set of face-related perceptual understanding models (\eg, face detection, deep face embedding extraction, and facial attribute recognition), to tackle aforementioned challenges and to generate truthful personalized portraits, with only a handful of portrait images as input. Concretely, we inject several SOTA face models into the generation procedure, achieving a more efficient label-tagging, data-processing, and model post-processing compared to previous solutions, such as DreamBooth ~\cite{ruiz2023dreambooth} , InstantBooth ~\cite{shi2023instantbooth} , or other LoRA-only approaches ~\cite{hu2021lora} . Besides, based on FaceChain, we further develop several applications to build a broader playground for better showing its value, including virtual try on and 2D talking head. We hope it can grow to serve the burgeoning needs from the communities. Note that this is an ongoing work that will be consistently refined and improved upon. FaceChain is open-sourced under Apache-2.0 license at \url{https://github.com/modelscope/facechain}.

FaceChain: A Playground for Human-centric Artificial Intelligence Generated Content

TL;DR

Abstract

Authors

References22 items

Effective Whole-body Pose Estimation with Two-stages Distillation

InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning

Adding Conditional Control to Text-to-Image Diffusion Models

Improving Training and Inference of Face Recognition Models via Random Temperature Scaling

VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wild

SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation

DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation

ABPN: Adaptive Blend Pyramid Network for Real-Time Local Retouching of Ultra High-Resolution Photo

Masked-attention Mask Transformer for Universal Image Segmentation

Blended Diffusion for Text-driven Editing of Natural Images

8-bit Optimizers via Block-wise Quantization

Towards Real-World Blind Face Restoration with Generative Facial Prior

FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age for Bias Measurement and Mitigation

A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild

Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer

Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields

Neural Architecture Search with Reinforcement Learning

A Computational Approach to Edge Detection

DamoFD: Digging into Backbone Design on Face Detection

High resolution image

Low-rank adaptation of large language models

synthesis with latent diffusion models