Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis

Published in

Computer Vision and Pattern Recognition(2024)

External Links:

Generate Graph DownloadPDF

TL;DR

To generate more realistic texture details, a hybrid- granularity attention module is proposed to encode multi-scale fine-grained appearance features as bias terms to augment the coarse-grained prompt.

Abstract

Diffusion model is a promising approach to image generation and has been employed for Pose-Guided Person Image Synthesis (PGPIS) with competitive performance. While existing methods simply align the person appearance to the target pose, they are prone to overfitting due to the lack of a high-level semantic understanding on the source person image. In this paper, we propose a novel Coarse-to-Fine Latent Diffusion (CFLD) method for PGPIS. In the absence of image-caption pairs and textual prompts, we de-velop a novel training paradigm purely based on images to control the generation process of a pre-trained text-to-image diffusion model. A perception-refined decoder is designed to progressively refine a set of learnable queries and extract semantic understanding of person images as a coarse-grained prompt. This allows for the decoupling of fine-grained appearance and pose information controls at different stages, and thus circumventing the potential over-fitting problem. To generate more realistic texture details, a hybrid- granularity attention module is proposed to encode multi-scale fine-grained appearance features as bias terms to augment the coarse-grained prompt. Both quantitative and qualitative experimental results on the DeepFashion benchmark demonstrate the superiority of our method over the state of the arts for PGPIS. Code is available at https://github.com/YanzuoLu/CFLD.

Authors

Yanzuo Lu

1 papers

Manlin Zhang

1 papers

Andy J. Ma

1 papers

References64 items

MLNet: Mutual Learning Network with Neighborhood Invariance for Universal Domain Adaptation

Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation

MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model

HAP: Structure-Aware Masked Image Modeling for Human-Centric Perception

Collaborative Learning of Diverse Experts for Source-free Universal Domain Adaptation

Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis

Published in

Computer Vision and Pattern Recognition(2024)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Yanzuo Lu

1 papers

Manlin Zhang

1 papers

Andy J. Ma

1 papers

References64 items

MLNet: Mutual Learning Network with Neighborhood Invariance for Universal Domain Adaptation

Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation

MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model

HAP: Structure-Aware Masked Image Modeling for Human-Centric Perception

Collaborative Learning of Diverse Experts for Source-free Universal Domain Adaptation

Xiaohua Xie

1 papers

Jian-Huang Lai

1 papers

Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models

Controllable Person Image Synthesis with Pose-Constrained Latent Diffusion

IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models

T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models

Adding Conditional Control to Text-to-Image Diffusion Models

Paint by Example: Exemplar-based Image Editing with Diffusion Models

Person Image Synthesis via Denoising Diffusion Model

Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation

InstructPix2Pix: Learning to Follow Image Editing Instructions

Best Prompts for Text-to-Image Models and How to Find Them

Prompt-to-Prompt Image Editing with Cross Attention Control

Cross Attention Based Style Distribution for Controllable Person Image Synthesis

Classifier-Free Diffusion Guidance

Neural Texture Extraction and Distribution for Controllable Person Image Synthesis

Conditional Prompt Learning for Vision-Language Models

Exploring Dual-task Correlation for Pose Guided Person Image Generation

High-Resolution Image Synthesis with Latent Diffusion Models

Masked Autoencoders Are Scalable Vision Learners

Design Guidelines for Prompt Engineering Text-to-Image Generative Models

Learning to Prompt for Vision-Language Models

CoCosNet v2: Full-Resolution Correspondence Learning for Image Translation

Diffusion Models Beat GANs on Image Synthesis

Learning Semantic Person Image Generation by Region-Adaptive Normalization

PISE: Person Image Synthesis and Editing with Decoupled GAN

Learning Transferable Visual Models From Natural Language Supervision

Style and Pose Control for Image Synthesis of Humans from a Single Monocular View

Taming Transformers for High-Resolution Image Synthesis

Score-Based Generative Modeling through Stochastic Differential Equations

Denoising Diffusion Implicit Models

XingGAN for Person Image Generation

Denoising Diffusion Probabilistic Models

Controllable Person Image Synthesis With Attribute-Decomposed GAN

Deep Image Spatial Transformation for Person Image Generation

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesis

Progressive Pose Attention Transfer for Person Image Generation

Dense Intrinsic Appearance Flow for Human Pose Transfer

A Variational U-Net for Conditional Appearance and Shape Generation

DensePose: Dense Human Pose Estimation in the Wild

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

Deformable GANs for Pose-Based Human Image Generation

Disentangled Person Image Generation

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Attention is All you Need

Pose Guided Person Image Generation

Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields

DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations

Improved Techniques for Training GANs

Deep Residual Learning for Image Recognition

Scalable Person Re-identification: A Benchmark

U-Net: Convolutional Networks for Biomedical Image Segmentation

ImageNet: A large-scale hierarchical image database

Image quality assessment: from error visibility to structural similarity

Improving Pre-trained Masked Autoencoder via Locality Enhancement for Person Re-identification

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

A method for stochastic optimization

Diffusers: State-of-the-art diffusion models

Markov processes over denu-merable products of spaces, describing large systems of automata

6428 applicable license agreement with IEEE. Restrictions

Field of Study

Computer Science

Journal Information

Name

2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Page

4818-4829

Venue Information

Name

Computer Vision and Pattern Recognition

Type

conference

URL

https://ieeexplore.ieee.org/xpl/conhome.jsp?punumber=1000147

Alternate Names

CVPR
Comput Vis Pattern Recognit