FastComposer: Tuning-Free Multi-subject Image Generation with Localized Attention (2023-05-17T00:00:00.000000Z)

TL;DR

FastComposer proposes delayed subject conditioning in the denoising step to maintain both identity and editability in subject-driven image generation, and paves the way for efficient, personalized, and high-quality multi-subject image creation.

Abstract

Diffusion models excel at text-to-image generation, especially in subject-driven generation for personalized images. However, existing methods are inefficient due to the subject-specific fine-tuning, which is computationally intensive and hampers efficient deployment. Moreover, existing methods struggle with multi-subject generation as they often blend identity among subjects. We present FastComposer which enables efficient, personalized, multi-subject text-to-image generation without fine-tuning. FastComposer uses subject embeddings extracted by an image encoder to augment the generic text conditioning in diffusion models, enabling personalized image generation based on subject images and textual instructions with only forward passes. To address the identity blending problem in the multi-subject generation, FastComposer proposes cross-attention localization supervision during training, enforcing the attention of reference subjects localized to the correct regions in the target images. Naively conditioning on subject embeddings results in subject overfitting. FastComposer proposes delayed subject conditioning in the denoising step to maintain both identity and editability in subject-driven image generation. FastComposer generates images of multiple unseen individuals with different styles, actions, and contexts. It achieves 300×\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\times $$\end{document}–2500×\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\times $$\end{document} speedup compared to fine-tuning-based methods and requires zero extra storage for new subjects. FastComposer paves the way for efficient, personalized, and high-quality multi-subject image creation. Code, model, and dataset are available here (https://github.com/mit-han-lab/fastcomposer).

References65 items

Distilling Diffusion Models into Conditional GANs

Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing

InstantID: Zero-shot Identity-Preserving Generation in Seconds

PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding

IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models

Face0: Instantaneously Conditioning a Text-to-Image Model on a Face

Inserting Anybody in Diffusion Models via Celeb Basis

Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models

Break-A-Scene: Extracting Multiple Concepts from a Single Image

MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing

ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation

InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning

Unified Multi-Modal Latent Diffusion for Joint Subject and Text Conditional Image Generation

Scaling up GANs for Text-to-Image Synthesis

X&Fuse: Fusing Visual Information in Text-to-Image Generation

Collage Diffusion

ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation

Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models

Adding Conditional Control to Text-to-Image Diffusion Models

Zero-shot Image-to-Image Translation

Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models

BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

Muse: Text-To-Image Generation via Masked Generative Transformers

Multi-Concept Customization of Text-to-Image Diffusion

SpaText: Spatio-Textual Representation for Controllable Image Generation

Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation

SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

Token Merging: Your ViT But Faster

LAION-5B: An open large-scale dataset for training next generation image-text models

DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation

An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion

Prompt-to-Prompt Image Editing with Cross Attention Control

Classifier-Free Diffusion Guidance

Hierarchical Text-Conditional Image Generation with CLIP Latents

MyStyle

Pseudo Numerical Methods for Diffusion Models on Manifolds

High-Resolution Image Synthesis with Latent Diffusion Models

Masked-attention Mask Transformer for Universal Image Segmentation

Instance-Conditioned GAN

Pivotal Tuning for Latent-based Editing of Real Images

CogView: Mastering Text-to-Image Generation via Transformers

Emerging Properties in Self-Supervised Vision Transformers

Paint by Word

Learning Transferable Visual Models From Natural Language Supervision

Zero-Shot Text-to-Image Generation

Score-Based Generative Modeling through Stochastic Differential Equations

Denoising Diffusion Probabilistic Models

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

A Style-Based Generator Architecture for Generative Adversarial Networks

Training Deep Nets with Sublinear Memory Cost

Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks

Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop

FaceNet: A unified embedding for face recognition and clustering

Deep Unsupervised Learning using Nonequilibrium Thermodynamics

Deep Learning Face Attributes in the Wild

Microsoft COCO: Common Objects in Context

When stylegan meets stable diffusion: A W + adapter for personalized image generation

Designing an Encoder for Fast Personalization of Text-to-Image Models

SVDiff:Compactparameterspacefordiffusionﬁne-tuning

eDiff-I: Text-to-imagediffusionmodelswithanensembleofexpertdenois-ers

spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing

a photo of a [class noun] < A* > and a [class noun] < A* > working out at the gym together

Diffusers: State-of-the-artdiffusionmodels

Acknowledgements This