LayoutDiffusion: Controllable Diffusion Model for Layout-to-Image Generation (2023-03-30T00:00:00.000000Z)

TL;DR

A diffusion model named LayoutDiffusion is proposed that can obtain higher generation quality and greater controllability than the previous works and is proposed to construct a structural image patch with region information and transform the patched image into a special layout to fuse with the normal layout in a unified form.

Abstract

Recently, diffusion models have achieved great success in image synthesis. However, when it comes to the layout-to-image generation where an image often has a complex scene of multiple objects, how to make strong control over both the global layout map and each detailed object remains a challenging task. In this paper, we propose a diffusion model named LayoutDiffusion that can obtain higher generation quality and greater controllability than the previous works. To overcome the difficult multimodal fusion of image and layout, we propose to construct a structural image patch with region information and transform the patched image into a special layout to fuse with the normal layout in a unified form. Moreover, Layout Fusion Module (LFM) and Object-aware Cross Attention (OaCA) are proposed to model the relationship among multiple objects and designed to be object-aware and position-sensitive, allowing for precisely controlling the spatial related information. Extensive experiments show that our LayoutDiffusion out-performs the previous SOTA methods on FID, CAS by relatively 46.35%,26.70% on COCO-stuff and 44.29%,41.82% on VG. Code is available at https://github.com/ZGCTroy/LayoutDiffusion.

Authors

Zhongang Qi

4 papers

Guangcong Zheng

1 papers

Xianpan Zhou

1 papers

TL;DR

Abstract

Authors

References59 items

Referring Expression Comprehension Using Language Adaptive Inference

Language Adaptive Weight Generation for Multi-Task Visual Grounding

Diffusion-Based Scene Graph to Image Generation with Masked Contrastive Pre-Training

Efficient Diffusion Models for Vision: A Survey

Diffusion Models in Vision: A Survey

Diffusion Models: A Comprehensive Survey of Methods and Applications

Classifier-Free Diffusion Guidance

Entropy-driven Sampling and Training Scheme for Conditional Diffusion Generation

DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

Interactive Image Synthesis with Panoptic Layout Generation

Denoising Diffusion Restoration Models

RePaint: Inpainting using Denoising Diffusion Probabilistic Models

Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models

GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models

High-Resolution Image Synthesis with Latent Diffusion Models

Image Synthesis from Layout with Locality-Aware Mask Adaption

Recent Advances and Trends in Multimodal Deep Learning: A Review

Diffusion Models Beat GANs on Image Synthesis

Image Super-Resolution via Iterative Refinement

Improved Denoising Diffusion Probabilistic Models

Score-Based Generative Modeling through Stochastic Differential Equations

Denoising Diffusion Implicit Models

Learning Multi-Level Density Maps for Crowd Counting

Denoising Diffusion Probabilistic Models

YOLOv4: Optimal Speed and Accuracy of Object Detection

Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis

Specifying Object Attributes and Relations in Interactive Scene Generation

Image Synthesis From Reconfigurable Layout and Style

Generative Modeling by Estimating Gradients of the Data Distribution

Classification Accuracy Score for Conditional Generative Models

Semantic Image Synthesis With Spatially-Adaptive Normalization

A Style-Based Generator Architecture for Generative Adversarial Networks

Image Generation From Layout

Text2Scene: Generating Abstract Scenes from Textual Descriptions

Image Generation from Scene Graphs

Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

Panoptic Segmentation

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

Mixed Precision Training

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Towards Principled Methods for Training Generative Adversarial Networks

COCO-Stuff: Thing and Stuff Classes in Context

Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts

Image-to-Image Translation with Conditional Adversarial Networks

Learning What and Where to Draw

Improved Techniques for Training GANs

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Deep Residual Learning for Image Recognition

Rethinking the Inception Architecture for Computer Vision

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

You Only Look Once: Unified, Real-Time Object Detection

Deep Unsupervised Learning using Nonequilibrium Thermodynamics

Ranking consistency for image matching and object retrieval

Graph mode-based contextual kernels for robust SVM tracking

22498 Authorized licensed use limited to the terms of the applicable license agreement with IEEE

: Common objects in

license agreement with IEEE. Restrictions apply

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names