StoryImager: A Unified and Efficient Framework for Coherent Story Visualization and Completion (2024-04-09T00:00:00.000000Z)

TL;DR

The StoryImager enhances the storyboard generative ability inherited from the pre-trained text-to-image model for a bidirectional generation and introduces a Target Frame Masking Strategy to extend and unify different story image generation tasks.

Abstract

Story visualization aims to generate a series of realistic and coherent images based on a storyline. Current models adopt a frame-by-frame architecture by transforming the pre-trained text-to-image model into an auto-regressive manner. Although these models have shown notable progress, there are still three flaws. 1) The unidirectional generation of auto-regressive manner restricts the usability in many scenarios. 2) The additional introduced story history encoders bring an extremely high computational cost. 3) The story visualization and continuation models are trained and inferred independently, which is not user-friendly. To these ends, we propose a bidirectional, unified, and efficient framework, namely StoryImager. The StoryImager enhances the storyboard generative ability inherited from the pre-trained text-to-image model for a bidirectional generation. Specifically, we introduce a Target Frame Masking Strategy to extend and unify different story image generation tasks. Furthermore, we propose a Frame-Story Cross Attention Module that decomposes the cross attention for local fidelity and global coherence. Moreover, we design a Contextual Feature Extractor to extract contextual information from the whole storyline. The extensive experimental results demonstrate the excellent performance of our StoryImager. The code is available at https://github.com/tobran/StoryImager.

Authors

Changsheng Xu

2 papers

Ming Tao

3 papers

Bing-Kun Bao

1 papers

TL;DR

Abstract

Authors

References43 items

CogCartoon: Towards Practical Story Visualization

Causal-Story: Local Causal Attention Utilizing Parameter-Efficient Tuning for Visual Story Synthesis

Intelligent Grimm - Open-ended Visual Storytelling via Latent Diffusion Models

Interactive Story Visualization with Multiple Characters

GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesis

Make-A-Story: Visual Memory Conditioned Consistent Story Generation

Synthesizing Coherent Story with Auto-Regressive Latent Diffusion Models

Character-centric Story Visualization via Visual Planning and Token Alignment

Clustering Generative Adversarial Networks for Story Visualization

StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation

Word-Level Fine-Grained Story Visualization

Scaling Autoregressive Models for Content-Rich Text-to-Image Generation

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

Hierarchical Text-Conditional Image Generation with CLIP Latents

Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

High-Resolution Image Synthesis with Latent Diffusion Models

GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models

Vector Quantized Diffusion Model for Text-to-Image Synthesis

Integrating Visuospatial, Linguistic, and Commonsense Structure into Story Visualization

LoRA: Low-Rank Adaptation of Large Language Models

Cascaded Diffusion Models for High Fidelity Image Generation

Improving Generation and Evaluation of Visual Stories via Semantic Consistency

Diffusion Models Beat GANs on Image Synthesis

Zero-Shot Text-to-Image Generation

Improved Denoising Diffusion Probabilistic Models

Making Pre-trained Language Models Better Few-shot Learners

Denoising Diffusion Implicit Models

DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis

Denoising Diffusion Probabilistic Models

Language Models are Few-Shot Learners

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis

StoryGAN: A Sequential Conditional GAN for Story Visualization

Imagine This! Scripts to Compositions to Videos

AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks

DeepStory: Video Story QA by Deep Embedded Memory Networks

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Deep Unsupervised Learning using Nonequilibrium Thermodynamics

P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks

Character-Preserving Coherent Story Visualization

Language Models are Unsupervised Multitask Learners

Peft: State-of-the-art parameter-efficient fine-tuning methods

Field of Study

Venue Information

Name

Type

URL

Alternate Names