StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation (2022-09-13T00:00:00.000000Z)

TL;DR

This work enhances or 'retro-fit' the pretrained text-to-image synthesis models with task-specific modules for story continuation and facilitates copying of visual elements from the source image, thereby improving continuity in the generated visual story.

Abstract

Recent advances in text-to-image synthesis have led to large pretrained transformers with excellent capabilities to generate visualizations from a given text. However, these models are ill-suited for specialized tasks like story visualization, which requires an agent to produce a sequence of images given a corresponding sequence of captions, forming a narrative. Moreover, we find that the story visualization task fails to accommodate generalization to unseen plots and characters in new narratives. Hence, we first propose the task of story continuation, where the generated visual story is conditioned on a source image, allowing for better generalization to narratives with new characters. Then, we enhance or 'retro-fit' the pretrained text-to-image synthesis models with task-specific modules for (a) sequential image generation and (b) copying relevant elements from an initial frame. Then, we explore full-model finetuning, as well as prompt-based tuning for parameter-efficient adaptation, of the pre-trained model. We evaluate our approach StoryDALL-E on two existing datasets, PororoSV and FlintstonesSV, and introduce a new dataset DiDeMoSV collected from a video-captioning dataset. We also develop a model StoryGANc based on Generative Adversarial Networks (GAN) for story continuation, and compare it with the StoryDALL-E model to demonstrate the advantages of our approach. We show that our retro-fitting approach outperforms GAN-based models for story continuation and facilitates copying of visual elements from the source image, thereby improving continuity in the generated visual story. Finally, our analysis suggests that pretrained transformers struggle to comprehend narratives containing several characters. Overall, our work demonstrates that pretrained text-to-image synthesis models can be adapted for complex and low-resource tasks like story continuation.

References56 items

VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks

Improving language models by retrieving from trillions of tokens

Integrating Visuospatial, Linguistic, and Commonsense Structure into Story Visualization

UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning

Towards a Unified View of Parameter-Efficient Transfer Learning

CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders

BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models

LoRA: Low-Rank Adaptation of Large Language Models

Compacter: Efficient Low-Rank Hypercomplex Adapter Layers

Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks

Improving Generation and Evaluation of Visual Stories via Semantic Consistency

VideoGPT: Video Generation using VQ-VAE and Transformers

The Power of Scale for Parameter-Efficient Prompt Tuning

Learning Transferable Visual Models From Natural Language Supervision

Zero-Shot Text-to-Image Generation

Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts

Cross-Modal Contrastive Learning for Text-to-Image Generation

Taming Transformers for High-Resolution Image Synthesis

Parameter-Efficient Transfer Learning with Diff Pruning

Improved-StoryGAN for sequential images visualization

ContraGAN: Contrastive Learning for Conditional Image Generation

MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning

CPGAN: Full-Spectrum Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesis

PororoGAN: An Improved Story Visualization Model on Pororo-SV Dataset

Semantic Object Accuracy for Generative Text-to-Image Synthesis

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Semantics Disentangling for Text-To-Image Generation

DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis

MirrorGAN: Learning Text-To-Image Generation by Redescription

Parameter-Efficient Transfer Learning for NLP

A Style-Based Generator Architecture for Generative Adversarial Networks

StoryGAN: A Sequential Conditional GAN for Story Visualization

Model Cards for Model Reporting

Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning

Imagine This! Scripts to Compositions to Videos

YOLOv3: An Incremental Improvement

Universal Sentence Encoder

CartoonGAN: Generative Adversarial Networks for Photo Cartoonization

Generative Image Inpainting with Contextual Attention

AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks

Neural Discrete Representation Learning

Localizing Moments in Video with Natural Language

DeepStory: Video Story QA by Deep Embedded Memory Networks

CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning

StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks

Self-Critical Sequence Training for Image Captioning

Image-to-Image Translation with Conditional Adversarial Networks

Conditional Image Generation with PixelCNN Decoders

DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers