StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021-03-31T00:00:00.000000Z)

TL;DR

This work explores leveraging the power of recently introduced Contrastive Language-Image Pre-training (CLIP) models in order to develop a text-based interface for StyleGAN image manipulation that does not require such manual effort.

Abstract

Inspired by the ability of StyleGAN to generate highly realistic images in a variety of domains, much recent work has focused on understanding how to use the latent spaces of StyleGAN to manipulate generated and real images. However, discovering semantically meaningful latent manipulations typically involves painstaking human examination of the many degrees of freedom, or an annotated collection of images for each desired manipulation. In this work, we explore leveraging the power of recently introduced Contrastive Language-Image Pre-training (CLIP) models in order to develop a text-based interface for StyleGAN image manipulation that does not require such manual effort. We first introduce an optimization scheme that utilizes a CLIP-based loss to modify an input latent vector in response to a user-provided text prompt. Next, we describe a latent mapper that infers a text-guided latent manipulation step for a given input image, allowing faster and more stable text-based manipulation. Finally, we present a method for mapping text prompts to input-agnostic directions in StyleGAN’s style space, enabling interactive text-driven image manipulation. Extensive results and comparisons demonstrate the effectiveness of our approaches.

Authors

Eli Shechtman

20 papers

D. Lischinski

4 papers

Or Patashnik

6 papers

TL;DR

Abstract

Authors

References62 items

Generative Adversarial Networks

TediGAN: Text-Guided Diverse Face Image Generation and Manipulation

ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement

Paint by Word

Multimodal Neurons in Artificial Neural Networks

Learning Transferable Visual Models From Natural Language Supervision

Zero-Shot Text-to-Image Generation

Designing an encoder for StyleGAN image manipulation

Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search

Taming Transformers for High-Resolution Image Synthesis

StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation

Text-to-Image Generation Grounded by Fine-Grained User Attention

Describe What to Change: A Text-guided Unsupervised Image-to-image Translation Approach

StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows

Learning Visual Representations with Caption Annotations

Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation

Closed-Form Factorization of Latent Semantics in GANs

VirTex: Learning Visual Representations from Textual Annotations

Training Generative Adversarial Networks with Limited Data

Language Models are Few-Shot Learners

InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs

Face identity disentanglement via latent space mapping

Editing in Style: Uncovering the Local Semantics of GANs

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

GANSpace: Discovering Interpretable GAN Controls

StyleRig: Rigging StyleGAN for 3D Control Over Portrait Images

Unsupervised Discovery of Interpretable Directions in the GAN Latent Space

ManiGAN: Text-Guided Image Manipulation

StarGAN v2: Diverse Image Synthesis for Multiple Domains

Analyzing and Improving the Image Quality of StyleGAN

UNITER: UNiversal Image-TExt Representation Learning

VL-BERT: Pre-training of Generic Visual-Linguistic Representations

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

VisualBERT: A Simple and Performant Baseline for Vision and Language

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Interpreting the Latent Space of GANs for Semantic Face Editing

Object-Driven Text-To-Image Synthesis via Adversarial Training

A Style-Based Generator Architecture for Generative Adversarial Networks

Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language

Large Scale GAN Training for High Fidelity Natural Image Synthesis

ArcFace: Additive Angular Margin Loss for Deep Face Recognition

AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Semantic Image Synthesis via Adversarial Learning

Attention is All you Need

Generative Adversarial Text to Image Synthesis

The Cityscapes Dataset for Semantic Urban Scene Understanding

Perceptual Losses for Real-Time Style Transfer and Super-Resolution

LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop

Deep Learning Face Attributes in the Wild

Conditional Generative Adversarial Nets

Tooth and Claw

A Geometric Analysis of Deep Generative Image Models and Its Applications

VOGUE: Try-On by StyleGAN Interpolation Optimization

Only a matter of style: age transformation using a style-based regression model

DALL-E: CREATING IMAGES FROM TEXT

Generating images from prompts using CLIP and StyleGAN

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

trans-formers for language understanding

Input Mohawk Curly Afro Bobcut Bowlcut Purple Figure 15. Hair style manipulations obtained by the latent mapper. Except for the purple hair

A latent residual mapper, trained for a specific text prompt. Given a starting point in latent space (the input image to be manipulated)

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names