Stylized Face Sketch Extraction via Generative Prior with Limited Data (2024-03-17T00:00:00.000000Z)

TL;DR

It is shown that StyleSketch outperforms existing state‐of‐the‐art sketch extraction methods and few‐shot image adaptation methods for the task of extracting high‐resolution abstract face sketches and is extended by extending its use to other domains and exploring the possibility of semantic editing.

Abstract

Facial sketches are both a concise way of showing the identity of a person and a means to express artistic intention. While a few techniques have recently emerged that allow sketches to be extracted in different styles, they typically rely on a large amount of data that is difficult to obtain. Here, we propose StyleSketch, a method for extracting high‐resolution stylized sketches from a face image. Using the rich semantics of the deep features from a pretrained StyleGAN, we are able to train a sketch generator with 16 pairs of face and the corresponding sketch images. The sketch generator utilizes part‐based losses with two‐stage learning for fast convergence during training for high‐quality sketch extraction. Through a set of comparisons, we show that StyleSketch outperforms existing state‐of‐the‐art sketch extraction methods and few‐shot image adaptation methods for the task of extracting high‐resolution abstract face sketches. We further demonstrate the versatility of StyleSketch by extending its use to other domains and explore the possibility of semantic editing. The project page can be found in https://kwanyun.github.io/stylesketch_project.

Authors

Kwan Yun

2 papers

Kwanggyoon Seo

3 papers

Chang Wook Seo

2 papers

TL;DR

Abstract

Authors

References86 items

Semi-supervised reference-based sketch extraction using a contrastive learning framework

StyleDrop: Text-to-Image Generation in Any Style

Adding Conditional Control to Text-to-Image Diffusion Models

Muse: Text-To-Image Generation via Masked Generative Transformers

Reference Based Sketch Extraction via Attention Mechanism

DynaGAN: Dynamic Few-shot Adaptation of GANs to Multiple Domains

StylePortraitVideo: Editing Portrait Videos with Expression Optimization

MIDMs: Matching Interleaved Diffusion Models for Exemplar-based Image Translation

DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation

An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion

Temporally Consistent Semantic Video Editing

Diffusion-GAN: Training GANs with Diffusion

Pastiche Master: Exemplar-Based High-Resolution Portrait Style Transfer

Learning to generate line drawings that convey geometry and semantics

Few Shot Generative Model Adaption via Relaxed Spatial Structural Alignment

Polarity Sampling: Quality and Diversity Control of Pre-Trained Generative Networks via Singular Values

RefFaceNet: Reference-based Face Image Generation from Line Art Drawings

CLIPasso

Feature-Style Encoder for Style-Based GAN Inversion

StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets

Stitch it in Time: GAN-Based Facial Editing of Real Videos

BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations

Facial-sketch Synthesis: A New Challenge

JoJoGAN: One Shot Face Stylization

High-Resolution Image Synthesis with Latent Diffusion Models

GAN-Supervised Dense Visual Alignment

Mind the Gap: Domain Gap Control for Single Shot Domain Adaptation for Generative Adversarial Networks

StyleGAN-NADA

StyleVideoGAN: A Temporal Generative Model using a Pretrained StyleGAN

A Latent Transformer for Disentangled Face Editing in Images and Videos

A Good Image Generator Is What You Need for High-Resolution Video Synthesis

Few-shot Image Generation via Cross-domain Correspondence

DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort

Labels4Free: Unsupervised Segmentation using StyleGAN

Anycost GANs for Interactive Image Synthesis and Editing

Learning Transferable Visual Models From Natural Language Supervision

Designing an encoder for StyleGAN image manipulation

Perceptual-Aware Sketch Simplification Based on Integrated VGG Layers

Few-shot Image Generation with Elastic Weight Consolidation

A Brief Review of Domain Adaptation

StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows

Denoising Diffusion Probabilistic Models

Unpaired Portrait Drawing Generation via Asymmetric Cycle Mapping

Training Generative Adversarial Networks with Limited Data

Reference-Based Sketch Image Colorization Using Augmented-Self Reference and Dense Semantic Correspondence

Line Drawings for Face Portraits From Photos Using Global and Local Structure Based GANs

GANSpace: Discovering Interpretable GAN Controls

StyleGAN2 Distillation for Feed-forward Image Manipulation

Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs

MineGAN: Effective Knowledge Transfer From GANs to Target Domains With Few Images

StarGAN v2: Diverse Image Synthesis for Multiple Domains

Analyzing and Improving the Image Quality of StyleGAN

Discriminative Joint Probability Maximum Mean Discrepancy (DJP-MMD) for Domain Adaptation

Image2StyleGAN++: How to Edit the Embedded Images?

LinesToFacePhoto: Face Photo Generation From Lines With Conditional Self-Attention Generative Adversarial Networks

Interpreting the Latent Space of GANs for Semantic Face Editing

APDrawingGAN: Generating Artistic Portrait Drawings From Face Photos With Hierarchical GANs

Image Generation From Small Datasets via Batch Statistics Adaptation

Photo-Sketching: Inferring Contour Drawings From Images

A Style-Based Generator Architecture for Generative Adversarial Networks

Large Scale GAN Training for High Fidelity Natural Image Synthesis

BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation

Deep Transfer Network with Joint Distribution Adaptation: A New Intelligent Fault Diagnosis Framework for Industry Application

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

Deep extraction of manga structural lines

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Joint distribution optimal transportation for domain adaptation

Image-to-Image Translation with Conditional Adversarial Networks

Learning to simplify

Perceptual Losses for Real-Time Style Transfer and Super-Resolution

LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop

Holistically-Nested Edge Detection

Very Deep Convolutional Networks for Large-Scale Image Recognition

Style and abstraction in portrait sketching

XDoG: advanced image stylization with eXtended Difference-of-Gaussians

Coupled information-theoretic encoding for face photo-sketch recognition