Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation (2019-09-24T00:00:00.000000Z)

TL;DR

An interactive GAN-based sketch-to-image translation method that helps novice users easily create images of simple objects and introduces a gating-based approach for class conditioning, which allows for distinct classes without feature mixing, from a single generator network.

Abstract

We propose an interactive GAN-based sketch-to-image translation method that helps novice users easily create images of simple objects. The user starts with a sparse sketch and a desired object category, and the network then recommends its plausible completion(s) and shows a corresponding synthesized image. This enables a feedback loop, where the user can edit the sketch based on the network's recommendations, while the network is able to better synthesize the image that the user might have in mind. In order to use a single model for a wide array of object classes, we introduce a gating-based approach for class conditioning, which allows us to generate distinct classes without feature mixing, from a single generator network.

Authors

Richard Zhang

8 papers

Alexei A. Efros

23 papers

Eli Shechtman

20 papers

TL;DR

Abstract

Authors

References64 items

Im2Pencil: Controllable Pencil Illustration From Photographs

Semantic Image Synthesis With Spatially-Adaptive Normalization

A Style-Based Generator Architecture for Generative Adversarial Networks

Self-Attention Generative Adversarial Networks

Multimodal Unsupervised Image-to-Image Translation

Synthesizing Programs for Images using Reinforced Adversarial Learning

Spectral Normalization for Generative Adversarial Networks

Which Training Methods for GANs do actually Converge?

SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis

Sparse, Smart Contours to Represent and Edit Images

Convolutional Networks with Adaptive Inference Graphs

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

Toward Multimodal Image-to-Image Translation

Progressive Growing of GANs for Improved Quality, Stability, and Variation

FiLM: Visual Reasoning with a General Conditioning Layer

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

A Neural Representation of Sketch Drawings

Multi-agent Diverse Generative Adversarial Networks

Improved Training of Wasserstein GANs

Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks

Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization

Wasserstein GAN

Towards Principled Methods for Training Generative Adversarial Networks

NIPS 2016 Tutorial: Generative Adversarial Networks

Scribbler: Controlling Deep Image Synthesis with Sketch and Color

Image-to-Image Translation with Conditional Adversarial Networks

Mode Regularized Generative Adversarial Networks

Unrolled Generative Adversarial Networks

Conditional Image Synthesis with Auxiliary Classifier GANs

Generative Visual Manipulation on the Natural Image Manifold

Instance Normalization: The Missing Ingredient for Fast Stylization

Conditional Image Generation with PixelCNN Decoders

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets

Improved Techniques for Training GANs

Adversarially Learned Inference

Adversarial Feature Learning

Residual Networks Behave Like Ensembles of Relatively Shallow Networks

Context Encoders: Feature Learning by Inpainting

Colorful Image Colorization

Autoencoding beyond pixels using a learned similarity metric

Deep Residual Learning for Image Recognition

Rethinking the Inception Architecture for Computer Vision

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Deep multi-scale video prediction beyond mean square error

Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks

Fine-Grained Visual Comparisons with Local Learning

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Auto-Encoding Variational Bayes

ImageNet classification with deep convolutional neural networks

Photosketcher: Interactive Sketch-Based Image Synthesis

ShadowDraw: real-time user guidance for freehand drawing

On expert performance in 3D curve-drawing tasks

Where do people draw lines?

Extracting and composing robust features with denoising autoencoders

Texture synthesis by non-parametric sampling

Long Short-Term Memory

Why can't most people draw what they see?

Sketchpad a man-machine graphical communication system

Information processing in dynamical systems: foundations of harmony theory

Smoothing and differentiation of data by simplified least square procedure.

Sketchpad a Man-Machine Graphical Communication System

Faceshop: Deep sketch-based face image editing

Learning to simplify: fully convolutional networks for rough sketch cleanup

Supporting Online Material for Reducing the Dimensionality of Data with Neural Networks

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names