Real-time user-guided image colorization with learned deep priors (2017-05-08T00:00:00.000000Z)

TL;DR

This work proposes a deep learning approach for user-guided image colorization, which directly maps a grayscale image, along with sparse, local user "hints" to an output colorization with a Convolutional Neural Network.

Abstract

We propose a deep learning approach for user-guided image colorization. The system directly maps a grayscale image, along with sparse, local user "hints" to an output colorization with a Convolutional Neural Network (CNN). Rather than using hand-defined rules, the network propagates user edits by fusing low-level cues along with high-level semantic information, learned from large-scale data. We train on a million images, with simulated user inputs. To guide the user towards efficient input selection, the system recommends likely colors based on the input image and current user inputs. The colorization is performed in a single feed-forward pass, enabling real-time use. Even with randomly simulated user inputs, we show that the proposed system helps novice users quickly create realistic colorizations, and offers large improvements in colorization quality with just a minute of use. In addition, we demonstrate that the framework can incorporate other user "hints" to the desired colorization, showing an application to color histogram transfer.

Authors

Richard Zhang

8 papers

Phillip Isola

11 papers

Alexei A. Efros

23 papers

TL;DR

Abstract

Authors

References64 items

Outline Colorization through Tandem Adversarial Networks

Scribbler: Controlling Deep Image Synthesis with Sketch and Color

Image-to-Image Translation with Conditional Adversarial Networks

Deep joint demosaicking and denoising

Generative Visual Manipulation on the Natural Image Manifold

A 4D Light-Field Dataset and CNN Architectures for Material Recognition

Painting style transfer for head portraits using convolutional neural networks

Image Style Transfer Using Convolutional Neural Networks

DeepProp: Extracting Deep Features from a Single Image for Edit Propagation

Context Encoders: Feature Learning by Inpainting

Colorful Image Colorization

Learning Representations for Automatic Colorization

Deep Interactive Object Selection

Deep Colorization

Learning Large-Scale Automatic Image Colorization

Multi-Scale Context Aggregation by Dilated Convolutions

The Fast Bilateral Solver

Learning a Discriminative Model for the Perception of Realism in Composite Images

Palette-based photo recoloring

U-Net: Convolutional Networks for Biomedical Image Segmentation

Holistically-Nested Edge Detection

Image recoloring using geodesic distance based color harmonization

Automatic Photo Adjustment Using Deep Neural Networks

Learning Deep Features for Scene Recognition using Places Database

Material recognition in the wild with the Materials in Context Database

Hypercolumns for object segmentation and fine-grained localization

Very Deep Convolutional Networks for Large-Scale Image Recognition

ImageNet Large Scale Visual Recognition Challenge

AutoStyle: Automatic Style Transfer from Image Collections to Users' Images

Generative Adversarial Nets

Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

A sparse control model for image and video editing

ImageNet classification with deep convolutional neural networks

Manifold preserving edit propagation

Image colorization using similar images

Semantic colorization with internet images

Data-driven image color theme enhancement

Efficient affinity-based edit propagation using K-D tree

Automatic colorization of grayscale images using multiple images on the web

PatchMatch: a randomized correspondence algorithm for structural image editing

A unified active and semi-supervised learning framework for image compression

Automatic Image Colorization Via Multimodal Predictions

AppProp: all-pairs appearance-space edit propagation

Learning to compress images and videos

Manga colorization

An adaptive edge detection based colorization algorithm and its applications

Colorization by example

Colorization using optimization

Transferring color to greyscale images

Color Transfer between Images

Image Analogies

Ahmed Selim, Mohamed Elgharib, and Linda Doyle

Preferred Networks Inc. 2017

Paints Chainer

Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification

Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva

Richard Zhang, Phillip Isola, and Alexei A Efros

Learning to simplify: fully convolutional networks for rough sketch cleanup

Hypercolumns for Object Segmentation and Fine-grained Localization

Gradient-based learning applied to document recognition

On robust estimation of the location parameter

Eurographics Symposium on Rendering (2007) Jan Kautz and Sumanta Pattanaik (Editors) Natural Image Colorization

Eurographics Symposium on Rendering 2008 Scribbleboost: Adding Classification to Edge-aware Interpolation of Local Image and Video Adjustments

Real-Time User-Guided Image Colorization with Learned Deep Priors • 11

Field of Study

Journal Information

Name

Page

Volume

Venue Information

Name

Type

URL

Alternate Names