Perceptual Losses for Real-Time Style Transfer and Super-Resolution (2016-03-27T00:00:00.000000Z)

TL;DR

This work considers image transformation problems, and proposes the use of perceptual loss functions for training feed-forward networks for image transformation tasks, and shows results on image style transfer, where aFeed-forward network is trained to solve the optimization problem proposed by Gatys et al. in real-time.

Abstract

We consider image transformation problems, where an input image is transformed into an output image. Recent methods for such problems typically train feed-forward convolutional neural networks using a per-pixel loss between the output and ground-truth images. Parallel work has shown that high-quality images can be generated by defining and optimizing perceptual loss functions based on high-level features extracted from pretrained networks. We combine the benefits of both approaches, and propose the use of perceptual loss functions for training feed-forward networks for image transformation tasks. We show results on image style transfer, where a feed-forward network is trained to solve the optimization problem proposed by Gatys et al. in real-time. Compared to the optimization-based method, our network gives similar qualitative results but is three orders of magnitude faster. We also experiment with single-image super-resolution, where replacing a per-pixel loss with a perceptual loss gives visually pleasing results.

Authors

Justin Johnson

12 papers

Li Fei-Fei

37 papers

Alexandre Alahi

17 papers

TL;DR

Abstract

Authors

References71 items

Image Style Transfer Using Convolutional Neural Networks

Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks

Colorful Image Colorization

Texture Networks: Feed-forward Synthesis of Textures and Stylized Images

Deep Residual Learning for Image Recognition

Full-reference visual quality assessment for synthetic images: A subjective study

Deep Colorization

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

A Neural Algorithm of Artistic Style

Understanding Neural Networks Through Deep Visualization

Inverting Visual Representations with Convolutional Networks

Fast and accurate image upscaling with super-resolution forests

Single image super-resolution from transformed self-exemplars

Texture Synthesis Using Convolutional Neural Networks

Learning Deconvolution Network for Semantic Segmentation

Conditional Random Fields as Recurrent Neural Networks

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Image Super-Resolution Using Deep Convolutional Networks

Adam: A Method for Stochastic Optimization

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Understanding deep image representations by inverting them

Deep convolutional neural fields for depth estimation from a single image

Designing deep networks for surface normal estimation

Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture

Fully convolutional networks for semantic segmentation

A+: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution

cuDNN: Efficient Primitives for Deep Learning

Learning a Deep Convolutional Network for Image Super-Resolution

Single-Image Super-Resolution: A Benchmark

Very Deep Convolutional Networks for Large-Scale Image Recognition

ImageNet Large Scale Visual Recognition Challenge

Recurrent Convolutional Neural Networks for Scene Labeling

Depth Map Prediction from a Single Image using a Multi-Scale Deep Network

Microsoft COCO: Common Objects in Context

Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps

Intriguing properties of neural networks

HOGgles: Visualizing Object Detection Features

Learning Hierarchical Features for Scene Labeling

Benchmarking of quality metrics on ultra-high definition video sequences

Beta Process Joint Dictionary Learning for Coupled Feature Spaces with Application to Single Image Super-Resolution

Fast Image Super-Resolution Based on In-Place Example Regression

Recurrent Convolutional Neural Networks for Scene Parsing

From Bits to Images: Inversion of Local Binary Descriptors

Beyond bits: Reconstructing images from Local Binary Descriptors

Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding

FSIM: A Feature Similarity Index for Image Quality Assessment

Image and video upscaling from local self-examples

Image Super-Resolution Via Sparse Representation

Non-Local Kernel Regression for Image and Video Restoration

Robust Web Image/Video Super-Resolution

On Single Image Scale-Up Using Sparse-Representations

Single-Image Super-Resolution Using Sparse Regression and Natural Image Prior

Super-resolution from a single image

Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures

Fast image/video upsampling

Image super-resolution as sparse representation of raw image patches

Image super-resolution using gradient profile prior

Scope of validity of PSNR in image/video quality assessment

Image Superresolution Using Support Vector Regression

A Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms

Image information and visual quality

Image up-sampling using total-variation regularization with a new observation model

Super-resolution through neighbor embedding

Image quality assessment: from error visibility to structural similarity

Image hallucination with primal sketch priors

Example-Based Super-Resolution

Improving resolution by image registration

Training and investigating residual nets

Future Data and Security Engineering

Torch7: A Matlab-like Environment for Machine Learning

Revised Selected Papers

Field of Study

Journal Information

Name

Volume

Venue Information