Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021-09-15T00:00:00.000000Z)

TL;DR

This work proposes a new method called large mask inpainting (LaMa), which generalizes surprisingly well to resolutions that are higher than those seen at train time, and achieves this at lower parameter&time costs than the competitive baselines.

Abstract

Modern image inpainting systems, despite the significant progress, often struggle with large missing areas, complex geometric structures, and high-resolution images. We find that one of the main reasons for that is the lack of an effective receptive field in both the inpainting network and the loss function. To alleviate this issue, we propose a new method called large mask inpainting (LaMa). LaMa is based on i) a new inpainting network architecture that uses fast Fourier convolutions (FFCs), which have the image-wide receptive field; ii) a high receptive field perceptual loss; iii) large training masks, which unlocks the potential of the first two components. Our inpainting network improves the state-of-the-art across a range of datasets and achieves excellent performance even in challenging scenarios, e.g. completion of periodic structures. Our model generalizes surprisingly well to resolutions that are higher than those seen at train time, and achieves this at lower parameter&time costs than the competitive baselines. The code is available at https://github.com/saic-mdal/lama.

Authors

Arsenii Ashukha

3 papers

V. Lempitsky

14 papers

Roman Suvorov

1 papers

TL;DR

Abstract

Authors

References69 items

Generative Adversarial Networks

Global Filter Networks for Image Classification

FNet: Mixing Tokens with Fourier Transforms

Image Inpainting by End-to-End Cascaded Refinement With Mask Awareness

Aggregated Contextual Transformations for High-Resolution Image Inpainting

Large Scale Image Completion via Co-Modulated Generative Adversarial Networks

Taming Transformers for High-Resolution Image Synthesis

Image Inpainting with Learnable Feature Imputation

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations

Recurrent Feature Reasoning for Image Inpainting

High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling

Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting

Learning to Incorporate Structure Knowledge for Image Inpainting

Image Fine-grained Inpainting

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Region Normalization for Image Inpainting

Regionwise Generative Adversarial Image Inpainting for Large Missing Areas

Image Inpainting With Learnable Bidirectional Attention Maps

StructureFlow: Image Inpainting via Structure-Aware Appearance Flow

Progressive Image Inpainting with Full-Resolution Residual Network

PEPSI : Fast Image Inpainting With Parallel Decoding Network

Coherent Semantic Attention for Image Inpainting

PEPSI++: Fast and Lightweight Network for Image Inpainting

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting

SDC – Stacked Dilated Convolution: A Unified Descriptor Network for Dense Matching Tasks

Pluralistic Image Completion

Foreground-Aware Image Inpainting

EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning

A Style-Based Generator Architecture for Generative Adversarial Networks

Image Inpainting via Generative Multi-column Convolutional Neural Networks

Semantic Image Inpainting with Progressive Generative Networks

ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

Free-Form Image Inpainting With Gated Convolution

Places: A 10 Million Image Database for Scene Recognition

Lightning

SPG-Net: Segmentation Prediction and Guidance Network for Image Inpainting

Image Inpainting for Irregular Holes Using Partial Convolutions

Shift-Net: Image Inpainting via Deep Feature Rearrangement

Generative Image Inpainting with Contextual Attention

Which Training Methods for GANs do actually Converge?

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients

Contextual-Based Image Inpainting: Infer, Match, and Translate

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Globally and locally consistent image completion

Understanding the Effective Receptive Field in Deep Convolutional Neural Networks

High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis

Image-to-Image Translation with Conditional Adversarial Networks

Semantic Image Inpainting with Deep Generative Models

Context Encoders: Feature Learning by Inpainting

Perceptual Losses for Real-Time Style Transfer and Super-Resolution

Deep Residual Learning for Image Recognition

Multi-Scale Context Aggregation by Dilated Convolutions

U-Net: Convolutional Networks for Biomedical Image Segmentation

Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

Adam: A Method for Stochastic Optimization

ImageNet Large Scale Visual Recognition Challenge

Fast Fourier Transform

Scene Completion Using Millions of Photographs

Object removal by exemplar-based inpainting

Simultaneous structure and texture image inpainting

Improving generalization performance using double backpropagation

Fast Fourier Convolution

Hydra - a framework for elegantly configuring complex applications

Pytorch lightning. GitHub

Advances in Neural Information Processing Systems , volume 29

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type