Restormer: Efficient Transformer for High-Resolution Image Restoration (2021-11-18T00:00:00.000000Z)

TL;DR

This work proposes an efficient Transformer model by making several key designs in the building blocks (multi-head attention and feed-forward network) such that it can capture long-range pixel interactions, while still remaining applicable to large images.

Abstract

Since convolutional neural networks (CNNs) perform well at learning generalizable image priors from large-scale data, these models have been extensively applied to image restoration and related tasks. Recently, another class of neural architectures, Transformers, have shown significant performance gains on natural language and high-level vision tasks. While the Transformer model mitigates the shortcomings of CNNs (i.e., limited receptive field and inadaptability to input content), its computational complexity grows quadratically with the spatial resolution, therefore making it infeasible to apply to most image restoration tasks involving high-resolution images. In this work, we propose an efficient Transformer model by making several key designs in the building blocks (multi-head attention and feed-forward network) such that it can capture long-range pixel interactions, while still remaining applicable to large images. Our model, named Restoration Transformer (Restormer), achieves state-of-the-art results on several image restoration tasks, including image deraining, single-image motion deblurring, defocus deblurring (single-image and dual-pixel data), and image denoising (Gaussian grayscale/color denoising, and real image denoising). The source code and pre-trained models are available at https://github.com/swz30/Restormer.

Authors

Salman Hameed Khan

14 papers

Munawar Hayat

12 papers

Ming-Hsuan Yang

23 papers

TL;DR

Abstract

Authors

References109 items

Burst Image Restoration and Enhancement

Dynamic Attentive Graph Learning for Image Restoration

SwinIR: Image Restoration Using Swin Transformer

Single Image Defocus Deblurring Using Kernel-Sharing Parallel Atrous Convolutions

Spatially-Adaptive Image Restoration using Distortion-Guided Networks

Rethinking Coarse-to-Fine Approach in Single Image Deblurring

Uformer: A General U-Shaped Transformer for Image Restoration

Adaptive Consistency Prior based Deep Network for Image Denoising

NTIRE 2021 Challenge for Defocus Deblurring Using Dual-pixel Images: Methods and Results

Iterative Filter Adaptive Network for Single Image Defocus Deblurring

SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers

NTIRE 2021 Challenge on Image Deblurring

Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

Colorization Transformer

Multi-Stage Progressive Image Restoration

Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Transformers in Vision: A Survey

Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

Training data-efficient image transformers & distillation through attention

Learning to Reduce Defocus Blur by Realistically Modeling Dual-Pixel Data

Pre-Trained Image Processing Transformer

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Deformable DETR: Deformable Transformers for End-to-End Object Detection

Plug-and-Play Image Restoration With Deep Denoiser Prior

Real-World Blur Dataset for Learning and Benchmarking Deblurring Algorithms

Dual Adversarial Network: Toward Real-world Noise Removal and Noise Generation

Learning Texture Transformer Network for Image Super-Resolution

Language Models are Few-Shot Learners

End-to-End Object Detection with Transformers

Defocus Deblurring Using Dual-Pixel Data

Spatially-Attentive Patch-Hierarchical Network for Adaptive Motion Deblurring

Deblurring by Realistic Blurring

Multi-Scale Progressive Fusion Network for Single Image Deraining

CycleISP: Real Image Restoration via Improved Data Synthesis

Learning Enriched Features for Real Image Restoration and Enhancement

Transfer Learning From Synthetic to Real-Noise Denoising With Adaptive Instance Normalization

Spatial-Adaptive Network for Single Image Denoising

Image denoising using deep CNN with batch renormalization

Deep Learning on Image Denoising: An overview

Multi-Temporal Recurrent Neural Networks For Progressive Non-Uniform Single Image Deblurring With Incremental Temporal Training

Self-Guided Network for Fast Image Denoising

Human-Aware Motion Deblurring

Variational Denoising Network: Toward Blind Noise Modeling and Removal

Mix & Match: training convnets with mixed image sizes for improved accuracy, speed and scale resiliency

DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Better

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Densely Residual Laplacian Super-Resolution

Robust and interpretable blind image denoising via bias-free convolutional neural networks

Dynamic Scene Deblurring With Parameter Selective Sharing and Nested Skip Connections

Deep Defocus Map Estimation Using Domain Adaptation

FOCNet: A Fractional Optimal Control Network for Image Denoising

Uncertainty Guided Multi-Scale Residual Learning-Using a Cycle Spinning CNN for Single Image De-Raining

Dilated Residual Networks with Symmetric Skip Connection for image denoising

NTIRE 2019 Challenge on Real Image Denoising: Methods and Results

NTIRE 2019 Challenge on Image Enhancement: Methods and Results

A Deep Journey into Super-resolution

Real Image Denoising With Feature Attention

Deep Stacked Hierarchical Multi-Patch Network for Image Deblurring

Dual Residual Networks Leveraging the Potential of Paired Operations for Image Restoration

Single Image Deraining: A Comprehensive Benchmark Analysis

Residual Non-local Attention Networks for Image Restoration

Progressive Image Deraining Networks: A Better and Simpler Baseline

Residual Dense Network for Image Restoration

Semi-Supervised Transfer Learning for Image Rain Removal

Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining

Toward Convolutional Blind Denoising of Real Photographs

Image Super-Resolution Using Very Deep Residual Channel Attention Networks

Identifying Recurring Patterns with Deep Neural Networks for Natural Image Denoising

Non-Local Recurrent Network for Image Restoration

Dynamic Scene Deblurring Using Spatially Variant Recurrent Neural Networks

A High-Quality Denoising Dataset for Smartphone Cameras

Self-Attention Generative Adversarial Networks

Multi-level Wavelet-CNN for Image Restoration

Edge-Based Defocus Blur Estimation With Adaptive Scale Selection

Density-Aware Single Image De-raining Using a Multi-stream Dense Network