HPRN: Holistic Prior-Embedded Relation Network for Spectral Super-Resolution (2021-12-29T00:00:00.000000Z)

TL;DR

This article proposes a novel holistic prior-embedded relation network (HPRN) to integrate comprehensive priors to regularize and optimize the solution space of SSR and develops a transformer-based channel relation module (TCRM), which breaks the habit of employing scalars as the descriptors of channelwise relations in the previous deep feature prior.

Abstract

Spectral super-resolution (SSR) refers to the hyperspectral image (HSI) recovery from an RGB counterpart. Due to the one-to-many nature of the SSR problem, a single RGB image can be reprojected to many HSIs. The key to tackle this ill-posed problem is to plug into multisource prior information such as the natural spatial context prior of RGB images, deep feature prior, or inherent statistical prior of HSIs so as to effectively alleviate the degree of ill-posedness. However, most current approaches only consider the general and limited priors in their customized convolutional neural networks (CNNs), which leads to the inability to guarantee the confidence and fidelity of reconstructed spectra. In this article, we propose a novel holistic prior-embedded relation network (HPRN) to integrate comprehensive priors to regularize and optimize the solution space of SSR. Basically, the core framework is delicately assembled by several multiresidual relation blocks (MRBs) that fully facilitate the transmission and utilization of the low-frequency content prior of RGBs. Innovatively, the semantic prior of RGB inputs is introduced to mark category attributes, and a semantic-driven spatial relation module (SSRM) is invented to perform the feature aggregation of clustered similar ranges for refining recovered characteristics. In addition, we develop a transformer-based channel relation module (TCRM), which breaks the habit of employing scalars as the descriptors of channelwise relations in the previous deep feature prior and replaces them with certain vectors to make the mapping function more robust and smoother. In order to maintain the mathematical correlation and spectral consistency between hyperspectral bands, the second-order prior constraints (SOPCs) are incorporated into the loss function to guide the HSI reconstruction. Finally, extensive experimental results on four benchmarks demonstrate that our HPRN can reach the state-of-the-art performance for SSR quantitatively and qualitatively. Furthermore, the effectiveness and usefulness of the reconstructed spectra are verified by the classification results on the remote sensing dataset. Codes are available at https://github.com/Deep-imagelab/HPRN.

References83 items

Spectral Super-Resolution via Model-Guided Cross-Fusion Network

Hider: A Hyperspectral Image Denoising Transformer With Spatial–Spectral Constraints for Hybrid Noise Removal

Panchromatic and Multispectral Image Fusion via Alternating Reverse Filtering Network

A Spatio-Spectral Fusion Method for Hyperspectral Images Using Residual Hyper-Dense Network

Semisupervised Cross-Scale Graph Prototypical Network for Hyperspectral Image Classification

Semantic-Aware Dehazing Network With Adaptive Feature Fusion

Multi-Scale Single Image Dehazing Using Laplacian and Gaussian Pyramids

Deep Amended Gradient Descent for Efficient Spectral Reconstruction From Single RGB Images

Deep Hybrid 2-D–3-D CNN Based on Dual Second-Order Attention With Camera Spectral Sensitivity Prior for Spectral Super-Resolution

Information Fusion for Classification of Hyperspectral and LiDAR Data Using IP-CNN

SpectralFormer: Rethinking Hyperspectral Image Classification With Transformers

Hyperspectral Image Super-Resolution via Deep Spatiospectral Attention Convolutional Neural Networks

Progressive Spatial–Spectral Joint Network for Hyperspectral Image Reconstruction

Endmember-Guided Unmixing Network (EGU-Net): A General Deep Learning Framework for Self-Supervised Hyperspectral Unmixing

CvT: Introducing Convolutions to Vision Transformers

Pre-Trained Image Processing Transformer

Spectral Response Function-Guided Deep Optimization-Driven Network for Spectral Super-Resolution

Correction to: Single Image Super-Resolution via a Holistic Attention Network

More Diverse Means Better: Multimodal Deep Learning Meets Remote-Sensing Imagery Classification

Hyperspectral Target Detection With RoI Feature Transformation and Multiscale Spectral Attention

Hybrid 2-D–3-D Deep Residual Attentional Network With Structure Tensor Constraints for Spectral Super-Resolution of RGB Images

Reconstruction of Hyperspectral Data From RGB Images With Prior Category Information

Residual Pixel Attention Network for Spectral Reconstruction from RGB Images

Adaptive Weighted Attention Network with Camera Spectral Sensitivity Prior for Spectral Reconstruction from RGB Images

Nonlocal Graph Convolutional Networks for Hyperspectral Image Classification

Hierarchical Regression Network for Spectral Reconstruction from RGB Images

NTIRE 2020 Challenge on Spectral Reconstruction from an RGB Image

Light Weight Residual Dense Attention Net for Spectral Reconstruction from RGB Images

3-D Quasi-Recurrent Neural Network for Hyperspectral Image Denoising

Spatial and Spectral Joint Super-Resolution Using Convolutional Neural Network

Dual Snapshot Hyperspectral Imaging System for 41-Band Spectral Analysis and Stereo Reconstruction

Second-Order Attention Network for Single Image Super-Resolution

Pixel-aware Deep Function-mixture Network for Spectral Super-Resolution

Medical Hyperspectral Image Classification Based on End-to-End Fusion Deep Neural Network

Learning a Low Tensor-Train Rank Representation for Hyperspectral Image Super-Resolution

Spectral Super-resolution for RGB Images using Class-based BP Neural Networks

Joint Camera Spectral Sensitivity Selection and Hyperspectral Image Recovery

Accurate Spectral Super-resolution from Single RGB Image Using Multi-scale CNN

Reconstructing Spectral Images from RGB-Images Using a Convolutional Neural Network

HSCNN+: Advanced CNN-Based Hyperspectral Recovery from RGB Images

NTIRE 2018 Challenge on Spectral Reconstruction from RGB Images

Hyperspectral Image Classification With Imbalanced Data Based on Orthogonal Complement Subspace Projection

Deep Hyperspectral Image Sharpening

Multisource Remote Sensing Data Classification Based on Convolutional Neural Network

Classification of Hyperspectral Imagery Using a New Fully Convolutional Neural Network

Aerial Spectral Super-Resolution using Conditional Adversarial Networks

Automatic differentiation in PyTorch

Adaptive Nonlocal Sparse Representation for Dual-Camera Compressive Hyperspectral Imaging

In Defense of Shallow Learned Spectral Reconstruction from RGB Images

HSCNN: CNN-Based Hyperspectral Image Recovery from Spectrally Undersampled Projections

Attention is All you Need

Learned Spectral Super-Resolution

Hyperspectral CNN for image classification & band selection, with application to face recognition

Sparse Recovery of Hyperspectral Signal from Natural RGB Images

Do It Yourself Hyperspectral Imaging with Everyday Digital Cameras

Single Image Spectral Reconstruction for Multimedia Applications

HyperCam: hyperspectral imaging for ubiquitous computing applications

Adam: A Method for Stochastic Optimization

A+: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution

Hyperspectral Remote Sensing Data Analysis and Future Challenges

SLIC Superpixels Compared to State-of-the-Art Superpixel Methods

Detection and Analysis of the Intestinal Ischemia Using Visible and Invisible Hyperspectral Imaging

Video rate spectral imaging using a coded aperture snapshot spectral imager.

Spatio-spectral reconstruction of the multispectral datacube using sparse recovery

Single disperser design for coded aperture snapshot spectral imaging.

Spectrograph Design Fundamentals

Multispectral Imaging Using Multiplexed Illumination

Single-shot compressive spectral imaging with a dual-disperser architecture.

Classification of hyperspectral remote sensing images with support vector machines

Generalized Mosaicing: Wide Field of View Multispectral Imaging

Imaging Spectroscopy and the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS)

Hyperspectral image classification and dimensionality reduction: an orthogonal subspace projection approach

Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition

Simultaneous acquisition of spectral image information.

Convolutional Neural Networks for Multimodal Remote Sensing Data Classification

HASIC-Net: Hybrid Attentional Convolutional Neural Network With Structure Information Consistency for Spectral Super-Resolution of RGB Images

Supplementary File: Image Super-Resolution with Non-Local Sparse Attention

Introduction to PyTorch

Enhancing Deep Spectral Super-resolution from RGB Images by Enforcing the Metameric Constraint

he has been a Professor with the School of Telecommunications Engineering

Medical hyperspectral imaging: a review

K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation

of Xidian University