SpectralFormer: Rethinking Hyperspectral Image Classification With Transformers (2021-07-07T00:00:00.000000Z)

TL;DR

This work rethink HS image classification from a sequential perspective with transformers and proposes a novel backbone network called SpectralFormer, which is capable of learning spectrally local sequence information from neighboring bands of HS images, yielding groupwise spectral embeddings.

Abstract

Hyperspectral (HS) images are characterized by approximately contiguous spectral information, enabling the fine identification of materials by capturing subtle spectral discrepancies. Due to their excellent locally contextual modeling ability, convolutional neural networks (CNNs) have been proven to be a powerful feature extractor in HS image classification. However, CNNs fail to mine and represent the sequence attributes of spectral signatures well due to the limitations of their inherent network backbone. To solve this issue, we rethink HS image classification from a sequential perspective with transformers and propose a novel backbone network called SpectralFormer. Beyond bandwise representations in classic transformers, SpectralFormer is capable of learning spectrally local sequence information from neighboring bands of HS images, yielding groupwise spectral embeddings. More significantly, to reduce the possibility of losing valuable information in the layerwise propagation process, we devise a cross-layer skip connection to convey memory-like components from shallow to deep layers by adaptively learning to fuse “soft” residuals across layers. It is worth noting that the proposed SpectralFormer is a highly flexible backbone network, which can be applicable to both pixelwise and patchwise inputs. We evaluate the classification performance of the proposed SpectralFormer on three HS datasets by conducting extensive experiments, showing the superiority over classic transformers and achieving a significant improvement in comparison with state-of-the-art backbone networks. The codes of this work will be available at https://github.com/danfenghong/IEEE_TGRS_SpectralFormer for the sake of reproducibility.

Authors

References43 items

Low-Rank and Sparse Representation for Hyperspectral Image Processing: A review

Interpretable Hyperspectral Artificial Intelligence: When nonconvex modeling meets hyperspectral remote sensing

Endmember-Guided Unmixing Network (EGU-Net): A General Deep Learning Framework for Self-Supervised Hyperspectral Unmixing

Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth

l₀-l₁ Hybrid Total Variation Regularization and its Applications on Hyperspectral Image Mixed Noise Removal and Compressed Sensing

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Joint and Progressive Subspace Analysis (JPSA) With Spatial–Spectral Manifold Alignment for Semisupervised Hyperspectral Dimensionality Reduction

Ensemble EMD-Based Spectral-Spatial Feature Extraction for Hyperspectral Image Classification

More Diverse Means Better: Multimodal Deep Learning Meets Remote-Sensing Imagery Classification

Graph Convolutional Networks for Hyperspectral Image Classification

Semisupervised Hypergraph Discriminant Learning for Dimensionality Reduction of Hyperspectral Image

Rethinking Positional Encoding in Language Pre-training

Joint Classification of Hyperspectral and LiDAR Data Using Hierarchical Random Walk and Deep CNN Architecture

Feature Extraction for Hyperspectral Imagery: The Evolution From Shallow to Deep: Overview and Toolbox

Improved Collaborative Non-Negative Matrix Factorization and Total Variation for Hyperspectral Unmixing

Feature Extraction for Classification of Hyperspectral and LiDAR Data Using Patch-to-Patch CNN

Invariant Attribute Profiles: A Spatial-Frequency Joint Feature Extractor for Hyperspectral Image Classification

Learning to propagate labels on graphs: An iterative multitask regression framework for semi-supervised hyperspectral dimensionality reduction

Deep Learning for Hyperspectral Image Classification: An Overview

Capsule Networks for Hyperspectral Image Classification

Self-Paced Joint Sparse Representation for the Classification of Hyperspectral Images

Nonconvex-Sparsity and Nonlocal-Smoothness-Based Blind Hyperspectral Unmixing

An Augmented Linear Mixing Model to Address Spectral Variability for Hyperspectral Unmixing

A Robust PCA Approach With Noise Structure Learning and Spatial–Spectral Low-Rank Modeling for Hyperspectral Image Restoration

Dimensionality Reduction of Hyperspectral Image Using Spatial Regularized Local Graph Discriminant Embedding

Generative Adversarial Networks for Hyperspectral Image Classification

Non-local Neural Networks

Decoupled Weight Decay Regularization

Hyperspectral Image Restoration Via Total Variation Regularized Low-Rank Tensor Decomposition

Attention is All you Need

Learning a Robust Local Manifold Representation for Hyperspectral Dimensionality Reduction

Cascaded Recurrent Neural Networks for Hyperspectral Image Classification

Deep Feature Extraction and Classification of Hyperspectral Images Based on Convolutional Neural Networks

TL;DR

Abstract

Authors

References43 items

Low-Rank and Sparse Representation for Hyperspectral Image Processing: A review

Interpretable Hyperspectral Artificial Intelligence: When nonconvex modeling meets hyperspectral remote sensing

Endmember-Guided Unmixing Network (EGU-Net): A General Deep Learning Framework for Self-Supervised Hyperspectral Unmixing

Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth

l₀-l₁ Hybrid Total Variation Regularization and its Applications on Hyperspectral Image Mixed Noise Removal and Compressed Sensing

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Joint and Progressive Subspace Analysis (JPSA) With Spatial–Spectral Manifold Alignment for Semisupervised Hyperspectral Dimensionality Reduction

Ensemble EMD-Based Spectral-Spatial Feature Extraction for Hyperspectral Image Classification

More Diverse Means Better: Multimodal Deep Learning Meets Remote-Sensing Imagery Classification

Graph Convolutional Networks for Hyperspectral Image Classification

Semisupervised Hypergraph Discriminant Learning for Dimensionality Reduction of Hyperspectral Image

Rethinking Positional Encoding in Language Pre-training

Joint Classification of Hyperspectral and LiDAR Data Using Hierarchical Random Walk and Deep CNN Architecture

Feature Extraction for Hyperspectral Imagery: The Evolution From Shallow to Deep: Overview and Toolbox

Improved Collaborative Non-Negative Matrix Factorization and Total Variation for Hyperspectral Unmixing

Feature Extraction for Classification of Hyperspectral and LiDAR Data Using Patch-to-Patch CNN

Invariant Attribute Profiles: A Spatial-Frequency Joint Feature Extractor for Hyperspectral Image Classification

Learning to propagate labels on graphs: An iterative multitask regression framework for semi-supervised hyperspectral dimensionality reduction

Deep Learning for Hyperspectral Image Classification: An Overview

Capsule Networks for Hyperspectral Image Classification

Self-Paced Joint Sparse Representation for the Classification of Hyperspectral Images

Nonconvex-Sparsity and Nonlocal-Smoothness-Based Blind Hyperspectral Unmixing

An Augmented Linear Mixing Model to Address Spectral Variability for Hyperspectral Unmixing

A Robust PCA Approach With Noise Structure Learning and Spatial–Spectral Low-Rank Modeling for Hyperspectral Image Restoration

Dimensionality Reduction of Hyperspectral Image Using Spatial Regularized Local Graph Discriminant Embedding

Generative Adversarial Networks for Hyperspectral Image Classification

Non-local Neural Networks

Decoupled Weight Decay Regularization

Hyperspectral Image Restoration Via Total Variation Regularized Low-Rank Tensor Decomposition

Attention is All you Need

Learning a Robust Local Manifold Representation for Hyperspectral Dimensionality Reduction

Cascaded Recurrent Neural Networks for Hyperspectral Image Classification

Deep Feature Extraction and Classification of Hyperspectral Images Based on Convolutional Neural Networks

Gaussian Error Linear Units (GELUs)

Deep Residual Learning for Image Recognition

U-Net: Convolutional Networks for Biomedical Image Segmentation

Adam: A Method for Stochastic Optimization

Deep Learning-Based Classification of Hyperspectral Data

Principal Component Analysis (PCA)

Learning long-term dependencies with gradient descent is difficult

CyCU-Net: Cycle-Consistency Unmixing Network by Learning Cascaded Autoencoders

Deep Learning

the skip connection

Field of Study

Journal Information

Name

Page

Volume

Venue Information

Name

Type

URL

Alternate Names