DBAT: Dynamic Backward Attention Transformer for Material Segmentation with Cross-Resolution Patches (2023-05-06T00:00:00.000000Z)

TL;DR

By analysing the cross-resolution features and the attention weights, this paper interprets how the DBAT learns from image patches and shows that the proposed model can extract material-related features better than other methods.

Abstract

The objective of dense material segmentation is to identify the material categories for every image pixel. Recent studies adopt image patches to extract material features. Although the trained networks can improve the segmentation performance, their methods choose a fixed patch resolution which fails to take into account the variation in pixel area covered by each material. In this paper, we propose the Dynamic Backward Attention Transformer (DBAT) to aggregate cross-resolution features. The DBAT takes cropped image patches as input and gradually increases the patch resolution by merging adjacent patches at each transformer stage, instead of fixing the patch resolution during training. We explicitly gather the intermediate features extracted from cross-resolution patches and merge them dynamically with predicted attention masks. Experiments show that our DBAT achieves an accuracy of 86.85%, which is the best performance among state-of-the-art real-time models. Like other successful deep learning solutions with complex architectures, the DBAT also suffers from lack of interpretability. To address this problem, this paper examines the properties that the DBAT makes use of. By analysing the cross-resolution features and the attention weights, this paper interprets how the DBAT learns from image patches. We further align features to semantic labels, performing network dissection, to infer that the proposed model can extract material-related features better than other methods. We show that the DBAT model is more robust to network initialisation, and yields fewer variable predictions compared to other models. The project code is available at https://github.com/heng-yuwen/Dynamic-Backward-Attention-Transformer.

Authors

Yuwen Heng

2 papers

S. Dasmahapatra

2 papers

Hansung Kim

3 papers

TL;DR

Abstract

Authors

References72 items

Material Recognition for Immersive Interactions in Virtual/Augmented Reality

Room Acoustic Properties Estimation from a Single 360° Photo

Multimodal Material Segmentation

Effects of Similarity Score Functions in Attention Mechanisms on the Performance of Neural Question Answering Systems

Swin Transformer V2: Scaling Up Capacity and Resolution

Do Vision Transformers See Like Convolutional Neural Networks?

Interpretable Compositional Convolutional Neural Networks

Improved Transformer Net for Hyperspectral Image Classification

Segmenter: Transformer for Semantic Segmentation

CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification

Learning Transferable Visual Models From Natural Language Supervision

ACCLVOS: Atrous Convolution with Spatial-Temporal ConvLSTM for Video Object Segmentation

Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

Training data-efficient image transformers & distillation through attention

Transformer Interpretability Beyond Attention Visualization

Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Understanding the role of individual units in a deep neural network

An Encoder-Decoder Network Based FCN Architecture for Semantic Segmentation

Using a Patch-Wise M-Net Convolutional Neural Network for Tissue Segmentation in Brain MRI Images

Rethinking Pre-training and Self-training

DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution

End-to-End Object Detection with Transformers

Simultaneous Material Segmentation and 3D Reconstruction in Industrial Scenarios

CNN Explainer: Learning Convolutional Neural Networks with Interactive Visualization

Exploring Self-Attention for Image Recognition

ResNeSt: Split-Attention Networks

Semantic Labeling in Remote Sensing Corpora Using Feature Fusion-Based Enhanced Global Convolutional Network with High-Resolution Representations and Depthwise Atrous Convolution

Weakly-Supervised Semantic Segmentation via Self-training

Dynamic Convolution: Attention Over Convolution Kernels

Context‐Aware Mixed Reality: A Learning‐Based Framework for Semantic‐Level Interaction

Tool Substitution with Shape and Material Reasoning Using Dual Neural Networks

Polynomial Learning Rate Policy with Warm Restart for Deep Neural Network

Leveraging Local and Global Patterns for Self-Attention Networks

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Boundary Loss for Remote Sensing Imagery Semantic Segmentation

Similarity of Neural Network Representations Revisited

Immersive Spatial Audio Reproduction for VR/AR Using Room Acoustic Modelling from 360° Images

Attention Branch Network: Learning of Attention Mechanism for Visual Explanation

A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation

GAN Dissection: Visualizing and Understanding Generative Adversarial Networks

Parallel Feature Pyramid Network for Object Detection

Synthesizing Game Audio Using Deep Neural Networks

Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi-Supervised Semantic Segmentation

Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation

Recognizing Material Properties from Images

Image processing pipeline for segmentation and material classification based on multispectral high dynamic range polarimetric images.

Interpreting Deep Visual Representations via Network Dissection

Interpretable Convolutional Neural Networks

PiCANet: Learning Pixel-Wise Contextual Attention for Saliency Detection

Interpreting CNN knowledge via an Explanatory Graph

Scene Parsing through ADE20K Dataset

Attention is All you Need

Network Dissection: Quantifying Interpretability of Deep Visual Representations

Feature Pyramid Networks for Object Detection

Pyramid Scene Parsing Network

Material Recognition from Local Appearance in Global Context

Semantic Understanding of Scenes Through the ADE20K Dataset

Deep Residual Learning for Image Recognition

Material recognition in the wild with the Materials in Context Database

Efficient mini-batch training for stochastic optimization

ImageNet classification with deep convolutional neural networks

ImageNet: A large-scale hierarchical image database

CAM-SegNet: A Context-Aware Dense Material Segmentation Network for Sparsely Labelled Datasets

Enhancing Material Features Using Dynamic Backward Attention on Cross-Resolution Patches

Hyperspectral Classification via Global-Local Hierarchical Weighting Fusion Network

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

“OpenSurfaces: A richly annotated catalog of surface appearance,”

Feature Selection via Dependence Maximization

The comparison between networks trained for object and material tasks, in terms of the percentage of neurons aligned to each semantic concepts

ResNet-152 (b) ResNest-101 (c) EfficientNet-b5 (d) Swin-t

Field of Study

Journal Information

Name

Volume