Siamese Network for RGB-D Salient Object Detection and Beyond (2020-08-26T00:00:00.000000Z)

TL;DR

The proposed JL-DCF module provides robust saliency feature learning by exploiting cross-modal commonality via a Siamese network, while the DCF module is introduced for complementary feature discovery.

Abstract

Existing RGB-D salient object detection (SOD) models usually treat RGB and depth as independent information and design separate networks for feature extraction from each. Such schemes can easily be constrained by a limited amount of training data or over-reliance on an elaborately designed training process. Inspired by the observation that RGB and depth modalities actually present certain commonality in distinguishing salient objects, a novel joint learning and densely cooperative fusion (<italic/>JL-DCF<italic/>) architecture is designed to learn from both RGB and depth inputs through a shared network backbone, known as the <italic>Siamese architecture</italic>. In this paper, we propose two effective components: joint learning (JL), and densely cooperative fusion (DCF). The JL module provides robust saliency feature learning by exploiting cross-modal commonality via a Siamese network, while the DCF module is introduced for complementary feature discovery. Comprehensive experiments using five popular metrics show that the designed framework yields a robust RGB-D saliency detector with good generalization. As a result, JL-DCF significantly advances the state-of-the-art models by an average of <inline-formula><tex-math notation="LaTeX">$\sim 2.0\%$</tex-math><alternatives><mml:math><mml:mrow><mml:mo>∼</mml:mo><mml:mn>2</mml:mn><mml:mo>.</mml:mo><mml:mn>0</mml:mn><mml:mo>%</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="fan-ieq1-3073689.gif"/></alternatives></inline-formula> (max F-measure) across seven challenging datasets. In addition, we show that <italic/>JL-DCF<italic/> is readily applicable to other related multi-modal detection tasks, including RGB-T (thermal infrared) SOD and video SOD, achieving comparable or even better performance against state-of-the-art methods. We also link <italic/>JL-DCF<italic/> to the RGB-D semantic segmentation field, showing its capability of outperforming several semantic segmentation models on the task of RGB-D SOD. These facts further confirm that the proposed framework could offer a potential solution for various applications and provide more insight into the cross-modal complementarity task.

References154 items

Revisiting Video Saliency Prediction in the Deep Learning Era

RGBT Salient Object Detection: Benchmark and A Novel Cooperative Ranking Approach

Salient Object Detection by Spatiotemporal and Semantic Features in Real-Time Video Processing Systems

Uncertainty Inspired RGB-D Saliency Detection

Going From RGB to RGBD Saliency: A Depth-Guided Transformation Model

Malleable 2.5D Convolution: Learning Receptive Fields along the Depth-axis for RGB-D Scene Parsing

Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation

Bifurcated Backbone Strategy for RGB-D Salient Object Detection

BBS-Net: RGB-D Salient Object Detection with a Bifurcated Backbone Strategy Network

Video Salient Object Detection via Robust Seeds Extraction and Multi-Graphs Manifold Propagation

Learning Selective Self-Mutual Attention for RGB-D Saliency Detection

A2dele: Adaptive and Attentive Depth Distiller for Efficient RGB-D Salient Object Detection

Multi-Mutual Consistency Induced Transfer Subspace Learning for Human Motion Segmentation

Select, Supplement and Focus for RGB-D Saliency Detection

Is Depth Really Necessary for Salient Object Detection?

Multi-interactive Encoder-decoder Network for RGBT Salient Object Detection

Bilateral Attention Network for RGB-D Salient Object Detection

JL-DCF: Joint Learning and Densely-Cooperative Fusion Framework for RGB-D Salient Object Detection

UC-Net: Uncertainty Inspired RGB-D Saliency Detection via Conditional Variational Autoencoders

Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentation

Pyramid Constrained Self-Attention Network for Fast Video Salient Object Detection

Siamese Box Adaptive Network for Visual Tracking

Adaptive Graph Convolutional Network With Attention Graph Clustering for Co-Saliency Detection

RGB-T Salient Object Detection via Fusing Multi-Level CNN Features

Deep Learning for Visual Tracking: A Comprehensive Survey

Siam R-CNN: Visual Tracking by Re-Detection

SiamCAR: Siamese Fully Convolutional Classification and Regression for Visual Tracking

Depth-Induced Multi-Scale Recurrent Attention Network for Saliency Detection

Zero-Shot Video Object Segmentation via Attentive Graph Neural Networks

Salient object detection for RGB-D image by single stream recurrent convolution neural network

Deepside: A general deep framework for salient object detection

3D Neighborhood Convolution: Learning Depth-Aware Features for RGB-D and RGB Semantic Segmentation

EGNet: Edge Guidance Network for Salient Object Detection

Deep Light-field-driven Saliency Detection from a Single View

Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarks

RFBNet: Deep Multimodal Networks with Residual Fusion Blocks for RGB-D Semantic Segmentation

Contrast Prior and Fluid Pyramid Integration for RGBD Salient Object Detection

Shifting More Attention to Video Salient Object Detection

Attentive Feedback Network for Boundary-Aware Salient Object Detection

An Iterative and Cooperative Top-Down and Bottom-Up Inference Network for Salient Object Detection

Learning Unsupervised Video Object Segmentation Through Visual Attention

Salient Object Detection With Pyramid Attention and Salient Edges

RGB-T Image Saliency Detection via Collaborative Graph Learning

Video Saliency Detection via Graph Clustering With Motion Energy and Spatiotemporal Objectness

Video Saliency Detection via Sparsity-Based Reconstruction and Propagation

A Simple Pooling-Based Design for Real-Time Salient Object Detection

Salient Object Detection in the Deep Learning Era: An In-Depth Survey

Res2Net: A New Multi-Scale Backbone Architecture

Multi-modal fusion network with multi-scale multi-path and cross-modal interactions for RGB-D salient object detection

Three-Stream Attention-Aware Network for RGB-D Salient Object Detection

Adaptive Fusion for RGB-D Salient Object Detection

Fast Online Object Tracking and Segmentation: A Unifying Approach

CNNs-Based RGB-D Saliency Detection via Cross-View Transfer and Multiview Fusion

RGBD Salient Object Detection using Spatially Coherent Deep Learning Framework

Deep Affinity Network for Multiple Object Tracking

Dual Attention Network for Scene Segmentation

Pyramid Dilated Deeper ConvLSTM for Video Salient Object Detection

Unsupervised Video Object Segmentation with Motion-Based Bilateral Networks

StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction

CBAM: Convolutional Block Attention Module

Progressively Complementarity-Aware Fusion Network for RGB-D Salient Object Detection

Flow Guided Recurrent Neural Encoder for Video Salient Object Detection

Salient Object Detection Driven by Fixation Prediction

Fast Video Object Segmentation by Reference-Guided Mask Propagation

Progressive Attention Guided Recurrent Network for Salient Object Detection

Fast and Accurate Online Video Object Segmentation via Tracking Parts

Enhanced-alignment Measure for Binary Foreground Map Evaluation

PDNet: Prior-Model Guided Depth-Enhanced Network for Salient Object Detection

Depth-aware CNN for RGB-D Segmentation

Salient Objects in Clutter: Bringing Salient Object Detection to the Foreground

Stereoscopic saliency model using contrast and depth-guided-background prior

Saliency Detection for Unconstrained Videos Using Superpixel-Level Graph and Spatiotemporal Propagation

Non-local Neural Networks

RDFNet: RGB-D Multi-level Residual Feature Fusion for Indoor Semantic Segmentation

Deep Cropping via Attention Box Prediction and Aesthetics Assessment

3D Graph Neural Networks for RGBD Semantic Segmentation

Cascaded Feature Network for Semantic Segmentation of RGB-D Images

Squeeze-and-Excitation Networks

Learning Uncertain Convolutional Features for Accurate Saliency Detection

Amulet: Aggregating Multi-level Convolutional Features for Salient Object Detection

Structure-Measure: A New Way to Evaluate Foreground Maps

Saliency Detection on Light Field

Learning to Detect Salient Objects with Image-Level Supervision

Locality-Sensitive Deconvolution Networks with Gated Fusion for RGB-D Indoor Semantic Segmentation

Depth-Aware Salient Object Detection and Segmentation via Multiscale Discriminative Saliency Fusion and Bootstrap Learning

Learning RGB-D Salient Object Detection Using Background Enclosure, Depth Contrast, and Top-Down Features

Co-Saliency Detection via a Self-Paced Multiple-Instance Learning Framework

Residual Attention Network for Image Classification

Salient Object Segmentation via Effective Integration of Saliency and Objectness

RGB-D Salient Object Detection via Minimum Barrier Distance Transform and Saliency Fusion

End-to-End Learning of Geometry and Context for Deep Stereo Regression

Video Salient Object Detection via Fully Convolutional Networks

FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

Pyramid Scene Parsing Network

Semantic Scene Completion from a Single Depth Image

FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture

Creating a Color Map to be used to Convert a Gray Image to Color Image

Deeply Supervised Salient Object Detection with Short Connections

A Benchmark Dataset and Saliency-Guided Stacked Autoencoders for Video-Based Salient Object Detection

100

Correspondence Driven Saliency Transfer

101

Image Co-segmentation via Saliency Co-fusion

102

Densely Connected Convolutional Networks

103

Learning Common and Specific Features for RGB-D Semantic Segmentation with Deconvolutional Networks

104

RGBD Salient Object Detection via Deep Fusion

105

Salient object detection for RGB-D image via saliency evolution

106

Fully-Convolutional Siamese Networks for Object Tracking

107

Local Background Enclosure for RGB-D Salient Object Detection

108

DHSNet: Deep Hierarchical Saliency Network for Salient Object Detection

109

Efficient Deep Learning for Stereo Matching

110

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

111

A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation

112

Detection of Co-salient Objects by Looking Deep and Wide

113

Saliency Detection for Stereoscopic Images Based on Depth Confidence Analysis and Multiple Cues Fusion

114

LSTM-CF: Unifying Context Modeling and Fusion with LSTMs for RGB-D Scene Labeling

115

Learning to Refine Object Segments

116

Deep Residual Learning for Image Recognition

117

DeepSaliency: Multi-Task Deep Neural Network Model for Salient Object Detection

118

Consistent Video Saliency Using Local Gradient Flow Optimization and Global Refinement

119

U-Net: Convolutional Networks for Biomedical Image Segmentation

120

Spatiotemporal Saliency Detection for Video Sequences Based on Random Walk With Restart

121

Learning to compare image patches via convolutional neural networks

122

Database Saliency for Fast Image Retrieval

123

Salient Object Detection: A Benchmark

124

Salient object detection: A survey

125

Fully convolutional networks for semantic segmentation

126

Depth saliency based on anisotropic center-surround difference

127

Going deeper with convolutions

128

Computing the stereo matching cost with a convolutional neural network

129

RGBD Salient Object Detection: A Benchmark and Algorithms

130

Very Deep Convolutional Networks for Large-Scale Image Recognition

131

Learning Rich Features from RGB-D Images for Object Detection and Segmentation

132

Depth Enhanced Saliency Detection Method

133

Caffe: Convolutional Architecture for Fast Feature Embedding

134

DeepFace: Closing the Gap to Human-Level Performance in Face Verification

135

A Model of Visual Attention for Natural Image Retrieval

136

Indoor Semantic Segmentation using depth information

137

Unsupervised Salient Object Segmentation Based on Kernel Density Estimation and Two-Phase Graph Cut

138

Leveraging stereopsis for saliency analysis

139

Saliency filters: Contrast based filtering for salient region detection

140

Global contrast based salient region detection

141

Importance filtering for image retargeting

142

Context-aware saliency detection

143

Sketch2Photo: internet image montage

144

Frequency-tuned salient region detection

145

Attention Based Auto Image Cropping

146

Unsupervised extraction of visual attention objects in color images

147

A generic framework of user attention model and its application in video summarization

148

Learning a similarity metric discriminatively, with application to face verification

149

Is bottom-up attention useful for object recognition?

150

Signature Verification Using A "Siamese" Time Delay Neural Network

151

Siamese Neural Networks for One-Shot Image Recognition

152

A Novel Multiresolution Spatiotemporal Saliency Detection Model and Its Applications in Image and Video Compression

153

A framework for visual saliency detection with applications to image thumbnailing

154