TS-NET: Combining Modality Specific and Common Features for Multimodal Patch Matching
TL
TL;DR
This work proposes a three-stream architecture, dubbed as TS-Net, combining the benefits of the Siamese and Pseudo-Siamese networks, and shows that adding extra constraints in the intermediate layers of such networks further boosts the performance.
Abstract
Multimodal patch matching addresses the problem of finding the correspondences between image patches from two different modalities, e.g. RGB vs sketch or RGB vs near-infrared. The comparison of patches of different modalities can be done by discovering the information common to both modalities (Siamese like approaches) or the modality-specific information (Pseudo-Siamese like approaches). We observed that none of these two scenarios is optimal. This motivates us to propose a three-stream architecture, dubbed as TS-Net, combining the benefits of the two. In addition, we show that adding extra constraints in the intermediate layers of such networks further boosts the performance. Experimentations on three multimodal datasets show significant performance gains in comparison with Siamese and Pseudo-Siamese networks† †Codes and resources available at http://github.com/ensv/TS-Net.
References20 items
1
L2-Net: Deep Learning of Discriminative Patch Descriptor in Euclidean Space
2
Exploiting Deep Matching and SAR Data for the Geo-Localization Accuracy Improvement of Optical Satellite Images
3
Cross-spectral image patch similarity using convolutional neural network
4
Cross-Spectral Local Descriptors via Quadruplet Network
5