Deep CNNs Meet Global Covariance Pooling: Better Representation and Generalization (2019-04-15T00:00:00.000000Z)

TL;DR

The proposed MPN-COV conforms to a robust covariance estimator, very suitable for scenario of high dimension and small sample size, and can be regarded as Power-Euclidean metric between covariances, effectively exploiting their geometry.

Abstract

Compared with global average pooling in existing deep convolutional neural networks (CNNs), global covariance pooling can capture richer statistics of deep features, having potential for improving representation and generalization abilities of deep CNNs. However, integration of global covariance pooling into deep CNNs brings two challenges: (1) robust covariance estimation given deep features of high dimension and small sample size; (2) appropriate usage of geometry of covariances. To address these challenges, we propose a <italic>global Matrix Power Normalized COVariance (MPN-COV) Pooling</italic>. Our MPN-COV conforms to a robust covariance estimator, very suitable for scenario of high dimension and small sample size. It can also be regarded as Power-Euclidean metric between covariances, effectively exploiting their geometry. Furthermore, a global Gaussian embedding network is proposed to incorporate first-order statistics into MPN-COV. For fast training of MPN-COV networks, we implement an iterative matrix square root normalization, avoiding GPU unfriendly eigen-decomposition inherent in MPN-COV. Additionally, progressive <inline-formula><tex-math notation="LaTeX">$1\times 1$</tex-math><alternatives><mml:math><mml:mrow><mml:mn>1</mml:mn><mml:mo>×</mml:mo><mml:mn>1</mml:mn></mml:mrow></mml:math><inline-graphic xlink:href="li-ieq1-2974833.gif"/></alternatives></inline-formula> convolutions and group convolution are introduced to compress covariance representations. The proposed methods are highly modular, readily plugged into existing deep CNNs. Extensive experiments are conducted on large-scale object classification, scene categorization, fine-grained visual recognition and texture classification, showing our methods outperform the counterparts and obtain state-of-the-art performance.

References78 items

CBAM: Convolutional Block Attention Module

Places: A 10 Million Image Database for Scene Recognition

Bilinear Convolutional Neural Networks for Fine-Grained Visual Recognition

Covariance Pooling for Facial Expression Recognition

Statistically Motivated Second Order Pooling

Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization

Higher-Order Integration of Hierarchical Convolutional Activations for Fine-Grained Visual Categorization

Deep Scene Image Classification with the MFAFVNet

Squeeze-and-Excitation Networks

Improved Bilinear Pooling with CNNs

Kernel Pooling for Convolutional Neural Networks

G2DeNet: Global Gaussian Distribution Embedding Network and Its Application to Visual Recognition

FASON: First and Second Order Information Fusion Network for Texture Recognition

Local Log-Euclidean Multivariate Gaussian Descriptor and Its Application to Image Classification

Is Second-Order Information Helpful for Large-Scale Visual Recognition?

SORT: Second-Order Response Transform for Visual Recognition

Higher-Order Occurrence Pooling for Bags-of-Words: Visual Concept Detection

Deep TEN: Texture Encoding Network

Object based Scene Representations using Fisher Scores of Local Subspace Projections

Factorized Bilinear Models for Image Recognition

Aggregated Residual Transformations for Deep Neural Networks

Low-Rank Bilinear Pooling for Fine-Grained Classification

Densely Connected Convolutional Networks

Geometry-Aware Similarity Learning on SPD Manifolds for Visual Recognition

Deep FisherNet for Object Classification

RAID-G: Robust Estimation of Approximate Infinite Dimensional Gaussian with Application to Material Recognition

Deep Residual Learning for Image Recognition

Matrix Backpropagation for Deep Networks with Structured Layers

NetVLAD: CNN Architecture for Weakly Supervised Place Recognition

Compact Bilinear Pooling

Training Deep Networks with Structured Layers by Matrix Backpropagation

Towards effective codebookless model for image classification

Deep filter banks for texture recognition and segmentation

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Free-Form Region Description with Second-Order Pooling

Bilinear CNN Models for Fine-Grained Visual Recognition

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

MatConvNet: Convolutional Neural Networks for MATLAB

Encoding High Dimensional Local Features by Sparse Coding Based Fisher Vectors

Fully convolutional networks for semantic segmentation

Going deeper with convolutions

Very Deep Convolutional Networks for Large-Scale Image Recognition

Elementary Estimators for Sparse Covariance Matrices and other Structured Moments

Return of the Devil in the Details: Delving Deep into Convolutional Nets

Multi-scale Orderless Pooling of Deep Convolutional Activation Features

OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks

Network In Network

3D Object Representations for Fine-Grained Categorization

A Novel Earth Mover's Distance Methodology for Image Matching with Gaussian Mixture Models

Describing Textures in the Wild

Visualizing and Understanding Convolutional Networks

Optimal Shrinkage of Eigenvalues in the Spiked Covariance Model.

Fine-Grained Visual Classification of Aircraft

ImageNet classification with deep convolutional neural networks

Aggregating Local Image Descriptors into Compact Codes

Representation Learning: A Review and New Perspectives

The Caltech-UCSD Birds-200-2011 Dataset

Global Gaussian approach for scene categorization using information geometry

Functions of matrices: Theory and computation

Non-Euclidean statistics for covariance matrices, with applications to diffusion tensor imaging

Tensor Decompositions and Applications

Recognizing indoor scenes

ImageNet: A large-scale hierarchical image database

Shape of Gaussians as feature descriptors

Functions of matrices - theory and computation

Region Covariance: A Fast Descriptor for Detection and Classification

Fast and Simple Calculus on Tensors in the Log-Euclidean Framework

Distinctive Image Features from Scale-Invariant Keypoints

A well-conditioned estimator for large-dimensional covariance matrices

Shrinkage Estimators for Covariance Matrices

Multivariate Normal Distributions Parametrized as a Riemannian Symmetric Space

Lectures on the theory of estimation of many parameters

Author manuscript, published in "International Journal of Computer Vision (2013)" International Journal of Computer Vision manuscript No. (will be inserted by the editor) Image Classification with the Fisher Vector: Theory and Practice

Index Terms

A Riemannian Framework for Tensor Computing

Distinctive Image Features from Scale-Invariant Keypoints

A Riemannian geometry of the multivariate normal model