When and how CNNs generalize to out-of-distribution category-viewpoint combinations (2020-07-15T00:00:00.000000Z)

TL;DR

It is shown that increasing the number of in-distribution combinations substantially improves generalization to OOD combinations, even with the same amount of training data, and it is demonstrated that such OOD generalization is facilitated by the neural mechanism of specialization.

Abstract

Object recognition and viewpoint estimation lie at the heart of visual understanding. Recent works suggest that convolutional neural networks (CNNs) fail to generalize to out-of-distribution (OOD) category-viewpoint combinations, ie. combinations not seen during training. In this paper, we investigate when and how such OOD generalization may be possible by evaluating CNNs trained to classify both object category and 3D viewpoint on OOD combinations, and identifying the neural mechanisms that facilitate such OOD generalization. We show that increasing the number of in-distribution combinations (ie. data diversity) substantially improves generalization to OOD combinations, even with the same amount of training data. We compare learning category and viewpoint in separate and shared network architectures, and observe starkly different trends on in-distribution and OOD combinations, ie. while shared networks are helpful in-distribution, separate networks significantly outperform shared ones at OOD combinations. Finally, we demonstrate that such OOD generalization is facilitated by the neural mechanism of specialization, ie. the emergence of two types of neurons -- neurons selective to category and invariant to viewpoint, and vice versa.

Authors

H. Pfister

7 papers

F. Durand

7 papers

Spandan Madan

2 papers

TL;DR

Abstract

Authors

References71 items

CPS++: Improving Class-level 6D Pose and Shape Estimation From Monocular Images With Self-Supervised Learning

Frivolous Units: Wider Networks Are Not Really That Wide

Deep double descent: where bigger models and more data hurt

SPair-71k: A Large-scale Benchmark for Semantic Correspondence

Physics-Based Rendering for Improving Robustness to Rain

Which Tasks Should Be Learned Together in Multi-task Learning?

nuScenes: A Multimodal Dataset for Autonomous Driving

Gauge Equivariant Convolutional Networks and the Icosahedral CNN

Minimal Images in Deep Neural Networks: Fragile Object Recognition in Natural Images

Task representations in neural networks trained to perform many cognitive tasks

Strike (With) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects

Viewpoint Estimation-Insights & Model

Revisiting the Importance of Individual Units in CNNs via Ablation

Why do deep convolutional networks generalize so poorly to small image transformations?

Pixels, Voxels, and Views: A Study of Shape Representations for Single View 3D Object Shape Prediction

3D Pose Estimation and 3D Model Retrieval for Objects in the Wild

A Unified Framework for Multi-View Multi-Class Object Pose Estimation

On the importance of single directions for generalization

Spherical CNNs

Exploring the Landscape of Spatial Robustness

Convolutional Networks for Object Category and 3D Pose Estimation from 2D Images

PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes

CARLA: An Open Urban Driving Simulator

A Survey on Multi-Task Learning

An Overview of Multi-Task Learning in Deep Neural Networks

Emergence of Invariance and Disentanglement in Deep Representations

Network Dissection: Quantifying Interpretability of Deep Visual Representations

Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks

Aggregated Residual Transformations for Deep Neural Networks

Visual Cortex and Deep Networks: Learning Invariant Representations

Crafting a multi-task CNN for viewpoint estimation

UnrealCV: Connecting Computer Vision to Unreal Engine

Densely Connected Convolutional Networks

Improved Deep Learning of Object Category Using Pose Information

iLab-20M: A Large-Scale Controlled Object Dataset to Investigate Deep Learning

A Comparative Analysis and Study of Multiview CNN Models for Joint Object Categorization and Pose Estimation

Wide Residual Networks

Deep Residual Learning for Image Recognition

Rethinking the Inception Architecture for Computer Vision

Is object localization for free? - Weakly-supervised learning with convolutional neural networks

Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views

Pose Induction for Novel Object Categories

Adam: A Method for Stochastic Optimization

Object Detectors Emerge in Deep Scene CNNs

Viewpoints and keypoints

Is 2D Information Enough For Viewpoint Estimation?

Beyond PASCAL: A benchmark for 3D object detection in the wild

Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps

3D Object Representations for Fine-Grained Categorization

Visualizing and Understanding Convolutional Networks

Unbiased look at dataset bias

Selectivity and Tolerance (“Invariance”) Both Increase as Visual Information Propagates from Cortical Area V4 to IT

Measuring Invariances in Deep Networks

Pose estimation for category specific multiview object localization

3D generic object categorization, localization and pose estimation

An empirical evaluation of deep architectures on problems with many factors of variation

Invariant visual representation by single neurons in the human brain

Just One View: Invariances in Inferotemporal Cell Tuning

Multitask Learning

3D Object Recognition: A Model of View-Tuned Neurons

A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information

Learning, invariance, and generalization in high-order neural networks.

Blender Foundation, Stichting Blender Foundation

Esri CityEngine -a 3D city modeling software for urban design, visual effects

Blender - a 3D modelling and rendering package

ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models

An Overview of Multi-task Learning

Simultaneous Object Classification and Viewpoint Estimation using Deep Multi-task Convolutional Neural Network

Improving Object Classification using Pose Information

Gradient-based learning applied to document recognition

and ReLU activations

Field of Study