In Search of Lost Domain Generalization (2020-07-02T00:00:00.000000Z)

TL;DR

This paper implements DomainBed, a testbed for domain generalization including seven multi-domain datasets, nine baseline algorithms, and three model selection criteria, and finds that, when carefully implemented, empirical risk minimization shows state-of-the-art performance across all datasets.

Abstract

The goal of domain generalization algorithms is to predict well on distributions different from those seen during training. While a myriad of domain generalization algorithms exist, inconsistencies in experimental conditions -- datasets, architectures, and model selection criteria -- render fair and realistic comparisons difficult. In this paper, we are interested in understanding how useful domain generalization algorithms are in realistic settings. As a first step, we realize that model selection is non-trivial for domain generalization tasks. Contrary to prior work, we argue that domain generalization algorithms without a model selection strategy should be regarded as incomplete. Next, we implement DomainBed, a testbed for domain generalization including seven multi-domain datasets, nine baseline algorithms, and three model selection criteria. We conduct extensive experiments using DomainBed and find that, when carefully implemented, empirical risk minimization shows state-of-the-art performance across all datasets. Looking forward, we hope that the release of DomainBed, along with contributions from fellow researchers, will streamline reproducible and rigorous research in domain generalization.

Authors

David Lopez-Paz

6 papers

Ishaan Gulrajani

7 papers

TL;DR

Abstract

Authors

References135 items

Learning to Balance Specificity and Invariance for In and Out of Domain Generalization

Learning from Extrinsic and Intrinsic Supervisions for Domain Generalization

Adaptive Risk Minimization: A Meta-Learning Approach for Tackling Group Shift

Self-Challenging Improves Cross-Domain Generalization

The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization

Heterogeneous Domain Generalization Via Domain Mixup

Sequential Learning for Domain Generalization

Improving out-of-distribution generalization via multi-task self-supervised pretraining

Deep Domain-Adversarial Image Generation for Domain Generalisation

Out-of-Distribution Generalization via Risk Extrapolation (REx)

Unshuffling Data for Improved Generalization in Visual Question Answering

Invariant Risk Minimization Games

Improve Unsupervised Domain Adaptation with Mixup Training

Causality matters in medical imaging

Adversarial Domain Adaptation with Domain Mixup

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Correlation-aware Adversarial Domain Adaptation and Generalization

Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization

Domain Generalization Using a Mixture of Multiple Latent Domains

Adversarial target-invariant representation learning for domain generalization

Domain Generalization via Model-Agnostic Learning of Semantic Features

Reducing Domain Gap via Style-Agnostic Networks

Towards Robust CNN-based Object Detection through Augmentation with Synthetic Rain Variations

Towards Shape Biased Unsupervised Representation Learning for Domain Generalization

Hidden Covariate Shift: A Minimal Assumption For Domain Adaptation

Learning to Optimize Domain Specific Normalization for Domain Generalization

Invariant Risk Minimization

Domain Generalization via Multidomain Discriminant Analysis

When Unseen Domain Generalization is Unnecessary? Rethinking Data Augmentation

DIVA: Domain Invariant Variational Autoencoders

A Generalization Error Bound for Multi-class Domain Generalization

Adversarial Invariant Feature Learning with Accuracy Constraint for Domain Generalization

Domain Generalization by Solving Jigsaw Puzzles

Support and Invertibility in Domain-Invariant Representations

Feature-Critic Networks for Heterogeneous Domain Generalization

Episodic Training for Domain Generalization

On Learning Invariant Representation for Domain Adaptation

Multi-Component Image Translation for Deep Domain Generalization

A Survey of Unsupervised Deep Domain Adaptation

Moment Matching for Multi-Source Domain Adaptation

MetaReg: Towards Domain Generalization using Meta-Regularization

Strike (With) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects

Unsupervised domain adaptation for medical imaging segmentation with self-ensembling

Dark Model Adaptation: Semantic Image Segmentation from Daytime to Nighttime

Domain Generalization with Domain-Specific Aggregation Modules

ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

Learning Robust Representations by Projecting Superficial Statistics Out

Deep Domain Generalization via Conditional Invariant Adversarial Networks

Hallucinating Agnostic Images to Generalize Across Domains

Recognition in Terra Incognita

Best Sources Forward: Domain Generalization through Source-Specific Nets

Domain Generalization with Adversarial Feature Learning

Generalizing to Unseen Domains via Adversarial Data Augmentation

Domain Generalization via Conditional Invariant Representations

Deep learning for segmentation of brain tumors: Impact of cross‐institutional training and testing

Robust Place Categorization With Deep Domain Generalization

Generalizing Across Domains via Cross-Gradient Training

ConvNets and ImageNet Beyond Accuracy: Understanding Mistakes and Uncovering Biases

Domain Generalization by Marginal Transfer Learning

mixup: Beyond Empirical Risk Minimization

Learning to Generalize: Meta-Learning for Domain Generalization

Deeper, Broader and Artier Domain Generalization

Unified Deep Supervised Domain Adaptation and Generalization

Deep Hashing Network for Unsupervised Domain Adaptation

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Does Distributionally Robust Supervised Learning Give Robust Classifiers?

Deep CORAL: Correlation Alignment for Deep Domain Adaptation

Kernel Mean Embedding of Distributions: A Review and Beyonds

Deep Residual Learning for Image Recognition

Return of Frustratingly Easy Domain Adaptation

Scatter Component Analysis: A Unified Framework for Domain Adaptation and Domain Generalization

Domain Generalization for Object Recognition with Multi-task Autoencoders

The reusable holdout: Preserving validity in adaptive data analysis

Invariant Models for Causal Transfer Learning

Domain-Adversarial Training of Neural Networks

Visual Domain Adaptation: A survey of recent advances