Training Debiased Subnetworks with Contrastive Weight Pruning (2022-10-11T00:00:00.000000Z)

TL;DR

A Debiased Contrastive Weight Pruning algorithm is proposed, which probes unbiased subnetworks without expensive group annotations and elucidates the importance of bias-conflicting samples on structure learning.

Abstract

Neural networks are often biased to spuriously correlated features that provide misleading statistical evidence that does not generalize. This raises an interesting question: “Does an optimal unbiased functional subnetwork exist in a severely biased network? If so, how to extract such subnetwork?” While empirical evidence has been accumulated about the existence of such unbiased subnetworks, these observations are mainly based on the guidance of ground-truth unbiased samples. Thus, it is unexplored how to discover the optimal subnetworks with biased training datasets in practice. To address this, here we first present our theoretical insight that alerts potential limitations of existing algorithms in exploring unbiased subnetworks in the presence of strong spurious correlations. We then further elucidate the importance of bias-conflicting samples on structure learning. Motivated by these observations, we propose a Debiased Contrastive Weight Pruning (DCWP) algorithm, which probes unbiased subnetworks without expensive group annotations. Experimental results demonstrate that our approach significantly outperforms state-of-the-art debiasing methods despite its considerable reduction in the number of parameters.

Authors

Jong-Chul Ye

2 papers

Geon Yeong Park

1 papers

Sangmin Lee

1 papers

TL;DR

Abstract

Authors

References44 items

Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations

Robust Contrastive Learning Using Negative Samples with Diminished Semantics

NAS-OoD: Neural Architecture Search for Out-of-Distribution Generalization

BiaSwap: Removing Dataset Bias with Bias-Tailored Swapping Augmentation

Just Train Twice: Improving Group Robustness without Training Group Information

Learning Debiased Representation via Disentangled Feature Augmentation

A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness

Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?

EnD: Entangling and Disentangling deep representations for bias correction

No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems

Understanding the Failure Modes of Out-of-Distribution Generalization

Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks

Model Patching: Closing the Subgroup Performance Gap with Data Augmentation

An Investigation of Why Overparameterization Exacerbates Spurious Correlations

Supervised Contrastive Learning

Picking Winning Tickets Before Training by Preserving Gradient Flow

A Simple Framework for Contrastive Learning of Visual Representations

Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks

Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation

Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization

Learning De-biased Representations with Biased Representations

Adversarial Examples Are Not Bugs, They Are Features

REPAIR: Removing Representation Bias by Dataset Resampling

Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

A Style-Based Generator Architecture for Generative Adversarial Networks

ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning

Robustness May Be at Odds with Accuracy

Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Annotation Artifacts in Natural Language Inference Data

Does Distributionally Robust Supervised Learning Give Robust Classifiers?

Categorical Reparameterization with Gumbel-Softmax

Certifying and Removing Disparate Impact

Deep Learning Face Attributes in the Wild

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

Learning

Appendix for “Unbiased Classification Through Bias-Contrastive and Bias-Balanced Learning”

Learning Debiased Models with Dynamic Gradient Alignment and Bias-conflicting Sample Mining

Learning from Failure: De-biasing Classifier from Biased Classifier

An investigation of why overparameterization 7937 Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply. exacerbates spurious correlations

Translation tutorial: 21 fairness definitions and their politics

We use the official implementations of Rebias, LfF, DisEnt and JTT released by authors, and reproduce EnD References

Specifically, majority of training images correspond to either young women (i.e., aged 10-29) or old men (i.e

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names