Lossy Compression for Lossless Prediction
Published in
Neural Information Processing Systems(2021)External Links:
TL
TL;DR
This paper describes the bit-rate required to ensure high performance on all predictive tasks that are invariant under a set of transformations, such as data augmentations, and designs unsupervised objectives for training neural compressors.
Abstract
Most data is automatically collected and only ever"seen"by algorithms. Yet, data compressors preserve perceptual fidelity rather than just the information needed by algorithms performing downstream tasks. In this paper, we characterize the bit-rate required to ensure high performance on all predictive tasks that are invariant under a set of transformations, such as data augmentations. Based on our theory, we design unsupervised objectives for training neural compressors. Using these objectives, we train a generic image compressor that achieves substantial rate savings (more than $1000\times$ on ImageNet) compared to JPEG on 8 datasets, without decreasing downstream classification performance.
References169 items
1
Generative Adversarial Networks
2
VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning
3
Self-supervised Representation Learning with Relative Predictive Coding
4
Barlow Twins: Self-Supervised Learning via Redundancy Reduction
5