mixup: Beyond Empirical Risk Minimization (2017-10-25T00:00:00.000000Z)

TL;DR

This work proposes mixup, a simple learning principle that trains a neural network on convex combinations of pairs of examples and their labels, which improves the generalization of state-of-the-art neural network architectures.

Abstract

Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples. In this work, we propose mixup, a simple learning principle to alleviate these issues. In essence, mixup trains a neural network on convex combinations of pairs of examples and their labels. By doing so, mixup regularizes the neural network to favor simple linear behavior in-between training examples. Our experiments on the ImageNet-2012, CIFAR-10, CIFAR-100, Google commands and UCI datasets show that mixup improves the generalization of state-of-the-art neural network architectures. We also find that mixup reduces the memorization of corrupt labels, increases the robustness to adversarial examples, and stabilizes the training of generative adversarial networks.

Authors

Yann Dauphin

4 papers

Hongyi Zhang

1 papers

Moustapha Cissé

1 papers

TL;DR

Abstract

Authors

References39 items

Statistical Learning Theory

Random Erasing Data Augmentation

Spectrally-normalized margin bounds for neural networks

A Closer Look at Memorization in Deep Networks

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

Sobolev Training for Neural Networks

Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation

Parseval Networks: Improving Robustness to Adversarial Examples

Improved Training of Wasserstein GANs

Nearly-tight VC-dimension bounds for piecewise linear neural networks

Dataset Augmentation in Feature Space

Regularizing Neural Networks by Penalizing Confident Output Distributions

Aggregated Residual Transformations for Deep Neural Networks

Understanding deep learning requires rethinking generalization

Densely Connected Convolutional Networks

Wide Residual Networks

Identity Mappings in Deep Residual Networks

Mastering the game of Go with deep neural networks and tree search

Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin

Rethinking the Inception Architecture for Computer Vision

Adam: A Method for Stochastic Optimization

Striving for Simplicity: The All Convolutional Net

Explaining and Harnessing Adversarial Examples

Very Deep Convolutional Networks for Large-Scale Image Recognition

ImageNet Large Scale Visual Recognition Challenge

Intriguing properties of neural networks

One billion word benchmark for measuring progress in statistical language modeling

Speech recognition with deep recurrent neural networks

ImageNet classification with deep convolutional neural networks

Improving generalization performance using double backpropagation

GENERATIVE ADVERSARIAL NETS

Dropout: a simple way to prevent neural networks from overfitting

UCI machine learning repository

FUNDAMENTAL TECHNOLOGIES IN MODERN SPEECH RECOGNITION Digital Object Identifier 10.1109/MSP.2012.2205597

SMOTE: Synthetic Minority Over-sampling Technique

Vicinal Risk Minimization

Gradient-based learning applied to document recognition

Transformation Invariance in Pattern Recognition-Tangent Distance and Tangent Propagation

Chervonenkis: On the uniform convergence of relative frequencies of events to their probabilities

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names