Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning (2020-12-17T00:00:00.000000Z)

TL;DR

It is proved that self-distillation can also be viewed as implicitly combining ensemble and knowledge distillation to improve test accuracy, and it sheds light on how ensemble works in deep learning in a way that is completely different from traditional theorems.

Abstract

We formally study how Ensemble of deep learning models can improve test accuracy, and how the superior performance of ensemble can be distilled into a single model using Knowledge Distillation. We consider the challenging case where the ensemble is simply an average of the outputs of a few independently trained neural networks with the SAME architecture, trained using the SAME algorithm on the SAME data set, and they only differ by the random seeds used in the initialization. We empirically show that ensemble/knowledge distillation in deep learning works very differently from traditional learning theory, especially differently from ensemble of random feature mappings or the neural-tangent-kernel feature mappings, and is potentially out of the scope of existing theorems. Thus, to properly understand ensemble and knowledge distillation in deep learning, we develop a theory showing that when data has a structure we refer to as "multi-view", then ensemble of independently trained neural networks can provably improve test accuracy, and such superior test accuracy can also be provably distilled into a single model by training a single model to match the output of the ensemble instead of the true label. Our result sheds light on how ensemble works in deep learning in a way that is completely different from traditional theorems, and how the "dark knowledge" is hidden in the outputs of the ensemble -- that can be used in knowledge distillation -- comparing to the true data labels. In the end, we prove that self-distillation can also be viewed as implicitly combining ensemble and knowledge distillation to improve test accuracy.

Authors

Zeyuan Allen-Zhu

2 papers

Yuanzhi Li

2 papers

TL;DR

Abstract

Authors

References100 items

Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK

Feature Purification: How Adversarial Training Performs Robust Deep Learning

When can Wasserstein GANs minimize Wasserstein Distance?

Making Method of Moments Great Again? -- How can GANs learn the target distribution

Neural Kernels Without Tangents

Self-Distillation Amplifies Regularization in Hilbert Space

Backward Feature Correction: How Deep Learning Performs Deep Learning

Ensembles for feature selection: A review and future trends

Finite Depth and Width Corrections to the Neural Tangent Kernel

Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks

Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks

What Can ResNet Learn Efficiently, Going Beyond Kernels?

Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation

Linearized two-layers neural networks in high dimension

On Exact Computation with an Infinitely Wide Neural Net

Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding

On the Power and Limitations of Random Features for Understanding Neural Networks

Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation

Toward Moderate Overparameterization: Global Convergence Guarantees for Training Shallow Neural Networks

Can SGD Learn Recurrent Neural Networks with Provable Generalization?

Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks

Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks

Gradient Descent Finds Global Minima of Deep Neural Networks

Learning Two Layer Rectified Neural Networks in Polynomial Time

On the Convergence Rate of Training Recurrent Neural Networks

Gradient Descent Provably Optimizes Over-parameterized Neural Networks

Learning Two-layer Neural Networks with Symmetric Inputs

A Convergence Theory for Deep Learning via Over-Parameterization

Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data

Feature selection in machine learning: A new perspective

Neural Tangent Kernel: Convergence and Generalization in Neural Networks

Knowledge Distillation by On-the-Fly Native Ensemble

Born Again Neural Networks

Gradient Descent for One-Hidden-Layer Neural Networks: Polynomial Convergence and SQ Lower Bounds

Polynomial Convergence of Gradient Descent for Training One-Hidden-Layer Neural Networks

A high-bias, low-variance introduction to Machine Learning for physicists

Algorithmic Regularization in Over-parameterized Matrix Sensing and Neural Networks with Quadratic Activations

Learning One-hidden-layer Neural Networks with Landscape Design

Theoretical properties of the global optimizer of two layer neural network

Efficient Knowledge Distillation from an Ensemble of Teachers

Theoretical Insights Into the Optimization Landscape of Over-Parameterized Shallow Neural Networks

Provable Alternating Gradient Descent for Non-negative Matrix Factorization with Strong Correlations

Recovery Guarantees for One-hidden-layer Neural Networks

Convergence Analysis of Two-layer Neural Networks with ReLU Activation

Knowledge distillation across ensembles of multilingual models for low-resource languages

An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis

SGD Learns the Conjugate Kernel Class of the Network

Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs

Ensemble Distillation for Neural Machine Translation

Recovery Guarantee of Non-negative Matrix Factorization via Alternating Updates

Diverse Neural Network Learns True Target Functions

Diversity Leads to Generalization in Neural Networks

Distilling Knowledge from Ensembles of Neural Networks for Speech Recognition

No bad local minima: Data independent training error guarantees for multilayer neural networks

Wide Residual Networks

Deep Learning without Poor Local Minima

Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity

Distilling the Knowledge in a Neural Network

Comparison and anti-concentration bounds for maxima of Gaussian random vectors

Fast decorrelated neural network ensembles with random weights

Comparison and anti-concentration bounds for maxima of Gaussian random vectors

Semantic Road Segmentation via Multi-scale Ensembles of Learned Features

A Review on Ensembles for the Class Imbalance Problem: Bagging-, Boosting-, and Hybrid-Based Approaches

Neural network with ensembles

Ensemble-based classifiers

Pattern Classification Using Ensemble Methods

On Feature Selection, Bias-Variance, and Bagging

Data Mining with Decision Trees - Theory and Applications

Dynamic Weighted Majority: An Ensemble Method for Drifting Concepts

Rotation Forest: A New Classifier Ensemble Method

Managing Diversity in Regression Ensembles

An experimental bias-variance analysis of SVM ensembles based on resampling techniques

Combining Pattern Classifiers: Methods and Algorithms

Diversity in search strategies for ensemble feature selection

Bias-Variance Analysis of Support Vector Machines for the Development of SVM-Based Ensemble Methods

Diversity creation methods: a survey and categorisation