Ensemble Distillation for Robust Model Fusion in Federated Learning (2020-06-12T00:00:00.000000Z)

TL;DR

This work proposes ensemble distillation for model fusion, i.e. training the central classifier through unlabeled data on the outputs of the models from the clients, which allows flexible aggregation over heterogeneous client models that can differ e.g. in size, numerical precision or structure.

Abstract

Federated Learning (FL) is a machine learning setting where many devices collaboratively train a machine learning model while keeping the training data decentralized. In most of the current training schemes the central model is refined by averaging the parameters of the server model and the updated parameters from the client side. However, directly averaging model parameters is only possible if all models have the same structure and size, which could be a restrictive constraint in many scenarios. In this work we investigate more powerful and more flexible aggregation schemes for FL. Specifically, we propose ensemble distillation for model fusion, i.e. training the central classifier through unlabeled data on the outputs of the models from the clients. This knowledge distillation technique mitigates privacy risk and cost to the same extent as the baseline FL algorithms, but allows flexible aggregation over heterogeneous client models that can differ e.g. in size, numerical precision or structure. We show in extensive empirical experiments on various CV/NLP datasets (CIFAR-10/100, ImageNet, AG News, SST2) and settings (heterogeneous models/data) that the server model can be trained much faster, requiring fewer communication rounds than any existing FL technique so far.

Authors

Tao Lin

2 papers

Lingjing Kong

1 papers

Sebastian U. Stich

2 papers

TL;DR

Abstract

Authors

References86 items

Distilled One-Shot Federated Learning

Federated Model Distillation with Noise-Free Differential Privacy

Group Knowledge Transfer: Collaborative Training of Large CNNs on the Edge

Distributed Gradient Methods for Convex Machine Learning Problems in Networks: Distributed Optimization

Dynamic Model Pruning with Feedback

Adaptive Personalized Federated Learning

Federated Visual Classification with Real-World Data Distribution

Adaptive Federated Optimization

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Federated Learning with Matched Averaging

Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer

Contrastive Representation Distillation

Overcoming Forgetting in Federated Learning on Non-IID Data

SCAFFOLD: Stochastic Controlled Averaging for Federated Learning

SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning

Model Fusion via Optimal Transport

FedMD: Heterogenous Federated Learning via Model Distillation

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

The Non-IID Data Quagmire of Decentralized Machine Learning

FEED: Feature-level Ensemble for Knowledge Distillation

Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification

Federated Learning: Challenges, Methods, and Future Directions

Similarity-Preserving Knowledge Distillation

Distilled Person Re-Identification: Towards a More Scalable System

Fair Resource Allocation in Federated Learning

LIT: Learned Intermediate Representation Training for Model Compression

Bayesian Nonparametric Federated Learning of Neural Networks

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Zero-shot Knowledge Transfer via Adversarial Belief Matching

Zero-Shot Knowledge Distillation in Deep Networks

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

Knowledge Flow: Improve Upon Your Teachers

Variational Information Distillation for Knowledge Transfer

Diversity With Cooperation: Ensemble Methods for Few-Shot Classification

One-Shot Federated Learning

A Simple Baseline for Bayesian Uncertainty in Deep Learning

Towards Federated Learning at Scale: System Design

Agnostic Federated Learning

Federated Optimization in Heterogeneous Networks

LEAF: A Benchmark for Federated Settings

Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data

Large Scale GAN Training for High Fidelity Natural Image Synthesis

Don't Use Large Mini-Batches, Use Local SGD

ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design

Algorithms and Theory for Multiple-Source Adaptation

Born Again Neural Networks

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Group Normalization

Averaging Weights Leads to Wider Optima and Better Generalization

Large scale distributed neural network training through online distillation

Paraphrasing Complex Network: Network Compression via Factor Transfer

MobileNetV2: Inverted Residuals and Linear Bottlenecks

Differentially Private Federated Learning: A Client Level Perspective

Learning from Multiple Teacher Networks

A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets

Federated Multi-Task Learning

Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer

Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations

Densely Connected Convolutional Networks

XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks

Communication-Efficient Learning of Deep Networks from Decentralized Data

Deep Residual Learning for Image Recognition

Character-level Convolutional Networks for Text Classification

Privacy-preserving deep learning

Distilling the Knowledge in a Neural Network

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

FitNets: Hints for Thin Deep Nets

Very Deep Convolutional Networks for Large-Scale Image Recognition

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

ImageNet classification with deep convolutional neural networks

A theory of learning from different domains

Domain Adaptation with Multiple Sources

Model compression

Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy

2020a) for model update (quantized/pruned model is used during the backward pass)