FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction (2022-12-03T00:00:00.000000Z)

TL;DR

FedRolex is proposed, a partial training (PT)-based approach that enables model-heterogeneous FL and can train a global server model larger than the largest client model and reduces the gap between model- heterogeneous and model-homogeneous FL, especially under the large-model large-dataset regime.

Abstract

Most cross-device federated learning (FL) studies focus on the model-homogeneous setting where the global server model and local client models are identical. However, such constraint not only excludes low-end clients who would otherwise make unique contributions to model training but also restrains clients from training large models due to on-device resource bottlenecks. In this work, we propose FedRolex, a partial training (PT)-based approach that enables model-heterogeneous FL and can train a global server model larger than the largest client model. At its core, FedRolex employs a rolling sub-model extraction scheme that allows different parts of the global server model to be evenly trained, which mitigates the client drift induced by the inconsistency between individual client models and server model architectures. We show that FedRolex outperforms state-of-the-art PT-based model-heterogeneous FL methods (e.g. Federated Dropout) and reduces the gap between model-heterogeneous and model-homogeneous FL, especially under the large-model large-dataset regime. In addition, we provide theoretical statistical analysis on its advantage over Federated Dropout and evaluate FedRolex on an emulated real-world device distribution to show that FedRolex can enhance the inclusiveness of FL and boost the performance of low-end devices that would otherwise not benefit from FL. Our code is available at: https://github.com/AIoT-MLSys-Lab/FedRolex

Authors

Samiul Alam

1 papers

Luyang Liu

1 papers

Ming Yan

1 papers

TL;DR

Abstract

Authors

References34 items

Federated Select: A Primitive for Communication- and Memory-Efficient Federated Learning

Does Federated Dropout actually work?

Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning

Scaling Language Model Size in Cross-Device Federated Learning

Partial Variable Training for Efficient on-Device Federated Learning

On the Opportunities and Risks of Foundation Models

A Field Guide to Federated Optimization

Does Knowledge Distillation Really Work?

FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout

HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients

FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning

Siloed Federated Learning for Multi-Centric Histopathology Datasets

Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training With Non-IID Private Data

Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge

Attack of the Tails: Yes, You Really Can Backdoor Federated Learning

Ensemble Distillation for Robust Model Fusion in Federated Learning

Advances and Open Problems in Federated Learning

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Federated Learning with Personalization Layers

SCAFFOLD: Stochastic Controlled Averaging for Federated Learning

Learning Data Augmentation Strategies for Object Detection

Federated Optimization in Heterogeneous Networks

Expanding the Reach of Federated Learning by Reducing Client Resource Requirements

Ray: A Distributed Framework for Emerging AI Applications

Attention is All you Need

Practical Secure Aggregation for Federated Learning on User-Held Data

Communication-Efficient Learning of Deep Networks from Decentralized Data

Deep Residual Learning for Image Recognition

Distilling the Knowledge in a Neural Network

Introduction to probability models

Percentage distribution of household income in the u.s. in 2020

Dropout: a simple way to prevent neural networks from overfitting

Learning Multiple Layers of Features from Tiny Images

The Double Dixie Cup Problem

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names