Federated Multi-Task Learning under a Mixture of Distributions

Published in

Neural Information Processing Systems(2021)

External Links:

Generate Graph

TL;DR

This work proposes to study federated MTL under the flexible assumption that each local data distribution is a mixture of unknown underlying distributions, which encompasses most of the existing personalized FL approaches and leads to federated EM-like algorithms for both client-server and fully decentralized settings.

Abstract

The increasing size of data generated by smartphones and IoT devices motivated the development of Federated Learning (FL), a framework for on-device collaborative training of machine learning models. First efforts in FL focused on learning a single global model with good average performance across clients, but the global model may be arbitrarily bad for a given client, due to the inherent heterogeneity of local data distributions. Federated multi-task learning (MTL) approaches can learn personalized models by formulating an opportune penalized optimization problem. The penalization term can capture complex relations among personalized models, but eschews clear statistical assumptions about local data distributions. In this work, we propose to study federated MTL under the flexible assumption that each local data distribution is a mixture of unknown underlying distributions. This assumption encompasses most of the existing personalized FL approaches and leads to federated EM-like algorithms for both client-server and fully decentralized settings. Moreover, it provides a principled way to serve personalized models to clients not seen at training time. The algorithms' convergence is analyzed through a novel federated surrogate optimization framework, which can be of general interest. Experimental results on FL benchmarks show that our approach provides models with higher accuracy and fairness than state-of-the-art methods.

Authors

A. Bellet

3 papers

Othmane Marfoq

1 papers

G. Neglia

1 papers

References75 items

Federated Expectation Maximization with heterogeneity mitigation and variance reduction

Personalized Federated Learning using Hypernetworks

FedBN: Federated Learning on Non-IID Features via Local Batch Normalization

Personalized Federated Learning with First Order Model Optimization

Privacy Amplification by Decentralization

Federated Multi-Task Learning under a Mixture of Distributions

Published in

Neural Information Processing Systems(2021)

External Links:

Generate Graph

TL;DR

Abstract

Authors

A. Bellet

3 papers

Othmane Marfoq

1 papers

G. Neglia

1 papers

References75 items

Federated Expectation Maximization with heterogeneity mitigation and variance reduction

Personalized Federated Learning using Hypernetworks

FedBN: Federated Learning on Non-IID Features via Local Batch Normalization

Personalized Federated Learning with First Order Model Optimization

Privacy Amplification by Decentralization

Laetitia Kameni

1 papers

Richard Vidal

1 papers

Ditto: Fair and Robust Federated Learning Through Personalization

Throughput-Optimal Topology Design for Cross-Silo Federated Learning

Lower Bounds and Optimal Algorithms for Personalized Federated Learning

Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization

Personalized Cross-Silo Federated Learning on Non-IID Data

Federated Learning with Compression: Unified Analysis and Sharp Guarantees

An Efficient Framework for Clustered Federated Learning

Personalized Federated Learning with Moreau Envelopes

Adaptive Personalized Federated Learning

A Unified Theory of Decentralized SGD with Changing Topology and Local Updates

Adaptive Federated Optimization

Decentralized gradient methods: does topology matter?

Three Approaches for Personalization with Applications to Federated Learning

Personalized Federated Learning: A Meta-Learning Approach

Federated Learning with Matched Averaging

Federated Learning of a Mixture of Global and Local Models

Advances and Open Problems in Federated Learning

PyTorch: An Imperative Style, High-Performance Deep Learning Library

SCAFFOLD: Stochastic Controlled Averaging for Federated Learning

Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints

Weighted Emprirical Risk Minimization: Transfer Learning based on Importance Sampling

Improving Federated Learning Personalization via Model Agnostic Meta Learning

An Investigation Into On-device Personalization of End-to-end Automatic Speech Recognition Models

Federated Learning: Challenges, Methods, and Future Directions

Variational Federated Multi-Task Learning

Adaptive Gradient-Based Meta-Learning Methods

When can unlabeled data improve the learning rate?

The Role of Network Topology for Distributed Machine Learning

Agnostic Federated Learning

Fully Decentralized Joint Learning of Personalized Models and Collaboration Graphs

Federated Optimization in Heterogeneous Networks

On the Convergence of Federated Optimization in Heterogeneous Networks

LEAF: A Benchmark for Federated Settings

Local SGD Converges Fast and Communicates Little

D2: Decentralized Training over Decentralized Data

MobileNetV2: Inverted Residuals and Linear Bottlenecks

Asynchronous Decentralized Parallel Stochastic Gradient Descent

Network Topology and Communication-Computation Tradeoffs in Decentralized Optimization