Federated Multi-Task Learning under a Mixture of Distributions (2021-08-23T00:00:00.000000Z)

TL;DR

This work proposes to study federated MTL under the flexible assumption that each local data distribution is a mixture of unknown underlying distributions, which encompasses most of the existing personalized FL approaches and leads to federated EM-like algorithms for both client-server and fully decentralized settings.

Abstract

The increasing size of data generated by smartphones and IoT devices motivated the development of Federated Learning (FL), a framework for on-device collaborative training of machine learning models. First efforts in FL focused on learning a single global model with good average performance across clients, but the global model may be arbitrarily bad for a given client, due to the inherent heterogeneity of local data distributions. Federated multi-task learning (MTL) approaches can learn personalized models by formulating an opportune penalized optimization problem. The penalization term can capture complex relations among personalized models, but eschews clear statistical assumptions about local data distributions. In this work, we propose to study federated MTL under the flexible assumption that each local data distribution is a mixture of unknown underlying distributions. This assumption encompasses most of the existing personalized FL approaches and leads to federated EM-like algorithms for both client-server and fully decentralized settings. Moreover, it provides a principled way to serve personalized models to clients not seen at training time. The algorithms' convergence is analyzed through a novel federated surrogate optimization framework, which can be of general interest. Experimental results on FL benchmarks show that our approach provides models with higher accuracy and fairness than state-of-the-art methods.

Authors

A. Bellet

3 papers

Othmane Marfoq

1 papers

G. Neglia

1 papers

TL;DR

Abstract

Authors

References75 items

Federated Expectation Maximization with heterogeneity mitigation and variance reduction

Personalized Federated Learning using Hypernetworks

FedBN: Federated Learning on Non-IID Features via Local Batch Normalization

Personalized Federated Learning with First Order Model Optimization

Privacy Amplification by Decentralization

Ditto: Fair and Robust Federated Learning Through Personalization

Throughput-Optimal Topology Design for Cross-Silo Federated Learning

Lower Bounds and Optimal Algorithms for Personalized Federated Learning

Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization

Personalized Cross-Silo Federated Learning on Non-IID Data

Federated Learning with Compression: Unified Analysis and Sharp Guarantees

An Efficient Framework for Clustered Federated Learning

Personalized Federated Learning with Moreau Envelopes

Adaptive Personalized Federated Learning

A Unified Theory of Decentralized SGD with Changing Topology and Local Updates

Graphical Models

Adaptive Federated Optimization

Decentralized gradient methods: does topology matter?

Three Approaches for Personalization with Applications to Federated Learning

Personalized Federated Learning: A Meta-Learning Approach

Federated Learning with Matched Averaging

Federated Learning of a Mixture of Global and Local Models

Advances and Open Problems in Federated Learning

PyTorch: An Imperative Style, High-Performance Deep Learning Library

SCAFFOLD: Stochastic Controlled Averaging for Federated Learning

Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints

Weighted Emprirical Risk Minimization: Transfer Learning based on Importance Sampling

Improving Federated Learning Personalization via Model Agnostic Meta Learning

An Investigation Into On-device Personalization of End-to-end Automatic Speech Recognition Models

Federated Learning: Challenges, Methods, and Future Directions

Variational Federated Multi-Task Learning

Adaptive Gradient-Based Meta-Learning Methods

When can unlabeled data improve the learning rate?

The Role of Network Topology for Distributed Machine Learning

Agnostic Federated Learning

Fully Decentralized Joint Learning of Personalized Models and Collaboration Graphs

Federated Optimization in Heterogeneous Networks

On the Convergence of Federated Optimization in Heterogeneous Networks

LEAF: A Benchmark for Federated Settings

Random Graphs

Local SGD Converges Fast and Communicates Little

D2: Decentralized Training over Decentralized Data

MobileNetV2: Inverted Residuals and Linear Bottlenecks

Asynchronous Decentralized Parallel Stochastic Gradient Descent

Network Topology and Communication-Computation Tradeoffs in Decentralized Optimization

Federated Multi-Task Learning

Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent

Personalized and Private Peer-to-Peer Machine Learning

EMNIST: Extending MNIST to handwritten letters

Federated Learning: Strategies for Improving Communication Efficiency

Decentralized Collaborative Learning of Personalized Models over Networks

Communication-Efficient Learning of Deep Networks from Decentralized Data

Convex Optimization: Algorithms and Complexity

Introductory Lectures on Convex Optimization - A Basic Course

Clustered Multi-Task Learning Via Alternating Structure Optimization

Learning Bounds for Importance Weighting

Torchvision the machine-vision package of torch

A Convex Formulation for Learning Task Relationships in Multi-Task Learning

Sample Selection Bias Correction Theory

Direct Importance Estimation with Model Selection and Its Application to Covariate Shift Adaptation

Pachinko allocation: DAG-structured mixture models of topic correlations

A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data

Optimization Transfer Using Surrogate Objective Functions

Long Short-Term Memory

FedU: A Unified Framework for Federated Multi-Task Learning with Laplacian Regularization

Debiasing Model Updates for Improving Personalized Federated Training

2020) for a unified theoretical analysis of decentralized SGD

Unlabeled Data Does Provably Help

Author manuscript, published in "International Conference on Machine Learning (ICML 2013) (2013)" Optimization with First-Order Surrogate Functions

Learning Multiple Layers of Features from Tiny Images

Does Unlabeled Data Provably Help? Worst-case Analysis of the Sample Complexity of Semi-Supervised Learning

Fastest Mixing Markov Chain on a Graph

R du × V → R is a partial first-order surrogate of f (u, v)

Algorithm 1 FedEM 1: Input: data S 1:T ; number of mixture distributions M; number of communication rounds K 2: for iterations k = 1

Using Jensen inequality

Field of Study