Personalized Federated Learning with Moreau Envelopes (2020-06-01T00:00:00.000000Z)

TL;DR

This work proposes an algorithm for personalized FL (pFedMe) using Moreau envelopes as clients' regularized loss functions, which help decouple personalized model optimization from the global model learning in a bi-level problem stylized for personalizedFL.

Abstract

Federated learning (FL) is a decentralized and privacy-preserving machine learning technique in which a group of clients collaborate with a server to learn a global model without sharing clients' data. One challenge associated with FL is statistical diversity among clients, which restricts the global model from delivering good performance on each client's task. To address this, we propose an algorithm for personalized FL (pFedMe) using Moreau envelopes as clients' regularized loss functions, which help decouple personalized model optimization from the global model learning in a bi-level problem stylized for personalized FL. Theoretically, we show that pFedMe's convergence rate is state-of-the-art: achieving quadratic speedup for strongly convex and sublinear speedup of order 2/3 for smooth nonconvex objectives. Experimentally, we verify that pFedMe excels at empirical performance compared with the vanilla FedAvg and Per-FedAvg, a meta-learning based personalized FL algorithm.

Authors

Canh T. Dinh

1 papers

N. H. Tran

1 papers

Tuan Dung Nguyen

2 papers

TL;DR

Abstract

Authors

References57 items

Preserving Data Privacy via Federated Learning: Challenges and Solutions

Adaptive Personalized Federated Learning

Adaptive Federated Optimization

Three Approaches for Personalization with Applications to Federated Learning

Personalized Federated Learning: A Meta-Learning Approach

Federated Learning of a Mixture of Global and Local Models

Robust Aggregation for Federated Learning

Advances and Open Problems in Federated Learning

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Federated Learning with Personalization Layers

Hyper-Sphere Quantization: Communication-Efficient SGD for Federated Learning

On the Convergence of Local Descent Methods in Federated Learning

Federated Evaluation of On-device Personalization

Communication-Efficient Local Decentralized SGD Methods

SCAFFOLD: Stochastic Controlled Averaging for Federated Learning

FedMD: Heterogenous Federated Learning via Model Distillation

FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization

Improving Federated Learning Personalization via Model Agnostic Meta Learning

Tighter Theory for Local SGD on Identical and Heterogeneous Data

First Analysis of Local GD on Heterogeneous Data

On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithms

Federated Learning: Challenges, Methods, and Future Directions

Unified Optimal Analysis of the (Stochastic) Gradient Method

On the Convergence of FedAvg on Non-IID Data

Adaptive Gradient-Based Meta-Learning Methods

On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization

One-Shot Federated Learning

Federated Heavy Hitters Discovery with Differential Privacy

Agnostic Federated Learning

Federated Optimization in Heterogeneous Networks

Lectures on Convex Optimization

New Convergence Aspects of Stochastic Gradient Algorithms

Federated Learning for Mobile Keyboard Prediction

Don't Use Large Mini-Batches, Use Local SGD

Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms

A Tight Convergence Analysis for Stochastic Gradient Descent with Delayed Updates

Federated Learning with Non-IID Data

cpSGD: Communication-efficient and differentially-private distributed SGD

Local SGD Converges Fast and Communicates Little

On First-Order Meta-Learning Algorithms

Reptile: a Scalable Metalearning Algorithm

Federated Meta-Learning for Recommendation

Federated Meta-Learning with Fast Convergence and Efficient Communication

Catalyst Acceleration for First-order Convex Optimization: from Theory to Practice

Learning Differentially Private Recurrent Language Models

Federated Multi-Task Learning

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Decentralized Collaborative Learning of Personalized Models over Networks

Communication-Efficient Learning of Deep Networks from Decentralized Data

Strongly Convex Functions, Moreau Envelopes, and the Generic Nature of Convex Functions with Strong Minimizers

Convex Optimization: Algorithms and Complexity

Privacy Aware Learning

Practical Aspects of the Moreau-Yosida Regularization: Theoretical Preliminaries

A regularization interpretation of the proximal point method for weakly convex functions

Efficient Meta Learning via Minibatch Proximal Update

Gradient-based learning applied to document recognition

Propriétés des applications “prox

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names