Robust Aggregation for Federated Learning (2019-12-31T00:00:00.000000Z)

TL;DR

The experiments show that RFA is competitive with the classical aggregation when the level of corruption is low, while demonstrating greater robustness under high corruption, and establishes the convergence of the robust federated learning algorithm for the stochastic learning of additive models with least squares.

Abstract

We present a novel approach to federated learning that endows its aggregation process with greater robustness to potential poisoning of local data or model parameters of participating devices. The proposed approach, Robust Federated Aggregation (RFA), relies on the aggregation of updates using the geometric median, which can be computed efficiently using a Weiszfeld-type algorithm. RFA is agnostic to the level of corruption and aggregates model updates without revealing each device’s individual contribution. We establish the convergence of the robust federated learning algorithm for the stochastic learning of additive models with least squares. We also offer two variants of RFA: a faster one with one-step robust aggregation, and another one with on-device personalization. We present experimental results with additive models and deep networks for three tasks in computer vision and natural language processing. The experiments show that RFA is competitive with the classical aggregation when the level of corruption is low, while demonstrating greater robustness under high corruption.

Authors

Zaïd Harchaoui

5 papers

S. Kakade

4 papers

Krishna Pillutla

1 papers

TL;DR

Abstract

Authors

References94 items

Nonlinear Programming

Collaborative Unsupervised Visual Representation Learning from Decentralized Data

A Field Guide to Federated Optimization

Federated Learning: A signal processing perspective

A Superquantile Approach to Federated Learning with Heterogeneous Devices

The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation

Secure Single-Server Aggregation with (Poly)Logarithmic Overhead

Personalized Federated Learning with Moreau Envelopes

Federated Residual Learning

An Invitation to Statistics in Wasserstein Space

Adaptive Federated Optimization

A Collaborative Learning Framework via Federated Meta-Learning

Federated Variance-Reduced Stochastic Gradient Descent With Robustness to Byzantine Attacks

Advances and Open Problems in Federated Learning

Can You Really Backdoor Federated Learning?

SCAFFOLD: Stochastic Controlled Averaging for Federated Learning

Distributed Gradient Descent Algorithm Robust to an Arbitrary Number of Byzantine Attackers

Federated Learning: Challenges, Methods, and Future Directions

Hypothesis Testing Interpretations and Renyi Differential Privacy

Federated Learning-Based Computation Offloading Optimization in Edge Computing-Supported Internet of Things

A Pragmatic Introduction to Secure Multi-Party Computation

Patient Clustering Improves Efficiency of Federated Machine Learning to predict mortality and hospital stay time using distributed Electronic Medical Records

Towards Federated Learning at Scale: System Design

Agnostic Federated Learning

Federated Collaborative Filtering for Privacy-Preserving Personalized Recommendation System

Robust Distributed Diffusion Recursive Least Squares Algorithms With Side Information for Adaptive Networks

Resilient Distributed Parameter Estimation With Heterogeneous Data

Federated Optimization in Heterogeneous Networks

On the Convergence of Federated Optimization in Heterogeneous Networks

APPLIED FEDERATED LEARNING: IMPROVING GOOGLE KEYBOARD QUERY SUGGESTIONS

LEAF: A Benchmark for Federated Settings

High-Dimensional Robust Mean Estimation in Nearly-Linear Time

Robust Statistics

RSA: Byzantine-Robust Stochastic Aggregation Methods for Distributed Learning from Heterogeneous Datasets

COLA: Decentralized Linear Learning

DRACO: Byzantine-resilient Distributed Training via Redundant Gradients

Byzantine Stochastic Gradient Descent

Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates

Improved asynchronous parallel optimization analysis for stochastic incremental methods

Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent

Robust machine learning by median-of-means: Theory and practice

Practical Secure Aggregation for Privacy-Preserving Machine Learning

A Formal Foundation for Secure Remote Execution of Enclaves

A Markov Chain Theory Approach to Characterizing the Minimax Optimality of Stochastic Gradient Descent (for Least Squares)

Robust Distributed Estimation by Networked Agents

Federated Multi-Task Learning

Distributed Statistical Machine Learning in Adversarial Settings

EMNIST: an extension of MNIST to handwritten letters

Regularization, sparse recovery, and median-of-means tournaments

Understanding deep learning requires rethinking generalization

CoCoA: A General Framework for Communication-Efficient Distributed Optimization

Federated Learning: Strategies for Improving Communication Efficiency

Parallelizing Stochastic Gradient Descent for Least Squares Regression: Mini-batching, Averaging, and Model Misspecification

Risk minimization by median-of-means tournaments

Geometric median in nearly linear time

Robust Estimators in High Dimensions without the Computational Intractability

Communication-Efficient Learning of Deep Networks from Decentralized Data

Distributed optimization with arbitrary local solvers

Robust covariance and scatter matrix estimation under Huber’s contamination model

On the Convergence of Alternating Minimization for Convex Programming with Applications to Iteratively Reweighted Least Squares and Decomposition Schemes

GloVe: Global Vectors for Word Representation

Adaptation, Learning, and Optimization over Networks

Incremental Majorization-Minimization Optimization with Application to Large-Scale Machine Learning

Geometric median and robust estimation in Banach spaces

Loss Minimization and Parameter Estimation with Heavy Tails

Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)

Smoothing and First Order Methods: A Unified Framework

Computing arbitrary functions of encrypted data

Distributed Subgradient Methods for Multi-Agent Optimization

Robust Statistics: Theory and Methods

Calibrating Noise to Sensitivity in Private Data Analysis

A modified Weiszfeld algorithm for the Fermat-Weber location problem

Long Short-Term Memory

Another proof of the defect relation for moving targets

The Byzantine Generals Problem

Local convergence in Fermat's problem