Secure Distributed Training at Scale (2021-06-21T00:00:00.000000Z)

TL;DR

This work proposes a novel protocol for secure (Byzantine-tolerant) decentralized training that emphasizes communication efficiency and rigorously analyzed this protocol provides theoretical bounds for its resistance against Byzantine and Sybil attacks and shows that it has a marginal communication overhead.

Abstract

Some of the hardest problems in deep learning can be solved with the combined effort of many independent parties, as is the case for volunteer computing and federated learning. These setups rely on high numbers of peers to provide computational resources or train on decentralized datasets. Unfortunately, participants in such systems are not always reliable. Any single participant can jeopardize the entire training run by sending incorrect updates, whether deliberately or by mistake. Training in presence of such peers requires specialized distributed training algorithms with Byzantine tolerance. These algorithms often sacrifice efficiency by introducing redundant communication or passing all updates through a trusted server. As a result, it can be infeasible to apply such algorithms to large-scale distributed deep learning, where models can have billions of parameters. In this work, we propose a novel protocol for secure (Byzantine-tolerant) decentralized training that emphasizes communication efficiency. We rigorously analyze this protocol: in particular, we provide theoretical bounds for its resistance against Byzantine and Sybil attacks and show that it has a marginal communication overhead. To demonstrate its practical effectiveness, we conduct large-scale experiments on image classification and language modeling in presence of Byzantine attackers.

Authors

Max Ryabinin

4 papers

Michael Diskin

2 papers

Eduard A. Gorbunov

1 papers

TL;DR

Abstract

Authors

References115 items

Byzantine Fault-Tolerance in Decentralized Optimization under 2f-Redundancy

Distributed Deep Learning Using Volunteer Computing-Like Paradigm

MARINA: Faster Non-Convex Distributed Learning with Compression

Byzantine Fault-Tolerance in Peer-to-Peer Distributed Gradient-Descent

Byzantine-Resilient Non-Convex Stochastic Gradient Descent

Learning from History for Byzantine Robust Optimization

Privacy and Robustness in Federated Learning: Attacks and Defenses

Towards Building a Robust and Fair Federated Learning System

Local SGD: Unified Theory and New Efficient Methods

Why are Adaptive Methods Good for Attention Models?

Linearly Converging Error Compensated SGD

Dynamic Federated Learning Model for Identifying Adversarial Clients

Data Poisoning Attacks Against Federated Learning Systems

GossipSub: Attack-Resilient Message Propagation in the Filecoin and ETH2.0 Networks

ByGARS: Byzantine SGD with Arbitrary Number of Attackers.

Byzantine-Robust Learning on Heterogeneous Datasets via Resampling

Language Models are Few-Shot Learners

Byzantine-Robust Decentralized Stochastic Optimization

On Distributed Stochastic Gradient Descent for Nonconvex Functions in the Presence of Byzantines

A Unified Theory of Decentralized SGD with Changing Topology and Local Updates

Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts

Robust Aggregation for Federated Learning

Federated Variance-Reduced Stochastic Gradient Descent With Robustness to Byzantine Attacks

Big Transfer (BiT): General Visual Representation Learning

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Efficient Multi-Party Private Set Intersection Against Malicious Adversaries

PipeDream: generalized pipeline parallelism for DNN training

ZeRO: Memory Optimization Towards Training A Trillion Parameter Models

SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

BRIDGE: Byzantine-Resilient Decentralized Gradient Descent

BRIDGE: Byzantine-resilient Decentralized Gradient Descent

DETOX: A Redundancy-based Framework for Faster and More Robust Gradient Aggregation

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Decentralized Deep Learning with Arbitrary Communication Compression

Genuinely distributed Byzantine machine learning

AGGREGATHOR: Byzantine Machine Learning via Robust Gradient Aggregation

Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

Patient Clustering Improves Efficiency of Federated Machine Learning to predict mortality and hospital stay time using distributed Electronic Medical Records

Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulation

A Little Is Enough: Circumventing Defenses For Distributed Learning

Towards Federated Learning at Scale: System Design

Distributed learning with compressed gradient differences

On the Ineffectiveness of Variance Reduced Optimization for Deep Learning

GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash

RSA: Byzantine-Robust Stochastic Aggregation Methods for Distributed Learning from Heterogeneous Datasets

Federated Learning for Mobile Keyboard Prediction

Stochastic Gradient Push for Distributed Deep Learning

Protection

Communication complexity of byzantine agreement, revisited

A hybrid GPU cluster and volunteer computing platform for scalable deep learning

DRACO: Byzantine-resilient Distributed Training via Redundant Gradients

Byzantine Stochastic Gradient Descent

Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates

The Hidden Vulnerability of Distributed Learning in Byzantium

Horovod: fast and easy distributed deep learning in TensorFlow

Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent

An Efficient Task-based All-Reduce for Machine Learning Applications

ByRDiE: Byzantine-Resilient Distributed Coordinate Descent for Decentralized Learning

Revisiting Unreasonable Effectiveness of Data in Deep Learning Era

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

QSGD: Communication-Optimal Stochastic Gradient Descent, with Applications to Training Neural Networks

Pointer Sentinel Mixture Models

SGDR: Stochastic Gradient Descent with Warm Restarts

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Robust Distributed Consensus Using Total Variation

Communication-Efficient Learning of Deep Networks from Decentralized Data

Observation of Gravitational Waves from a Binary Black Hole Merger

Deep Residual Learning for Image Recognition

Scaling Distributed Machine Learning with the Parameter Server

1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs

SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives

Introductory Lectures on Convex Optimization - A Basic Course