End-to-End Evaluation of Federated Learning and Split Learning for Internet of Things (2020-03-30T00:00:00.000000Z)

TL;DR

This work is the first attempt to provide empirical comparisons of FL and SplitNN in real-world IoT settings in terms of learning performance and device implementation overhead and demonstrates that neither FL or SplitNN can be applied to a heavy model, e.g., with several million parameters, on resource-constrained IoT devices because its training cost would be too expensive for such devices.

Abstract

Federated learning (FL) and split neural networks (SplitNN) are state-of-art distributed machine learning techniques to enable machine learning without directly accessing raw data on clients or end devices. In theory, such distributed machine learning techniques have great potential in distributed applications, in which data are typically generated and collected at the client-side while the collected data should be processed by the application deployed at the server-side. However, there is still a significant gap in evaluating the performance of those techniques concerning their practicality in the Internet of Things (IoT)-enabled distributed systems constituted by resource-constrained devices. This work is the first attempt to provide empirical comparisons of FL and SplitNN in real-world IoT settings in terms of learning performance and device implementation overhead. We consider a variety of datasets, different model architectures, multiple clients, and various performance metrics. For the learning performance (i.e., model accuracy and convergence time), we empirically evaluate both FL and SplitNN under different types of data distributions such as imbalanced and non-independent and identically distributed (non-IID) data. We show that the learning performance of SplitNN is better than FL under an imbalanced data distribution but worse than FL under an extreme non-IID data distribution. For implementation overhead, we mount both FL and SplitNN on Raspberry Pi devices and comprehensively evaluate their overhead, including training time, communication overhead, power consumption, and memory usage. Our key observations are that under the IoT scenario where the communication traffic is the primary concern, FL appears to perform better over SplitNN because FL has a significantly lower communication overhead compared with SplitNN. However, our experimental results also demonstrate that neither FL or SplitNN can be applied to a heavy model, e.g., with several million parameters, on resource-constrained IoT devices because its training cost would be too expensive for such devices. Source code is released and available: https://github.com/Minki-Kim95/Federated-Learning-and-Split-Learning-with-raspberry-pi.

References38 items

SplitFed: When Federated Learning Meets Split Learning

Federated Continual Learning with Adaptive Parameter Communication

AdderNet: Do We Really Need Multiplications in Deep Learning?

Detailed comparison of communication efficiency of split learning and federated learning

Federated Learning: Challenges, Methods, and Future Directions

Deep Learning With Edge Computing: A Review

On the Convergence of FedAvg on Non-IID Data

Patient Clustering Improves Efficiency of Federated Machine Learning to predict mortality and hospital stay time using distributed Electronic Medical Records

The Seven Sins of Personal-Data Processing Systems under GDPR

Towards Federated Learning at Scale: System Design

No Peek: A Survey of private distributed deep learning

Distributed learning of deep neural network over multiple agents

Federated Learning with Non-IID Data

DÏoT: A Federated Self-learning Anomaly Detection System for IoT

A Survey on Resource Management in IoT Operating Systems

Training wide residual networks for deployment using a single bit for each weight

Classification of ECG signals based on 1D convolution neural network

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Adversary Resistant Deep Neural Networks with an Application to Malware Detection

Deep learning approach for Network Intrusion Detection in Software Defined Networking

XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks

Real-Time Patient-Specific ECG Classification by 1-D Convolutional Neural Networks

Communication-Efficient Learning of Deep Networks from Decentralized Data

Distilling the Knowledge in a Neural Network

A survey of Algorithms and Analysis for Adaptive Online Learning

Neural network ensembles: evaluation of aggregation algorithms

The impact of the MIT-BIH Arrhythmia Database

Speech Commands

Deep Learning

FL applies to sequential models

Teach, learn, and make with raspberry pi

InvenSense Announces World's Smallest, Lowest Power I2S Microphone for IoT and Wearable Markets

Make with Raspberry Pi -Raspberry Pi

SplitNN always converges much faster than FL once it learns

c) Power Consumption: We use a plug-in powermeter, as shown in Fig. 8, to measure the power consumption. With this powermeter

Major breach found in biometrics system used by banks, UK police and defence firms

Amazon alexa user receives 1,700 audio recordings of a stranger through huuman error

evaluating FL and SplitNN across ﬁve clients with different model architectures