Sisyphus: A Cautionary Tale of Using Low-Degree Polynomial Activations in Privacy-Preserving Deep Learning (2021-07-26T00:00:00.000000Z)

TL;DR

This work analyzes the challenges of substituting ReLUs with polynomials, starting with simple drop-and-replace solutions to novel, more involved replace- and-retrain strategies, and finds all evaluated solutions suffer from the escaping activation problem.

Abstract

Privacy concerns in client-server machine learning have given rise to private inference (PI), where neural inference occurs directly on encrypted inputs. PI protects clients' personal data and the server's intellectual property. A common practice in PI is to use garbled circuits to compute nonlinear functions privately, namely ReLUs. However, garbled circuits suffer from high storage, bandwidth, and latency costs. To mitigate these issues, PI-friendly polynomial activation functions have been employed to replace ReLU. In this work, we ask: Is it feasible to substitute all ReLUs with low-degree polynomial activation functions for building deep, privacy-friendly neural networks? We explore this question by analyzing the challenges of substituting ReLUs with polynomials, starting with simple drop-and-replace solutions to novel, more involved replace-and-retrain strategies. We examine the limitations of each method and provide commentary on the use of polynomial activation functions for PI. We find all evaluated solutions suffer from the escaping activation problem: forward activation values inevitably begin to expand at an exponential rate away from stable regions of the polynomials, which leads to exploding values (NaNs) or poor approximations.

Authors

Karthik Garimella

1 papers

Nandan Kumar Jha

1 papers

Brandon Reagen

1 papers

TL;DR

Abstract

Authors

References55 items

Precise Approximation of Convolutional Neural Networks for Homomorphically Encrypted Data

DeepReDuce: ReLU Reduction for Fast Private Inference

Delphi: A Cryptographic Inference System for Neural Networks

CrypTFlow2: Practical 2-Party Secure Inference

CryptoNAS: Private Inference on a ReLU Budget

BLAZE: Blazing Fast Privacy-Preserving Machine Learning

Dynamic ReLU

Trident: Efficient 4PC Framework for Privacy Preserving Machine Learning

PyTorch: An Imperative Style, High-Performance Deep Learning Library

ASTRA: High Throughput 3PC over Rings with Application to Secure Prediction

Mish: A Self Regularized Non-Monotonic Neural Activation Function

Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks

EzPC: Programmable and Efficient Secure Two-Party Computation for Machine Learning

Efficient Evaluation of Activation Functions over Encrypted Data

XONN: XNOR-based Oblivious Deep Neural Network Inference

Towards Federated Learning at Scale: System Design

Faster CryptoNets: Leveraging Sparsity for Real-World Encrypted Inference

The AlexNet Moment for Homomorphic Encryption: HCNN, the First Homomorphic CNN on Encrypted Data with GPUs

ABY3: A Mixed Protocol Framework for Machine Learning

GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration

TAPAS: Tricks to Accelerate (encrypted) Prediction As a Service

Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware

Chiron: Privacy-preserving Machine Learning as a Service

Gazelle: A Low Latency Framework for Secure Neural Network Inference

Chameleon: A Hybrid Secure Computation Framework for Machine Learning Applications

CryptoDL: Deep Neural Networks over Encrypted Data

Oblivious Neural Network Predictions via MiniONN Transformations

Self-Normalizing Neural Networks

DeepSecure: Scalable Provably-Secure Deep Learning

SecureML: A System for Scalable Privacy-Preserving Machine Learning

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Federated Learning: Strategies for Improving Communication Efficiency

Deep Learning with Differential Privacy

Gaussian Error Linear Units (GELUs)

CryptoNets: Applying Neural Networks to Encrypted Data with High Throughput and Accuracy

Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units

Deep Residual Learning for Image Recognition

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

Empirical Evaluation of Rectified Activations in Convolutional Network

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Very Deep Convolutional Networks for Large-Scale Image Recognition

ImageNet classification with deep convolutional neural networks

Practical Bayesian Optimization of Machine Learning Algorithms

Differential Privacy

How to share a secret

Appendix to: BOTORCH: A Framework for Efficient Monte-Carlo Bayesian Optimization

Searching for Activation Functions

Privacy-Preserving Classification on Deep Neural Network

Intel SGX Explained

Tiny ImageNet Classiﬁcation with Convolutional Neural Networks

Fully Homomorphic Encryption

Rectifier Nonlinearities Improve Neural Network Acoustic Models

MNIST handwritten digit database

Cifar-10 (canadian institute for advanced research)

A fully homomorphic encryption scheme

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names