Deep Learning with Differential Privacy (2016-07-01T00:00:00.000000Z)

TL;DR

This work develops new algorithmic techniques for learning and a refined analysis of privacy costs within the framework of differential privacy, and demonstrates that deep neural networks can be trained with non-convex objectives, under a modest privacy budget, and at a manageable cost in software complexity, training efficiency, and model quality.

Abstract

Machine learning techniques based on neural networks are achieving remarkable results in a wide variety of domains. Often, the training of models requires large, representative datasets, which may be crowdsourced and contain sensitive information. The models should not expose private information in these datasets. Addressing this goal, we develop new algorithmic techniques for learning and a refined analysis of privacy costs within the framework of differential privacy. Our implementation and experiments demonstrate that we can train deep neural networks with non-convex objectives, under a modest privacy budget, and at a manageable cost in software complexity, training efficiency, and model quality.

Authors

H. B. McMahan

5 papers

Martín Abadi

3 papers

Andy Chu

1 papers

TL;DR

Abstract

Authors

References63 items

Rényi Differential Privacy

Differentially Private Stochastic Gradient Descent for in-RDBMS Analytics

Privacy Odometers and Filters: Pay-as-you-Go Composition

Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds

TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

Concentrated Differential Privacy

Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity

Large Scale Kernel Learning using Block Coordinate Descent

Differential Privacy Preservation for Deep Auto-Encoders: an Application of Human Behavior Prediction

Mastering the game of Go with deep neural networks and tree search

Algorithmic stability for adaptive data analysis

Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures

Efficient Per-Example Gradient Computations

Privacy-preserving deep learning

Privacy Preserving Data Mining

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Grammar as a Foreign Language

Move Evaluation in Go Using Deep Convolutional Neural Networks

Fractional Max-Pooling

Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Bounds

GloVe: Global Vectors for Word Representation

Going deeper with convolutions

The Algorithmic Foundations of Differential Privacy

Analyze gauss: optimal bounds for privacy-preserving principal component analysis

Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Bounds

Introductory Lectures on Convex Optimization - A Basic Course

Accelerating Stochastic Gradient Descent using Predictive Variance Reduction

Stochastic gradient descent with differentially private updates

The Composition Theorem for Differential Privacy

Efficient Estimation of Word Representations in Vector Space

ImageNet classification with deep convolutional neural networks

On Random Weights and Unsupervised Feature Learning

Scaling up biologically-inspired computer vision: A case study in unconstrained face recognition on facebook

Beyond simple features: A large-scale feature search approach to unconstrained face recognition

Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

A firm foundation for private data analysis

Boosting and Differential Privacy

Bounds on the sample complexity for private learning and private data release

Differentially Private Empirical Risk Minimization

What is the best multi-stage architecture for object recognition?

Differentially private recommender systems: building privacy into the net

Privacy integrated queries: an extensible platform for privacy-preserving data analysis

Differential privacy and robust statistics

Differentially private combinatorial optimization

The cost of privacy: destruction of data-mining utility in anonymized data publishing

What Can We Learn Privately?

Our Data, Ourselves: Privacy Via Distributed Noise Generation

Calibrating Noise to Sensitivity in Private Data Analysis

On k-Anonymity and the Curse of Dimensionality

On Private Scalar Product Computation for Privacy-Preserving Data Mining

k-Anonymity: A Model for Protecting Privacy

Lua—An Extensible Extension Language

Learning representations by back-propagating errors

Deep learning with differential privacy. CoRR, abs/1607

Private Convex Optimization for Empirical Risk Minimization with Applications to High-dimensional Regression.

Torch7: A Matlab-like Environment for Machine Learning

J. Machine Learning Research

Learning Multiple Layers of Features from Tiny Images

Introductory Lectures on Convex Optimization

Gradient-based learning applied to document recognition

The training parameters, especially the batch size and the noise scale σ

The accuracy is fairly stable over the network size. When we can only run smaller number of epochs, it is more beneﬁcial to use a larger network

TensorFlow convolutional neural networks tutorial

Field of Study

Journal Information

Name

Venue Information

Name

Type

URL

Alternate Names