DeltaGrad: Rapid retraining of machine learning models (2020-06-26T00:00:00.000000Z)

TL;DR

This work proposes the DeltaGrad algorithm for rapid retraining machine learning models based on information cached during the training phase, and shows that it compares favorably to the state of the art.

Abstract

Machine learning models are not static and may need to be retrained on slightly changed datasets, for instance, with the addition or deletion of a set of data points. This has many applications, including privacy, robustness, bias reduction, and uncertainty quantifcation. However, it is expensive to retrain models from scratch. To address this problem, we propose the DeltaGrad algorithm for rapid retraining machine learning models based on information cached during the training phase. We provide both theoretical and empirical support for the effectiveness of DeltaGrad, and show that it compares favorably to the state of the art.

Authors

Yinjun Wu

1 papers

Edgar Dobriban

1 papers

S. Davidson

1 papers

TL;DR

Abstract

Authors

References71 items

Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations

PrIU: A Provenance-Based Approach for Incrementally Updating Regression Models

Machine Unlearning

Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks

Certified Data Removal from Machine Learning Models

Making AI Forget You: Data Deletion in Machine Learning

A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descent

Data Shapley: Equitable Valuation of Data for Machine Learning

SGD: General Analysis and Improved Rates

Numerical Optimization

A modern maximum-likelihood theory for high-dimensional logistic regression

Deletion-Robust Submodular Maximization: Data Summarization with "the Right to be Forgotten"

Certified Defenses for Data Poisoning Attacks

PALM: Machine Learning Explanations For Iterative Debugging

Understanding Black-box Predictions via Influence Functions

A Roadmap for a Rigorous Science of Interpretability

Towards A Rigorous Science of Interpretable Machine Learning

Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition

Optimization Methods for Large-Scale Machine Learning

Deep Residual Learning for Image Recognition

The Expected Norm of a Sum of Independent Random Matrices: An Elementary Approach

Towards Making Systems Forget with Machine Unlearning

Iterative Solution of Nonlinear Equations in Several Variables

Global convergence of online limited memory BFGS

The Algorithmic Foundations of Differential Privacy

Robust linear regression: A review and comparison

Searching for exotic particles in high-energy physics with deep learning

Cross-conformal predictors

Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning

User-Friendly Tail Bounds for Sums of Random Matrices

Convex optimization

Concentration of the adjacency matrix and of the Laplacian in random graphs with independent edges

Privacy-preserving logistic regression

The Tradeoffs of Large Scale Learning

A tutorial on conformal prediction

RCV1: A New Benchmark Collection for Text Categorization Research

Solving large scale linear prediction problems using stochastic gradient descent algorithms

Incremental learning with support vector machines

Subsampling

Detection of Influential Observation in Linear Regression

Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables

On-line learning and stochastic approximations

Lazy Learning Meets the Recursive Least Squares Algorithm

Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization

A Limited Memory Algorithm for Bound Constrained Optimization

Representations of quasi-Newton matrices and their use in limited memory methods

Estimation of convergence rate for robust identification algorithms

Convergence of quasi-Newton matrices generated by the symmetric rank one update

Testing a class of methods for solving minimization problems with simple bounds on the variables

Matrix analysis

Updating Quasi-Newton Matrices With Limited Storage

A Theory of Adaptive Pattern Classifiers

NOTES ON BIAS IN ESTIMATION

A Stochastic Approximation Method

“Amnesia” – Towards Machine Learning Models That Can Forget User Data Very Fast

European Union, C. o. Council regulation (eu) no 2016/679

CLASSICAL RESULTS FOR RANDOM VARIABLES To analyze DeltaGrad with SGD, Bernsteins inequality (Oliveira

Large-Scale Machine Learning with Stochastic Gradient Descent

Neuro-Dynamic Programming

Learning Multiple Layers of Features from Tiny Images

Stochastic Learning

GradientBased Learning Applied to Document Recognition

Evaluating derivatives - principles and techniques of algorithmic differentiation, Second Edition

Incremental and Decremental Support Vector Machine Learning

On-Line Learning in Neural Networks

Gradient-based learning applied to document recognition

Online Learning and Stochastic Approximations

The solution of nonlinear finite element equations

Stochastic Approximation

INTERNATIONAL JOURNAL FOR NUMERICAL METHODS IN ENGINEERING

DeltaGrad

Field of Study

Venue Information

Name

Type

URL