Optimizing Millions of Hyperparameters by Implicit Differentiation

Published in

International Conference on Artificial Intellig...(2019)

External Links:

Generate Graph

TL;DR

An algorithm for inexpensive gradient-based hyperparameter optimization that combines the implicit function theorem (IFT) with efficient inverse Hessian approximations is proposed and used to train modern network architectures with millions of weights and millions of hyper-parameters.

Abstract

We propose an algorithm for inexpensive gradient-based hyperparameter optimization that combines the implicit function theorem (IFT) with efficient inverse Hessian approximations. We present results about the relationship between the IFT and differentiating through optimization, motivating our algorithm. We use the proposed approach to train modern network architectures with millions of weights and millions of hyper-parameters. For example, we learn a data-augmentation network - where every weight is a hyperparameter tuned for validation performance - outputting augmented training examples. Jointly tuning weights and hyperparameters with our approach is only a few times more costly in memory and compute than standard training.

Authors

Paul Vicol

3 papers

D. Duvenaud

7 papers

Jonathan Lorraine

2 papers

References79 items

Penalty Method for Inversion-Free Deep Bilevel Optimization

On Solving Minimax Optimization Locally: A Follow-the-Ridge Approach

Generalized Inner Loop Meta-Learning

Meta-Learning with Implicit Gradients

Deep Equilibrium Models

Optimizing Millions of Hyperparameters by Implicit Differentiation

Published in

International Conference on Artificial Intellig...(2019)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Paul Vicol

3 papers

D. Duvenaud

7 papers

Jonathan Lorraine

2 papers

References79 items

Penalty Method for Inversion-Free Deep Bilevel Optimization

On Solving Minimax Optimization Locally: A Follow-the-Ridge Approach

Generalized Inner Loop Meta-Learning

Meta-Learning with Implicit Gradients

Deep Equilibrium Models

AutoAssist: A Framework to Accelerate Training of Deep Neural Networks

Unsupervised Data Augmentation

Meta-Learning With Differentiable Convex Optimization

Adversarial Learning of General Transformations for Data Augmentation

Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly

Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions

MetaReg: Towards Domain Generalization using Meta-Regularization

Stable Opponent Shaping in Differentiable Games

Truncated Back-propagation for Bilevel Optimization

DARTS: Differentiable Architecture Search

Bilevel Programming for Hyperparameter Optimization and Meta-Learning

Parallel Architecture and Hyperparameter Search via Successive Halving and Classification

AutoAugment: Learning Augmentation Policies from Data

Learning to Reweight Examples for Robust Deep Learning

Reviving and Improving Recurrent Back-Propagation

On First-Order Meta-Learning Algorithms

Stochastic Hyperparameter Optimization through Hypernetworks

The Mechanics of n-Player Differentiable Games

Understanding Short-Horizon Bias in Stochastic Meta-Optimization

ScreenerNet: Learning Self-Paced Curriculum for Deep Neural Networks

Noisy Natural Gradient as Variational Inference

Population Based Training of Neural Networks

On Optimal Generalizability in Parametric Learning

Regularization for Deep Learning: A Taxonomy

Automatic differentiation in PyTorch

Learning with Opponent-Learning Awareness

SMASH: One-Shot Model Architecture Search through HyperNetworks

Regularizing and Optimizing LSTM Language Models

The Numerics of GANs

Understanding Black-box Predictions via Influence Functions

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Forward and Reverse Gradient-Based Hyperparameter Optimization

Categorical Reparameterization with Gumbel-Softmax

The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables

Learning to learn by gradient descent by gradient descent

Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

Second-Order Stochastic Optimization for Machine Learning in Linear Time

Hyperparameter optimization with approximate gradient

DrMAD: Distilling Reverse-Mode Automatic Differentiation for Optimizing Hyperparameters of Deep Neural Networks

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

Deep Residual Learning for Image Recognition

Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters

Variational Dropout and the Local Reparameterization Trick

Bilevel Optimization with Nonsmooth Lower Level Problems

U-Net: Convolutional Networks for Biomedical Image Segmentation

Optimizing Neural Networks with Kronecker-factored Approximate Curvature

Non-stochastic Best Arm Identification and Hyperparameter Optimization

Gradient-based Hyperparameter Optimization through Reversible Learning

Adam: A Method for Stochastic Optimization

Regularization of Neural Networks using DropConnect

ImageNet classification with deep convolutional neural networks

Practical Bayesian Optimization of Machine Learning Algorithms

Generic Methods for Optimization-Based Modeling

Random Search for Hyper-Parameter Optimization

Gradient-Based Optimization of Hyperparameters

Automatic Detection and Segmentation of Evolving Processes in 3D Medical Images: Application to Multiple Sclerosis

Long Short-Term Memory

Design and regularization of neural networks: the optimal use of a validation set

Fast Exact Multiplication by the Hessian

Building a Large Annotated Corpus of English: The Penn Treebank

Hedonic housing prices and the demand for clean air

On Bayesian Methods for Seeking the Extremum

Phu Mon Htut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho, and Soumith Chintala

GENERATIVE ADVERSARIAL NETS

A Efros. Dataset distillation

Neural Networks for Machine Learning

Learning Multiple Layers of Features from Tiny Images

Gradient-based learning applied to document recognition

Reverse aumulation and imploicit functions

Evolutionary principles in self-referential learning, or on learning how to learn: The meta-meta-. hook

Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales

Field of Study

Computer ScienceMathematics

Venue Information

Name

International Conference on Artificial Intelligence and Statistics

Type

conference

URL

N/A

Alternate Names

AISTATS
Int Conf Artif Intell Stat

TL;DR

Abstract

Authors

References79 items

Penalty Method for Inversion-Free Deep Bilevel Optimization

On Solving Minimax Optimization Locally: A Follow-the-Ridge Approach

Generalized Inner Loop Meta-Learning

Meta-Learning with Implicit Gradients

Deep Equilibrium Models

TL;DR

Abstract

Authors

References79 items

Penalty Method for Inversion-Free Deep Bilevel Optimization

On Solving Minimax Optimization Locally: A Follow-the-Ridge Approach

Generalized Inner Loop Meta-Learning

Meta-Learning with Implicit Gradients

Deep Equilibrium Models

AutoAssist: A Framework to Accelerate Training of Deep Neural Networks

Unsupervised Data Augmentation

Meta-Learning With Differentiable Convex Optimization

Adversarial Learning of General Transformations for Data Augmentation

Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly

Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions

MetaReg: Towards Domain Generalization using Meta-Regularization

Stable Opponent Shaping in Differentiable Games

Truncated Back-propagation for Bilevel Optimization

DARTS: Differentiable Architecture Search

Bilevel Programming for Hyperparameter Optimization and Meta-Learning

Parallel Architecture and Hyperparameter Search via Successive Halving and Classification

AutoAugment: Learning Augmentation Policies from Data

Learning to Reweight Examples for Robust Deep Learning

Reviving and Improving Recurrent Back-Propagation

On First-Order Meta-Learning Algorithms

Stochastic Hyperparameter Optimization through Hypernetworks

The Mechanics of n-Player Differentiable Games

Understanding Short-Horizon Bias in Stochastic Meta-Optimization

ScreenerNet: Learning Self-Paced Curriculum for Deep Neural Networks

Noisy Natural Gradient as Variational Inference

Population Based Training of Neural Networks

On Optimal Generalizability in Parametric Learning

Regularization for Deep Learning: A Taxonomy

Automatic differentiation in PyTorch

Learning with Opponent-Learning Awareness

SMASH: One-Shot Model Architecture Search through HyperNetworks

Regularizing and Optimizing LSTM Language Models

The Numerics of GANs

Concrete Dropout

Understanding Black-box Predictions via Influence Functions

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Forward and Reverse Gradient-Based Hyperparameter Optimization

Categorical Reparameterization with Gumbel-Softmax

The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables

Learning to learn by gradient descent by gradient descent

Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

Second-Order Stochastic Optimization for Machine Learning in Linear Time

Hyperparameter optimization with approximate gradient

DrMAD: Distilling Reverse-Mode Automatic Differentiation for Optimizing Hyperparameters of Deep Neural Networks

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

Deep Residual Learning for Image Recognition

Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters

Variational Dropout and the Local Reparameterization Trick

Bilevel Optimization with Nonsmooth Lower Level Problems

U-Net: Convolutional Networks for Biomedical Image Segmentation

Optimizing Neural Networks with Kronecker-factored Approximate Curvature

Non-stochastic Best Arm Identification and Hyperparameter Optimization

Gradient-based Hyperparameter Optimization through Reversible Learning

Adam: A Method for Stochastic Optimization

Regularization of Neural Networks using DropConnect

ImageNet classification with deep convolutional neural networks

Practical Bayesian Optimization of Machine Learning Algorithms

Generic Methods for Optimization-Based Modeling

Random Search for Hyper-Parameter Optimization

Gradient-Based Optimization of Hyperparameters

Automatic Detection and Segmentation of Evolving Processes in 3D Medical Images: Application to Multiple Sclerosis

Long Short-Term Memory

Design and regularization of neural networks: the optimal use of a validation set

Neural networks

Fast Exact Multiplication by the Hessian

Building a Large Annotated Corpus of English: The Penn Treebank