Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions (2019-03-07T00:00:00.000000Z)

TL;DR

This work aims to adapt regularization hyperparameters for neural networks by fitting compact approximations to the best-response function, which mapshyperparameters to optimal weights and biases, and outperforms competing hyperparameter optimization methods on large-scale deep learning problems.

Abstract

Hyperparameter optimization can be formulated as a bilevel optimization problem, where the optimal parameters on the training set depend on the hyperparameters. We aim to adapt regularization hyperparameters for neural networks by fitting compact approximations to the best-response function, which maps hyperparameters to optimal weights and biases. We show how to construct scalable best-response approximations for neural networks by modeling the best-response as a single network whose hidden units are gated conditionally on the regularizer. We justify this approximation by showing the exact best-response for a shallow linear network with L2-regularized Jacobian can be represented by a similar gating mechanism. We fit this model using a gradient-based hyperparameter optimization algorithm which alternates between approximating the best-response around the current hyperparameters and optimizing the hyperparameters using the approximate best-response function. Unlike other gradient-based approaches, we do not require differentiating the training loss with respect to the hyperparameters, allowing us to tune discrete hyperparameters, data augmentation hyperparameters, and dropout probabilities. Because the hyperparameters are adapted online, our approach discovers hyperparameter schedules that can outperform fixed hyperparameter values. Empirically, our approach outperforms competing hyperparameter optimization methods on large-scale deep learning problems. We call our networks, which update their own hyperparameters online during training, Self-Tuning Networks (STNs).

Authors

Paul Vicol

3 papers

D. Duvenaud

7 papers

R. Grosse

5 papers

TL;DR

Abstract

Authors

References67 items

Bilevel Programming for Hyperparameter Optimization and Meta-Learning

Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam

Stochastic Hyperparameter Optimization through Hypernetworks

Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches

Population Based Training of Neural Networks

Automatic differentiation in PyTorch

SMASH: One-Shot Model Architecture Search through HyperNetworks

Improved Regularization of Convolutional Neural Networks with Cutout

Regularizing and Optimizing LSTM Language Models

The Numerics of GANs

Learning Curve Prediction with Bayesian Neural Networks

Neural Architecture Search with Reinforcement Learning

Unrolled Generative Adversarial Networks

Hyperband: Bandit-Based Configuration Evaluation for Hyperparameter Optimization

beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework

HyperNetworks

Hyperparameter optimization with approximate gradient

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters

Weight Uncertainty in Neural Networks

Bilevel approaches for learning of variational imaging models

Non-stochastic Best Arm Identification and Hyperparameter Optimization

Scalable Bayesian Optimization Using Deep Neural Networks

Gradient-based Hyperparameter Optimization through Reversible Learning

Deep Fried Convnets

Recurrent Neural Network Regularization

Freeze-Thaw Bayesian Optimization

Introductory Lectures on Convex Optimization - A Basic Course

Regularization of Neural Networks using DropConnect

Predicting Parameters in Deep Learning

A Bilevel Optimization Approach for Parameter Learning in Variational Models

Efficient Evolutionary Algorithm for Single-Objective Bilevel Optimization

Variational Optimization

ImageNet classification with deep convolutional neural networks

Practical Bayesian Optimization of Machine Learning Algorithms

Generic Methods for Optimization-Based Modeling

Random Search for Hyper-Parameter Optimization

Algorithms for Hyper-Parameter Optimization

Sequential Model-Based Optimization for General Algorithm Configuration

Market Structure and Equilibrium

Curriculum learning

Efficient multiple hyperparameter learning for log-linear models

An overview of bilevel optimization

Cross-Validation Optimization for Large Scale Hierarchical Classification Kernel Methods

A Trust-Region Method for Nonlinear Bilevel Programming: Algorithm and Computational Experience

The Elements of Statistical Learning

Choosing Multiple Parameters for Support Vector Machines

Practical Bilevel Optimization: Algorithms and Applications

Long Short-Term Memory

Design and regularization of neural networks: the optimal use of a validation set

Descent approaches for quadratic bilevel programming

A ‘Self-Referential’ Weight Matrix

Building a Large Annotated Corpus of English: The Penn Treebank

A primer in game theory

Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks

Sensitivity and stability analysis for nonlinear programming

Numerical continuation methods - an introduction

New Branch-and-Bound Rules for Linear Bilevel Programming

GENERATIVE ADVERSARIAL NETS

Dropout: a simple way to prevent neural networks from overfitting

Learning Multiple Layers of Features from Tiny Images

Properties of the trace and matrix derivatives

Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

We letX ∈ RN×D denote the data matrix where N is the number of training examples and D is the dimensionality of the data. We let t ∈ R denote the associated targets

Chapter 3) that the optimal solution u∗(λ) minimizing Equation

Gradient-based learning applied to document recognition

Efﬁcient Evolutionary Algorithm for Single-Objective Bilevel Optimization

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names