Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

Published in

Journal of machine learning research(2016)

External Links:

Generate Graph

TL;DR

A novel algorithm is introduced, Hyperband, for hyperparameter optimization as a pure-exploration non-stochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations.

Abstract

Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation and early-stopping. We formulate hyperparameter optimization as a pure-exploration non-stochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations. We introduce a novel algorithm, Hyperband, for this framework and analyze its theoretical properties, providing several desirable guarantees. Furthermore, we compare Hyperband with popular Bayesian optimization methods on a suite of hyperparameter optimization problems. We observe that Hyperband can provide over an order-of-magnitude speedup over our competitor set on a variety of deep-learning and kernel-based learning problems.

Authors

Lisha Li

2 papers

Kevin G. Jamieson

2 papers

Giulia DeSalvo

1 papers

References57 items

Google Vizier: A Service for Black-Box Optimization

Multi-fidelity Bayesian Optimisation with Continuous Approximations

Bayesian Optimization with Robust Bayesian Neural Networks

Learning Curve Prediction with Bayesian Neural Networks

Hyperband: Bandit-Based Configuration Evaluation for Hyperparameter Optimization

Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

Published in

Journal of machine learning research(2016)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Lisha Li

2 papers

Kevin G. Jamieson

2 papers

Giulia DeSalvo

1 papers

References57 items

Google Vizier: A Service for Black-Box Optimization

Multi-fidelity Bayesian Optimisation with Continuous Approximations

Bayesian Optimization with Robust Bayesian Neural Networks

Learning Curve Prediction with Bayesian Neural Networks

Hyperband: Bandit-Based Configuration Evaluation for Hyperparameter Optimization

Afshin Rostamizadeh

4 papers

Ameet Talwalkar

10 papers

Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets

A Stratified Analysis of Bayesian Optimization Methods

TL;DR

Abstract

Authors

References57 items

Google Vizier: A Service for Black-Box Optimization

Multi-fidelity Bayesian Optimisation with Continuous Approximations

Bayesian Optimization with Robust Bayesian Neural Networks

Learning Curve Prediction with Bayesian Neural Networks

Hyperband: Bandit-Based Configuration Evaluation for Hyperparameter Optimization

TL;DR

Abstract

Authors

References57 items

Google Vizier: A Service for Black-Box Optimization

Multi-fidelity Bayesian Optimisation with Continuous Approximations

Bayesian Optimization with Robust Bayesian Neural Networks

Learning Curve Prediction with Bayesian Neural Networks

Hyperband: Bandit-Based Configuration Evaluation for Hyperparameter Optimization

Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets

A Stratified Analysis of Bayesian Optimization Methods

Efficient and Robust Automated Machine Learning

Thoughts on Massively Scalable Gaussian Processes

Optimization as Estimation with Gaussian Processes in Bandit Settings

Automating model search for large scale machine learning

Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves

Simple regret for infinitely many armed bandits

High Dimensional Bayesian Optimisation and Bandits via Additive Models

Non-stochastic Best Arm Identification and Hyperparameter Optimization

Scalable Bayesian Optimization Using Deep Neural Networks

Initializing Bayesian Hyperparameter Optimization via Meta-Learning

Using Meta-Learning to Initialize Bayesian Optimization of Hyperparameters

On the Complexity of Best-Arm Identification in Multi-Armed Bandit Models

Freeze-Thaw Bayesian Optimization

Best-arm identification algorithms for multi-armed bandits in the fixed confidence setting

lil' UCB : An Optimal Exploration Algorithm for Multi-Armed Bandits

Multi-Task Bayesian Optimization

Gaussian Process Optimization with Mutual Information

Least Squares Revisited: Scalable Approaches for Multi-class Prediction

Bayesian Optimization in High Dimensions via Random Embeddings

Almost Optimal Exploration in Multi-Armed Bandits

Sharp analysis of low-rank kernel matrix approximations

Practical Bayesian Optimization of Machine Learning Algorithms

Fast cross-validation via sequential testing

Convolutional neural networks applied to house numbers digit classification

Random Search for Hyper-Parameter Optimization

Oracle inequalities for computationally budgeted model selection

Algorithms for Hyper-Parameter Optimization

Information Rates of Nonparametric Gaussian Process Methods

Sequential Model-Based Optimization for General Algorithm Configuration

Regret Bounds for Gaussian Process Bandit Problems

On the Impact of Kernel Approximation on Learning Accuracy

Pure Exploration in Multi-armed Bandits Problems

Efficient Multi-Start Strategies for Local Search Algorithms

Empirical Bernstein stopping

Random Features for Large-Scale Kernel Machines

An empirical evaluation of deep architectures on problems with many factors of variation

Fast rates for support vector machines using Gaussian kernels

Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems

Bandit Problems

In Defense of One-Vs-All Classification

The Racing Algorithm: Model Selection for Lazy Learners

The Power of Adaptivity in Identifying Statistical Alternatives

Gaussian Process Bandit Optimisation with Multi-fidelity Evaluations

Auto-WEKA : combined selection and hyperparameter optimization of supervised machine learning algorithms

Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters

Reading Digits in Natural Images with Unsupervised Feature Learning

Learning Multiple Layers of Features from Tiny Images

Gaussian process optimization in the bandit setting: No regret and experimental design

ESTIMATING THE APPROXIMATION ERROR IN LEARNING THEORY

–armed Bandits

Field of Study

Journal Information

Name

Page

Volume

Venue Information

Name

Type

URL

Alternate Names