A System for Massively Parallel Hyperparameter Tuning

Published in

Conference on Machine Learning and Systems(2018)

External Links:

Generate Graph

TL;DR

This work introduces a simple and robust hyperparameter optimization algorithm called ASHA, which exploits parallelism and aggressive early-stopping to tackle large-scale hyperparameters optimization problems, and shows that ASHA outperforms existing state-of-the-art hyper parameter optimization methods.

Abstract

Modern learning models are characterized by large hyperparameter spaces and long training times. These properties, coupled with the rise of parallel computing and the growing demand to productionize machine learning workloads, motivate the need to develop mature hyperparameter optimization functionality in distributed computing settings. We address this challenge by first introducing a simple and robust hyperparameter optimization algorithm called ASHA, which exploits parallelism and aggressive early-stopping to tackle large-scale hyperparameter optimization problems. Our extensive empirical results show that ASHA outperforms existing state-of-the-art hyperparameter optimization methods; scales linearly with the number of workers in distributed settings; and is suitable for massive parallelism, as demonstrated on a task with 500 workers. We then describe several design decisions we encountered, along with our associated solutions, when integrating ASHA in Determined AI's end-to-end production-quality machine learning system that offers hyperparameter tuning as a service.

Authors

Moritz Hardt

10 papers

B. Recht

5 papers

Kevin G. Jamieson

2 papers

References52 items

Optuna: A Next-generation Hyperparameter Optimization Framework

Random Search and Reproducibility for Neural Architecture Search

A Generalized Framework for Population Based Training

CHOPT : Automated Hyperparameter Optimization Framework for Cloud-Based Machine Learning Platforms

Gandiva: Introspective Cluster Scheduling for Deep Learning

A System for Massively Parallel Hyperparameter Tuning

Published in

Conference on Machine Learning and Systems(2018)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Moritz Hardt

10 papers

B. Recht

5 papers

Kevin G. Jamieson

2 papers

References52 items

Optuna: A Next-generation Hyperparameter Optimization Framework

Random Search and Reproducibility for Neural Architecture Search

A Generalized Framework for Population Based Training

CHOPT : Automated Hyperparameter Optimization Framework for Cloud-Based Machine Learning Platforms

Gandiva: Introspective Cluster Scheduling for Deep Learning

Afshin Rostamizadeh

4 papers

Ameet Talwalkar

10 papers

Ekaterina Gonina

2 papers

Jonathan Ben-tzur

1 papers

Tune: A Research Platform for Distributed Model Selection and Training

BOHB: Robust and Efficient Hyperparameter Optimization at Scale

DARTS: Differentiable Architecture Search

Population Based Training of Neural Networks

100-epoch ImageNet Training with AlexNet in 24 Minutes

Google Vizier: A Service for Black-Box Optimization

Scaling SGD Batch Size to 32K for ImageNet Training

Regularizing and Optimizing LSTM Language Models

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

Learning Curve Prediction with Bayesian Neural Networks

Hyperband: Bandit-Based Configuration Evaluation for Hyperparameter Optimization

Paleo: A Performance Model for Deep Neural Networks

Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets

Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

Selecting Near-Optimal Learners via Incremental Data Allocation

Efficient and Robust Automated Machine Learning

Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves

Batch Bayesian Optimization via Local Penalization

Non-stochastic Best Arm Identification and Hyperparameter Optimization

Going deeper with convolutions

Recurrent Neural Network Regularization

Freeze-Thaw Bayesian Optimization

One weird trick for parallelizing convolutional neural networks

Multi-Task Bayesian Optimization

Almost Optimal Exploration in Multi-Armed Bandits

Practical Bayesian Optimization of Machine Learning Algorithms

Fast cross-validation via sequential testing

Convolutional neural networks applied to house numbers digit classification

Algorithms for Hyper-Parameter Optimization

Sequential Model-Based Optimization for General Algorithm Configuration

Efficient Multi-Start Strategies for Local Search Algorithms

Building a Large Annotated Corpus of English: The Penn Treebank

Asymptotic Minimax Character of the Sample Distribution Function and of the Classical Multinomial Estimator

2019) for designing CNN and RNN cells

2018) reported validation and test perplexi

Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA

2017a) in their evaluation of Hyperband. In this variant, which we refer to as “Hyperband (by bracket),” the incumbent is recorded after the completion of each SHA bracket

2017a) suggest that Fabolas is faster than Hyperband at finding a good configuration

Announcing tensorﬂow 0.8 now with distributed computing support!

Fabolas to be a reasonable competitor for tasks

Efﬁcient and Robust Automated Machine Learning

Experimental Setup for Neural Architecture Search Benchmarks For NAS benchmarks evaluated

Reading Digits in Natural Images with Unsupervised Feature Learning

Kriging is well-suited to parallelize optimization

Gaussian process optimization in the bandit setting: No regret and experimental design

Learning Multiple Layers of Features from Tiny Images

24th Annual Conference on Learning Theory Oracle inequalities for computationally budgeted model selection

Field of Study

Computer ScienceMathematics

Journal Information

Name

arXiv: Learning

Venue Information

Name

Conference on Machine Learning and Systems

Type

conference

URL

https://mlsys.org/

Alternate Names

MLSys
Conf Mach Learn Syst