MFES-HB: Efficient Hyperband with Multi-Fidelity Quality Measurements (2020-12-05T00:00:00.000000Z)

TL;DR

Fidelity Ensemble Surrogate (MFES) is proposed, an efficient Hyperband method that is capable of utilizing both the high-fidelity and low-f fidelity measurements to accelerate the convergence of HPO tasks.

Abstract

Hyperparameter optimization (HPO) is a fundamental problem in automatic machine learning (AutoML). However, due to the expensive evaluation cost of models (e.g., training deep learning models or training models on large datasets), vanilla Bayesian optimization (BO) is typically computationally infeasible. To alleviate this issue, Hyperband (HB) utilizes the early stopping mechanism to speed up configuration evaluations by terminating those badly-performing configurations in advance. This leads to two kinds of quality measurements: (1) many low-fidelity measurements for configurations that get early-stopped, and (2) few high-fidelity measurements for configurations that are evaluated without being early stopped. The state-of-the-art HB-style method, BOHB, aims to combine the benefits of both BO and HB. Instead of sampling configurations randomly in HB, BOHB samples configurations based on a BO surrogate model, which is constructed with the high-fidelity measurements only. However, the scarcity of high-fidelity measurements greatly hampers the efficiency of BO to guide the configuration search. In this paper, we present MFES-HB, an efficient Hyperband method that is capable of utilizing both the high-fidelity and low-fidelity measurements to accelerate the convergence of HPO tasks. Designing MFES-HB is not trivial as the low-fidelity measurements can be biased yet informative to guide the configuration search. Thus we propose to build a Multi-Fidelity Ensemble Surrogate (MFES) based on the generalized Product of Experts framework, which can integrate useful information from multi-fidelity measurements effectively. The empirical studies on the real-world AutoML tasks demonstrate that MFES-HB can achieve 3.3-8.9x speedups over the state-of-the-art approach --- BOHB.

References47 items

GARG: Anonymous Recommendation of Point-of-Interest in Mobile Networks by Graph Convolution Network

Efficient Automatic CASH via Rising Bandits

Snapshot boosting: a fast ensemble framework for deep neural networks

MMM: Multi-source Multi-net Micro-video Recommendation with Clustered Hidden Item Representation Learning

Multi-Fidelity Automatic Hyper-Parameter Tuning via Transfer Series Expansion

Bayesian Optimization Meets Bayesian Optimal Stopping

Survey on Automated Machine Learning

Practical Multi-fidelity Bayesian Optimization for Hyperparameter Tuning

Automated Machine Learning: From Principles to Practices

Noisy Blackbox Optimization with Multi-Fidelity Queries: A Tree Search Approach

BOHB: Robust and Efficient Hyperparameter Optimization at Scale

Combination of Hyperband and Bayesian Optimization for Hyperparameter Optimization in Deep Learning

Google Vizier: A Service for Black-Box Optimization

Practical Neural Network Performance Prediction for Early Stopping

Neural Collaborative Filtering

TencentBoost: A Gradient Boosting Tree System with Parameter Server

Multi-fidelity Bayesian Optimisation with Continuous Approximations

Learning Curve Prediction with Bayesian Neural Networks

Two-Stage Transfer Surrogate Model for Automatic Hyperparameter Optimization

Scalable Hyperparameter Optimization with Products of Gaussian Process Experts

Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets

Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

Multi-Information Source Optimization

Efficient and Robust Automated Machine Learning

Hyperparameter Optimization with Factorized Multilayer Perceptrons

Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves

Batch Bayesian Optimization via Local Penalization

Non-stochastic Best Arm Identification and Hyperparameter Optimization

Generalized Product of Experts for Automatic and Principled Fusion of Gaussian Process Predictions

Freeze-Thaw Bayesian Optimization

Efficient Transfer Learning Method for Automatic Hyperparameter Tuning

Multi-Task Bayesian Optimization

Collaborative hyperparameter tuning

Practical Bayesian Optimization of Machine Learning Algorithms

Algorithms for Hyper-Parameter Optimization

Sequential Model-Based Optimization for General Algorithm Configuration

Optimization

Efficient Global Optimization of Expensive Black-Box Functions

“Multi-ﬁdelity Bayesian Optimization with Max-value Entropy Search and its Parallelization”

Automated Machine Learning - Methods, Systems, Challenges

Scalable Meta-Learning for Bayesian Optimization using Ranking-Weighted Gaussian Process Ensembles

Hyperparameter optimization of deep neural networks: combining Hperband with Bayesian model selection

Deep Learning

Bayesian Optimization in High Dimensions via Random Embeddings

Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters

Products of Experts