Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning

Published in

Journal of machine learning research(2020)

External Links:

Generate Graph

TL;DR

PoSH Auto-sklearn is developed, which enables AutoML systems to work well on large datasets under rigid time limits by using a new, simple and meta-feature-free meta-learning technique and by employing a successful bandit strategy for budget allocation.

Abstract

Automated Machine Learning (AutoML) supports practitioners and researchers with the tedious task of designing machine learning pipelines and has recently achieved substantial success. In this paper, we introduce new AutoML approaches motivated by our winning submission to the second ChaLearn AutoML challenge. We develop PoSH Auto-sklearn, which enables AutoML systems to work well on large datasets under rigid time limits by using a new, simple and meta-feature-free meta-learning technique and by employing a successful bandit strategy for budget allocation. However, PoSH Auto-sklearn introduces even more ways of running AutoML and might make it harder for users to set it up correctly. Therefore, we also go one step further and study the design space of AutoML itself, proposing a solution towards truly hands-free AutoML. Together, these changes give rise to the next generation of our AutoML system, Auto-sklearn 2.0. We verify the improvements by these additions in an extensive experimental study on 39 AutoML benchmark datasets. We conclude the paper by comparing to other popular AutoML frameworks and Auto-sklearn 1.0, reducing the relative error by up to a factor of 4.5, and yielding a performance in 10 minutes that is substantially better than what Auto-sklearn 1.0 achieves within an hour.

Authors

Matthias Feurer

3 papers

Katharina Eggensperger

5 papers

S. Falkner

2 papers

References140 items

SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization

Auto-Pytorch: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL

Generalization in portfolio-based algorithm selection

Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves

Automated Machine Learning - a brief review at the end of the early years

Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning

Published in

Journal of machine learning research(2020)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Matthias Feurer

3 papers

Katharina Eggensperger

5 papers

S. Falkner

2 papers

References140 items

SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization

Auto-Pytorch: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL

Generalization in portfolio-based algorithm selection

Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves

Automated Machine Learning - a brief review at the end of the early years

M. Lindauer

6 papers

Practical and sample efficient zero-shot HPO

AutoML Pipeline Selection: Efficiently Navigating the Combinatorial Space

MUMBO: MUlti-task Max-value Bayesian Optimization

Array programming with NumPy

AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data

Using a thousand optimization tasks to learn hyperparameter search strategies

Extreme Algorithm Selection With Dyadic Feature Representation

FLAML: A Fast and Lightweight AutoML Library

OpenML-Python: an extensible Python API for OpenML

TPOT-SH: A Faster Optimization Algorithm to Solve the AutoML Problem on Large Datasets

DeepLine: AutoML Tool for Pipelines Generation using Deep Reinforcement Learning and Hierarchical Actions Filtering

SciPy 1.0: fundamental algorithms for scientific computing in Python

An Open Source AutoML Benchmark

Automatic Machine Learning by Pipeline Synthesis using Model-Based Reinforcement Learning and a Grammar

An ADMM Based Framework for AutoML Pipeline Configuration

Practical Multi-fidelity Bayesian Optimization for Hyperparameter Tuning

Multi-fidelity Bayesian Optimization with Max-value Entropy Search

Automated Algorithm Selection: Survey and Perspectives

Learning multiple defaults for machine learning algorithms

Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning

OBOE: Collaborative Filtering for AutoML Model Selection

BOHB: Robust and Efficient Hyperparameter Optimization at Scale

ML-Plan: Automated machine learning via hierarchical planning

Auto-Keras: An Efficient Neural Architecture Search System

Autostacker: a compositional evolutionary learning system

AutoPrognosis: Automated Clinical Prognostic Modeling via Bayesian Optimization with Structured Kernel Learning

LightGBM: A Highly Efficient Gradient Boosting Decision Tree

Speeding up Hyper-parameter Optimization by Extrapolation of Learning Curves Using Previous Builds

Bootstrapping the out-of-sample predictions for efficient and accurate cross-validation

OpenML Benchmarking Suites

Efficiency Through Procrastination: Approximately Optimal Algorithm Configuration with Runtime Guarantees

Probabilistic Matrix Factorization for Automated Machine Learning

Singularity: Scientific containers for mobility of compute

RECIPE: A Grammar-Based Framework for Automatically Evolving Classification Pipelines

Multi-fidelity Bayesian Optimisation with Continuous Approximations

Automatic construction of parallel portfolios via algorithm configuration

TPOT: A Tree-based Pipeline Optimization Tool for Automating Machine Learning

Learning Curve Prediction with Bayesian Neural Networks

Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets

Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science

XGBoost: A Scalable Tree Boosting System

Multi-Information Source Optimization

FLASH: Fast Bayesian Optimization for Data Analytic Pipelines

Automating Biomedical Data Science Through Tree-Based Pipeline Optimization

Learning hyperparameter optimization initializations

Efficient and Robust Automated Machine Learning

Sequential Model-Free Hyperparameter Tuning

The reusable holdout: Preserving validity in adaptive data analysis

Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves

Design of the 2015 ChaLearn AutoML challenge

Non-stochastic Best Arm Identification and Hyperparameter Optimization

Initializing Bayesian Hyperparameter Optimization via Meta-Learning

Automatic Configuration of Sequential Planning Portfolios

A Holistic Classification Optimization Framework with Feature Selection, Preprocessing, Manifold Learning and Classifiers

OpenML: networked science in machine learning

Freeze-Thaw Bayesian Optimization

Methods comparison

Docker: lightweight Linux containers for consistent development and deployment

ParamILS: An Automatic Algorithm Configuration Framework

Multi-Task Bayesian Optimization

Collaborative hyperparameter tuning

Almost Optimal Exploration in Multi-Armed Bandits

Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures

Towards a Framework for Designing Full Model Selection and Optimization Systems

Algorithm Selection for Combinatorial Search Problems: A Survey

Parallel SAT Solver Selection and Scheduling

Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms

Selecting Classification Algorithms with Active Testing

Resampling Methods for Meta-Model Validation with Recommendations for Evolutionary Computation

Meta-learning for evolutionary parameter optimization of classifiers

Algorithms for Hyper-Parameter Optimization

Scikit-learn: Machine Learning in Python

Sequential Model-Based Optimization for General Algorithm Configuration

Ensemble particle swarm model selection

Model Selection: Beyond the Bayesian/Frequentist Divide

Particle Swarm Model Selection

The WEKA data mining software: an update

Winning the KDD Cup Orange Challenge with Ensemble Selection

Cross-disciplinary perspectives on meta-learning for algorithm selection

Efficient Sensor Placement Optimization for Securing Large Water Distribution Networks

Analysis of the IJCNN 2007 agnostic learning vs. prior knowledge challenge

Multi-fidelity optimization via surrogate modelling

Model Selection and Assessment Using Cross-indexing

PSMS for Neural Networks on the IJCNN 2007 Agnostic vs Prior Knowledge Challenge

Matplotlib: A 2D Graphics Environment

Getting the Most Out of Ensemble Selection

Statistical Comparisons of Classifiers over Multiple Data Sets

Extremely randomized trees

Robust Reinforcement Learning

Ensemble selection from libraries of models

Representational Issues in Meta-Learning

The Elements of Statistical Learning

A Racing Algorithm for Configuring Metaheuristics

100

Reducing Rankings of Classifiers by Eliminating Redundant Classifiers

101

Greedy function approximation: A gradient boosting machine.

102

Algorithm portfolios

103

Zoomed Ranking: Selection of Classification Algorithms Based on Relevant Performance Information

104

A Comparison of Ranking Methods for Classification Algorithm Selection

105

Endangered Species: Minimum Population Grows Larger

106

A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection

107

Autmatic parameter selection by minimizing estimated error

108

Characterizing the Applicability of Classification Algorithms Using Meta-Level Learning

109

Original Contribution: Stacked generalization

110

Principles of Risk Minimization for Learning Theory

111

An analysis of approximations for maximizing submodular set functions—I

112

H2O AutoML: Scalable Automatic Machine Learning

113

Dynamic Algorithm Configuration: Foundation of a New Meta-Algorithmic Framework

114

Analysis of the AutoML Challenge Series 2015-2018

115

Towards Automatically-Tuned Deep Neural Networks

116

Automated Machine Learning: Methods, Systems, Challenges. Springer, 2019. Available for free at http://automl.org/book

117

Neural Architecture Search, pages 63–77

118

Practical Automated Machine Learning for the AutoML Challenge 2018

119

In Garnett et al

120

Towards Further Automation in AutoML

121

Scalable Gaussian process-based transfer surrogates for hyperparameter optimization

122

Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA

123

Automatic Frankensteining: Creating Complex Ensembles Autonomously

124

Dask Development Team. Dask: Library for dynamic task scheduling

125

AutoFolio: An Automatically Configured Algorithm Selector

126

Advances in meta-algorithmic software libraries for distributed automated algorithm configuration

127

Submodular Function Maximization

128

Hyperopt-Sklearn: Automatic Hyperparameter Configuration for Scikit-Learn

129

Hydra-MIP : Automated Algorithm Configuration and Selection for Mixed Integer Programming

130

Data Structures for Statistical Computing in Python

132

Krause et al., 2008), sensor set placement aims at maximizing a so-called penalty reduction

133

Online passiveaggressive algorithms

135

An Economics Approach to Hard Computational Problems

136

Neural Networks for Pattern Recognition

137

The simple plant location problem: Survey and synthesis

138

Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence (AAAI-10) Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

139

Série Scientifique Scientific Series No Unbiased Estimator of the Variance of K-fold Cross-validation No Unbiased Estimator of the Variance of K-fold Cross-validation

140

Penalty function penalty i (t): loss function L

Field of Study

Computer ScienceMathematics

Journal Information

Name

J. Mach. Learn. Res.

Page

261:1-261:61

Volume

Venue Information

Name

Journal of machine learning research

Type

journal

URL

http://www.ai.mit.edu/projects/jmlr/

Alternate Names

Journal of Machine Learning Research
J mach learn res
J Mach Learn Res