Analysis of the AutoML Challenge Series 2015-2018 (2019-01-01T00:00:00.000000Z)

TL;DR

This chapter analyzes the results of a machine learning competition of progressive difficulty, which was followed by a one-round AutoML challenge (PAKDD 2018), and provides details about the datasets, which were not revealed to the participants.

Abstract

The ChaLearn AutoML Challenge (The authors are in alphabetical order of last name, except the first author who did most of the writing and the second author who produced most of the numerical analyses and plots.) (NIPS 2015 – ICML 2016) consisted of six rounds of a machine learning competition of progressive difficulty, subject to limited computational resources. It was followed bya one-round AutoML challenge (PAKDD 2018). The AutoML setting differs from former model selection/hyper-parameter selection challenges, such as the one we previously organized for NIPS 2006: the participants aim to develop fully automated and computationally efficient systems, capable of being trained and tested without human intervention, with code submission. This chapter analyzes the results of these competitions and provides details about the datasets, which were not revealed to the participants. The solutions of the winners are systematically benchmarked over all datasets of all rounds and compared with canonical machine learning algorithms available in scikit-learn. All materials discussed in this chapter (data and code) have been made publicly available at http://automl.chalearn.org/.

Authors

Isabelle M Guyon

8 papers

Hugo Jair Escalante

5 papers

Sergio Escalera

31 papers

TL;DR

Abstract

Authors

References96 items

Why Most Published Research Findings Are False

Unsupervised Learning

BOHB: Robust and Efficient Hyperparameter Optimization at Scale

Efficient Neural Architecture Search via Parameter Sharing

DENSER: deep evolutionary network structured representation

Large-Scale Evolution of Image Classifiers

Fast Bayesian hyperparameter optimization on large datasets

A brief Review of the ChaLearn AutoML Challenge: Any-time Any-dataset Learning without Human Intervention

Towards Automatically-Tuned Neural Networks

Neural Architecture Search with Reinforcement Learning

Designing Neural Network Architectures using Reinforcement Learning

Theano: A Python framework for fast computation of mathematical expressions

Efficient and Robust Automated Machine Learning

Design of the 2015 ChaLearn AutoML challenge

AutoCompete: A Framework for Machine Learning Competition

Initializing Bayesian Hyperparameter Optimization via Meta-Learning

ChaLearn Looking at People Challenge 2014: Dataset and Results

OpenML: networked science in machine learning

Freeze-Thaw Bayesian Optimization

Multi-modal gesture recognition challenge 2013: dataset and results

Multi-Task Bayesian Optimization

OpenML: A Collaborative Science Platform

On statistics, computation and scalability

Collaborative hyperparameter tuning

Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures

Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms

Auto-WEKA: Automated Selection and Hyper-Parameter Optimization of Classification Algorithms

Full model selection in the space of data mining operators

Representation Learning: A Review and New Perspectives

Practical Bayesian Optimization of Machine Learning Algorithms

An Overview On Bilevel Programming

Random Search for Hyper-Parameter Optimization

Algorithms for Hyper-Parameter Optimization

Model selection for primal SVM

Hands-On Pattern Recognition: Challenges in Machine Learning, volume 1

Scikit-learn: Machine Learning in Python

Sequential Model-Based Optimization for General Algorithm Configuration

The segmented and annotated IAPR TC-12 benchmark

Nonsmooth Bilevel Programming for Hyperparameter Selection

Particle Swarm Model Selection

A Parameter-Free Classification Method for Large Scale Learning

Feature Selection with Ensembles, Artificial Variables, and Redundancy Elimination

Winning the KDD Cup Orange Challenge with Ensemble Selection

A comprehensive comparison of random forests and support vector machines for microarray-based cancer classification

Bilevel Optimization and Machine Learning

Prediction of recursive convex hull class assignments for protein residues

A New Probabilistic Approach in Rank Regression with Optimal Bayesian Partitioning

Compression-Based Averaging of Selective Naive Bayes Classifiers

L1‐regularization path algorithm for generalized linear models

Preventing Over-Fitting during Model Selection via Bayesian Regularisation of the Hyper-Parameters

An overview of bilevel optimization

An Efficient Method for Gradient-Based Adaptation of Hyperparameters in SVM Models

PerformancePrediction Challenge

Why Most Published Research Findings Are False

The elements of statistical learning: data mining, inference and prediction

The Entire Regularization Path for the Support Vector Machine

Ensemble selection from libraries of models

The Elements of Statistical Learning

Foundations of Bilevel Programming

Learning with Kernels: support vector machines, regularization, optimization, and beyond

Greedy function approximation: A gradient boosting machine.

Gaussian processes and SVM: Mean field results and leave-one-out

Bounds on Error Expectation for Support Vector Machines

Sequential MCMC for Bayesian model selection

Adaptation and Unsupervised Learning

Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms

Selection of Relevant Features and Examples in Machine Learning

Wrappers for Feature Subset Selection

In International Workshop on Automatic Machine Learning at ICML, 2018

Methods of the 2018 AutoML challenge E.1 Survey Analysis Eleven teams responded to a survey we conducted on methods used in the 2018 challenge

Combining Hyperband and Bayesian Optimization

Djajetic Implementation

Representation: Processed categorical features encoding descriptive information were made available with this data set

AutoML challenge: Rules for selecting neural network architectures for automl-gpu challenge

Challenges in Machine Learning book series. Microtome, 2011-2016

Methods for improving bayesian optimization for automl