AutonoML: Towards an Integrated Framework for Autonomous Machine Learning

Published in

(2020)

External Links:

Generate Graph DownloadPDF

TL;DR

This review seeks to motivate a more expansive perspective on what constitutes an automated/autonomous ML system, alongside consideration of how best to consolidate those elements, and develops a conceptual framework to illustrate one possible way of fusing high-level mechanisms into an autonomous ML system.

Abstract

Over the last decade, the long-running endeavour to automate high-level processes in machine learning (ML) has risen to mainstream prominence, stimulated by advances in optimisation techniques and their impact on selecting ML models/algorithms. Central to this drive is the appeal of engineering a computational system that both discovers and deploys high-performance solutions to arbitrary ML problems with minimal human interaction. Beyond this, an even loftier goal is the pursuit of autonomy, which describes the capability of the system to independently adjust an ML solution over a lifetime of changing contexts. However, these ambitions are unlikely to be achieved in a robust manner without the broader synthesis of various mechanisms and theoretical frameworks, which, at the present time, remain scattered across numerous research threads. Accordingly, this review seeks to motivate a more expansive perspective on what constitutes an automated/autonomous ML system, alongside consideration of how best to consolidate those elements. In doing so, we survey developments in the following research areas: hyperparameter optimisation, multi-component models, neural architecture search, automated feature engineering, meta-learning, multi-level ensembling, dynamic adaptation, multi-objective evaluation, resource constraints, flexible user involvement, and the principles of generalisation. We also develop a conceptual framework throughout the review, augmented by each topic, to illustrate one possible way of fusing high-level mechanisms into an autonomous ML system. Ultimately, we conclude that the notion of architectural integration deserves more discussion, without which the field of automated ML risks stifling both its technical advantages and general uptake.

Authors

Katarzyna Musial

2 papers

Bogdan Gabrys

2 papers

D. Kedziora

1 papers

References415 items

Automatic recommendation of feature selection algorithms based on dataset characteristics

AlphaD3M: Machine Learning Pipeline Synthesis

Automated Machine Learning Techniques for Data Streams

Exploring Opportunistic Meta-knowledge to Reduce Search Spaces for Automated Machine Learning

PyGlove: Symbolic Programming for Automated Machine Learning

AutonoML: Towards an Integrated Framework for Autonomous Machine Learning

Published in

(2020)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Katarzyna Musial

2 papers

Bogdan Gabrys

2 papers

D. Kedziora

1 papers

References415 items

Automatic recommendation of feature selection algorithms based on dataset characteristics

AlphaD3M: Machine Learning Pipeline Synthesis

Automated Machine Learning Techniques for Data Streams

Exploring Opportunistic Meta-knowledge to Reduce Search Spaces for Automated Machine Learning

PyGlove: Symbolic Programming for Automated Machine Learning

AutoWeka4MCPS-AVATAR: Accelerating Automated Machine Learning Pipeline Composition and Optimisation

Can AutoML outperform humans? An evaluation on popular OpenML datasets using AutoML Benchmark

NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size

On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice

A Review of Meta-level Learning in the Context of Multi-component, Multi-level Evolving Prediction Systems

AutoML Pipeline Selection: Efficiently Navigating the Combinatorial Space

Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation

AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks

Adaptation Strategies for Automated Machine Learning on Evolving Data

AutoHAS: Differentiable Hyper-parameter and Architecture Search

AutoHAS: Efficient Hyperparameter and Architecture Search.

A Comprehensive Survey of Neural Architecture Search

Deep Multimodal Neural Architecture Search

FedNAS: Federated Deep Learning via Neural Architecture Search

AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data

Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodology

AutoML-Zero: Evolving Machine Learning Algorithms From Scratch

RandomNet: Towards Fully Automatic Neural Architecture Design for Multimodal Learning

Federated Neural Architecture Search

AVATAR - Machine Learning Pipeline Evaluation Using Surrogate Model

Trust in AutoML: exploring information needs for establishing trust in automated machine learning systems

An Improved Online Learning Algorithm for General Fuzzy Min-Max Neural Network

Advances and Open Problems in Federated Learning

Optimizing Millions of Hyperparameters by Implicit Differentiation

Hyperparameter Importance for Image Classification by Residual Neural Networks

Human-AI Collaboration in Data Science

Towards Meta-learning of Deep Architectures for Efficient Domain Adaptation

Towards Automated Machine Learning: Evaluation and Comparison of AutoML Approaches and Tools

AutoGAN: Neural Architecture Search for Generative Adversarial Networks

AutoML: A Survey of the State-of-the-Art

Towards Automated Semi-Supervised Learning

AutoCV Challenge Design and Baseline Results

An Open Source AutoML Benchmark

Democratizing Data Science through Interactive Curation of ML Pipelines

Searching for a Robust Neural Architecture in Four GPU Hours

A Survey on Neural Architecture Search

An ADMM Based Framework for AutoML Pipeline Configuration

Benchmark and Survey of Automated Machine Learning Frameworks

NetPyNE, a tool for data-driven multiscale modeling of brain circuits

HARK Side of Deep Learning - From Grad Student Descent to Automated Machine Learning

Expert-augmented machine learning

How Far are we from Data Mining Democratisation? A Systematic Review

Towards human-guided machine learning

Evaluating the Search Phase of Neural Architecture Search

Random Search and Reproducibility for Neural Architecture Search

Democratisation of Usable Machine Learning in Computer Vision

GAMA: Genetic Automated Machine learning Assistant

The Evolved Transformer

Machine Learning and Data Mining in Pattern Recognition

Generic adaptation strategies for automated machine learning

InferLine: ML Inference Pipeline Composition Framework

Automated Algorithm Selection: Survey and Perspectives

Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead

A Very Brief and Critical Discussion on AutoML

Automated Machine Learning: From Principles to Practices

Self Hyper-Parameter Tuning for Data Streams

Preprocessor Selection for Machine Learning Pipelines

A Hybrid Approach for Automatic Model Recommendation

Hyper-parameter optimization tools comparison for multiple object tracking applications

Sherpa: Hyperparameter Optimization for Machine Learning Models

Learning to Design RNA

ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware

A Modern Take on the Bias-Variance Tradeoff in Neural Networks

ActivMetal: Algorithm Recommendation with Active Meta Learning

Automated Machine Learning Service Composition

Benchmarking Automatic Machine Learning Frameworks

Net-Net Auto Machine Learning (AutoML) Prediction of Complex Ecosystems

Northstar: An Interactive Data Science System

Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search

Automated Machine Learning Algorithm Mining for Classification Problem

Towards AutoML in the presence of Drift: first results

Automatic Gradient Boosting

ML-Plan: Automated machine learning via hierarchical planning

(WIP) Towards the Automated Composition of Machine Learning Services

AutoModeling: Integrated Approach for Automated Model Generation by Ensemble Selection of Feature Subset and Classifier

Analysis of the Complexity of the Automatic Pipeline Generation Problem

Learning the impact of data pre-processing in data analysis

Auto-Keras: An Efficient Neural Architecture Search System

DARTS: Differentiable Architecture Search

Lessons learned from the AutoML challenge

Multi-criteria optimisation for complex learning prediction systems

PRESISTANT: Data Pre-processing Assistant

PipeDream: Fast and Efficient Pipeline Parallel DNN Training

Impact of Hyper Parameter Optimization for Cross-Project Software Defect Prediction

A Configurable Cloud-Scale DNN Processor for Real-Time AI

Fairness Definitions Explained

Hyperparameters and tuning strategies for random forest

A Comparison of Evolutionary Algorithms and Gradient-based Methods for the Optimal Control Problem

MLtuner: System Support for Automatic Machine Learning Tuning

Quantum Machine Learning

Autostacker: a compositional evolutionary learning system

Transfer Learning with Neural AutoML

Tunability: Importance of Hyperparameters of Machine Learning Algorithms

Memory-based Parameter Adaptation

100

Efficient Neural Architecture Search via Parameter Sharing

101

Regularized Evolution for Image Classifier Architecture Search

102

The Impact of Automated Parameter Optimization on Defect Prediction Models

103

Neural Feature Learning From Relational Database

104

Towards a more efficient representation of imputation operators in TPOT

105

Metalearning and Algorithm Selection: progress, state of the art and introduction to the 2018 Special Issue

106

AS-ASL: Algorithm Selection with Auto-sklearn

107

Data complexity meta-features for regression problems

108

Discovering predictive ensembles for transfer learning and meta-learning

109

The online performance estimation framework: heterogeneous ensemble learning for data streams

110

Empirical hardness of finding optimal Bayesian network structures: algorithm selection and runtime prediction

111

ATM: A distributed, collaborative, scalable system for automated machine learning

112

Using meta-learning for model type selection in predictive big data analytics

113

Simultaneous Ensemble Generation and Hyperparameter Optimization for Regression

114

Complexity vs. performance: empirical analysis of machine learning as a service

115

End-to-end Training of Differentiable Pipelines Across Machine Learning Frameworks

116

Hyperparameter Importance Across Datasets

117

Sample, Estimate, Tune: Scaling Bayesian Auto-Tuning of Data Science Pipelines

118

Progressive sampling-based Bayesian optimization for efficient and automatic machine learning model selection

119

One-Shot Visual Imitation Learning via Meta-Learning

120

The Expressive Power of Neural Networks: A View from the Width

121

Feature Engineering for Predictive Modeling using Reinforcement Learning

122

OpenML Benchmarking Suites

123

Learning how to Active Learn: A Deep Reinforcement Learning Approach

124

Learning Feature Engineering for Classification

125

Challenges of automated machine learning on causal impact analytics for policy evaluation

126

Learning Transferable Architectures for Scalable Image Recognition

127

An Overview of the Algorithm Selection Problem

128

Automatic and adaptive preprocessing for the development of predictive models

129

Evolving imputation strategies for missing data in classification problems with TPOT

130

Metalearning for choosing feature selection algorithms in data mining: Proposal of a new framework

131

One button machine for automating feature engineering in relational databases

132

Modelling Multi-Component Predictive Systems as Petri Nets

133

Improving Algorithm Selection Methods using Meta-Learning by Considering Accuracy and Run Time

134

Training Deep Networks without Learning Rates Through Coin Betting

135

RECIPE: A Grammar-Based Framework for Automatically Evolving Classification Pipelines

136

In-datacenter performance analysis of a tensor processing unit

137

A genetic programming approach to designing convolutional neural network architectures

138

Bandit-Based Model Selection for Deformable Object Manipulation

139

On the Use of Default Parameter Settings in the Empirical Evaluation of Classification Algorithms

140

mlrMBO: A Modular Framework for Model-Based Optimization of Expensive Black-Box Functions

141

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

142

Large-Scale Evolution of Image Classifiers

143

The Program System for Automated Parameter Tuning of Optimization Algorithms

144

Automated Data Cleansing through Meta-Learning

145

Multiple adaptive mechanisms for data-driven soft sensors

146

Automatic Composition and Optimization of Multicomponent Predictive Systems With an Extended Auto-WEKA

147

Understanding and optimizing the performance of distributed machine learning applications on apache spark

148

A brief Review of the ChaLearn AutoML Challenge: Any-time Any-dataset Learning without Human Intervention

149

Towards Automatically-Tuned Neural Networks

150

A Strategy for Ranking Optimization Methods using Multiple Criteria

151

ExploreKit: Automatic Feature Generation and Selection

152

Beyond Machine Learning: Autonomous Learning

153

Neural Architecture Search with Reinforcement Learning

154

Designing Neural Network Architectures using Reinforcement Learning

155

KeystoneML: Optimizing Pipelines for Large-Scale Advanced Analytics

156

Reinforcement-based Simultaneous Algorithm and its Hyperparameters Selection

157

Effects of Change Propagation Resulting from Adaptive Preprocessing in Multicomponent Predictive Systems

158

Hyperparameter Optimization Machines

159

Algorithmic Bias: From Discrimination Discovery to Fairness-aware Data Mining

160

A Survey on Evolutionary Computation Approaches to Feature Selection

161

Adapting Multicomponent Predictive Systems using Hybrid Adaptation Strategies with Auto-WEKA in Process Industry

162

Learning to learn by gradient descent by gradient descent

163

PredicT-ML: a tool for automating machine learning model building with big clinical data

164

Optimized Distributed Hyperparameter Search and Simulation for Lung Texture Classification in CT Using Hadoop

165

Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets

166

A review of automatic selection methods for machine learning algorithms and hyper-parameter values

167

Bayesian Hyperparameter Optimization for Ensemble Learning

168

Automated Parameter Optimization of Classification Techniques for Defect Prediction Models

169

Towards Automatic Composition of Multicomponent Predictive Systems

170

Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

171

Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science

172

A recommendation system for meta-modeling: A meta-learning based approach

173

FLASH: Fast Bayesian Optimization for Data Analytic Pipelines

174

Coin Betting and Parameter-Free Online Learning

175

Automating Biomedical Data Science Through Tree-Based Pipeline Optimization

176

Deep Residual Learning for Image Recognition

177

Efficient and Robust Automated Machine Learning

178

Deep feature synthesis: Towards automating data science endeavors

179

Learning hyperparameter optimization initializations

180

Hidden Technical Debt in Machine Learning Systems

181

Having a Blast: Meta-Learning and Heterogeneous Ensembles for Data Streams

182

Fast Algorithm Selection Using Learning Curves

183

Hyperparameter Search Space Pruning - A New Component for Sequential Model-Based Hyperparameter Optimization

184

Efficient Hyper-parameter Optimization for NLP Applications

185

Automating model search for large scale machine learning

186

Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves

187

On sequences of different adaptive mechanisms in non-stationary regression problems

188

Beyond Manual Tuning of Hyperparameters

189

Model-Based Multi-objective Optimization: Taxonomy, Multi-Point Proposal, Toolbox and Benchmark

190

Automatic breast density classification using a convolutional neural network architecture search procedure

191

Flock: Hybrid Crowd-Machine Learning Classifiers

192

Non-stochastic Best Arm Identification and Hyperparameter Optimization

193

Towards cost-sensitive adaptation: When is it worth updating your predictive model?

194

Hyperparameter Search in Machine Learning

195

Efficient Benchmarking of Hyperparameter Optimizers via Surrogates

196

Initializing Bayesian Hyperparameter Optimization via Meta-Learning

197

Unsupervised Domain Adaptation by Backpropagation

198

Using Meta-mining to Support Data Mining Workflow Planning and Optimization

200

Recommending Learning Algorithms and Their Associated Hyperparameters

201

An Efficient Approach for Assessing Hyperparameter Importance

202