Learning curves for decision making in supervised machine learning: a survey (2022-01-28T00:00:00.000000Z)

TL;DR

A framework that categorises learning curve approaches using three criteria: the decision-making situation they address, the intrinsic learning curve question they answer and the type of resources they use is contributed.

Abstract

Learning curves are a concept from social sciences that has been adopted in the context of machine learning to assess the performance of a learning algorithm with respect to a certain resource, e.g., the number of training examples or the number of training iterations. Learning curves have important applications in several machine learning contexts, most notably in data acquisition, early stopping of model training, and model selection. For instance, learning curves can be used to model the performance of the combination of an algorithm and its hyperparameter configuration, providing insights into their potential suitability at an early stage and often expediting the algorithm selection process. Various learning curve models have been proposed to use learning curves for decision making. Some of these models answer the binary decision question of whether a given algorithm at a certain budget will outperform a certain reference performance, whereas more complex models predict the entire learning curve of an algorithm. We contribute a framework that categorises learning curve approaches using three criteria: the decision-making situation they address, the intrinsic learning curve question they answer and the type of resources they use. We survey papers from the literature and classify them into this framework.

Authors

F. Mohr

2 papers

J. N. V. Rijn

3 papers

TL;DR

Abstract

Authors

References87 items

Multi-Objective Hyperparameter Optimization in Machine Learning—An Overview

Fast and Informative Model Selection Using Learning Curve Cross-Validation

The Shape of Learning Curves: A Review

Recommending Training Set Sizes for Classification

Risk Monotonicity in Statistical Learning

Small Data, Big Decisions: Model Selection in the Small-Data Regime

Optimal Regularization Can Mitigate Double Descent

Deep double descent: where bigger models and more data hurt

Making Learners (More) Monotone

Estimating the Required Training Dataset Size for Transmitter Classification Using Deep Learning

Approximating Learning Curves for Imbalanced Big Data with Limited Labels

Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods

Performance Prediction Based on Neural Architecture Features

Minimizers of the Empirical Risk and Risk Monotonicity

Inductive Transfer for Neural Architecture Optimization

Don't Rule Out Simple Models Prematurely: A Large Scale Benchmark Comparing Linear and Non-linear Classifiers in OpenML

Is your dataset big enough? Sample size requirements when using artificial neural networks for discrete choice analysis

Efficient benchmarking of algorithm configurators via model-based surrogates

Efficient Hyperparameter Optimization in Convolutional Neural Networks by Learning Curves Prediction

Progressive sampling-based Bayesian optimization for efficient and automatic machine learning model selection

Speeding up Hyper-parameter Optimization by Extrapolation of Learning Curves Using Previous Builds

Accelerating Neural Architecture Search using Performance Prediction

Learning Curve Prediction with Bayesian Neural Networks

Predicting EEG Sample Size Required for Classification Calibration

Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets

Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

Selecting Near-Optimal Learners via Incremental Data Allocation

How much data is needed to train a medical image deep learning system to achieve necessary high accuracy

Cost-Efficient Sampling for Performance Prediction of Configurable Systems (T)

Fast Algorithm Selection Using Learning Curves

Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves

Non-stochastic Best Arm Identification and Hyperparameter Optimization

Freeze-Thaw Bayesian Optimization

The Dipping Phenomenon

Sample size planning for classification models.

Prediction of Learning Curves in Machine Translation

Predicting sample size required for classification performance

Scikit-learn: Machine Learning in Python

Active Testing Strategy to Predict the Best Classification Algorithm via Sampling and Metalearning

Resource-aware annotation through active learning

Learning curves in classification with microarray data.

Maximizing classifier utility when there are data acquisition and modeling costs

An Iterative Process for Building Learning Curves and Predicting Relative Performance of Classifiers

An Evaluation of Progressive Sampling for Imbalanced Data Sets

Maximizing classifier utility when training data is costly

Predicting relative performance of classifiers from samples

Improving Progressive Sampling via Meta-learning on Learning Curves

Learning from Little: Comparison of Classifiers Given Little Training

Predicting the Relationship Between the Size of Training Sample and the Predictive Power of Classifiers

Improving Progressive Sampling via Meta-learning

Learning When Training Data are Costly: The Effect of Class Distribution on Tree Induction

Tree Induction Vs Logistic Regression: A Learning Curve Analysis

Modelling Classification Performance for Large Data Sets

On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes

Mining high-speed data streams

Efficient progressive sampling

Parameter Convergence and Learning Curves for Neural Networks

Nonlinear versus Linear Models in Functional Neuroimaging: Learning Curves and Generalization Crossover

Static Versus Dynamic Sampling for Data Mining

Learning Curves: Asymptotic Values and Rate of Convergence

Statistical Theory of Learning Curves under Entropic Loss Criterion

Learning Curves, Model Selection and Complexity of Neural Networks

Statistical mechanics of learning from examples.

Linear and Nonlinear Extension of the Pseudo-Inverse Solution for Learning Boolean Functions

Potential

A heuristic approach to reinforcement learning control systems

Metalearning: Applications to Automated Machine Learning and Data Mining

Towards Model Selection using Learning Curve Cross-Validation

Machine learning for data streams: with practical examples in MOA

Deep Learning

Improving data mining utility with projective sampling

Active Learning Literature Survey

Selecting classifiers using metalearning with sampling landmarks and data characterization

Predicting and Optimizing Classifier Utility with the Power Law

Wrapped progressive sampling search for optimizing learning algorithm parameters

Modeling Performance of Different Classification Methods : Deviation from the Power Law