AutoML-Zero: Evolving Machine Learning Algorithms From Scratch

Published in

International Conference on Machine Learning(2020)

External Links:

Generate Graph

TL;DR

It is shown that AutoML can go further: it is possible today to automatically discover complete machine learning algorithms just using basic mathematical operations as building blocks, by introducing a novel framework that significantly reduces human bias through a generic search space.

Abstract

Machine learning research has advanced in multiple aspects, including model structures and learning methods. The effort to automate such research, known as AutoML, has also made significant progress. However, this progress has largely focused on the architecture of neural networks, where it has relied on sophisticated expert-designed layers as building blocks---or similarly restrictive search spaces. Our goal is to show that AutoML can go further: it is possible today to automatically discover complete machine learning algorithms just using basic mathematical operations as building blocks. We demonstrate this by introducing a novel framework that significantly reduces human bias through a generic search space. Despite the vastness of this space, evolutionary search can still discover two-layer neural networks trained by backpropagation. These simple neural networks can then be surpassed by evolving directly on tasks of interest, e.g. CIFAR-10 variants, where modern techniques emerge in the top algorithms, such as bilinear interactions, normalized gradients, and weight averaging. Moreover, evolution adapts algorithms to different task types: e.g., dropout-like techniques appear when little data is available. We believe these preliminary successes in discovering machine learning algorithms from scratch indicate a promising new direction for the field.

Authors

Chen Liang

4 papers

David R. So

2 papers

Esteban Real

5 papers

References114 items

Multiplicative Interactions and Where to Find Them

Meta-learning curiosity algorithms

NAS evaluation is frustratingly hard

AtomNAS: Fine-Grained End-to-End Neural Architecture Search

Improving Generalization in Meta Reinforcement Learning using Learned Objectives

AutoML-Zero: Evolving Machine Learning Algorithms From Scratch

Published in

International Conference on Machine Learning(2020)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Chen Liang

4 papers

David R. So

2 papers

Esteban Real

5 papers

References114 items

Multiplicative Interactions and Where to Find Them

Meta-learning curiosity algorithms

NAS evaluation is frustratingly hard

AtomNAS: Fine-Grained End-to-End Neural Architecture Search

Improving Generalization in Meta Reinforcement Learning using Learned Objectives

Quoc V. Le

42 papers

Randaugment: Practical automated data augmentation with a reduced search space

Towards modular and programmable architecture search

Weight Agnostic Neural Networks

SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition

NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection

ASAP: Architecture Search, Anneal and Prune

Exploring Randomly Wired Neural Networks for Image Recognition

NAS-Bench-101: Towards Reproducible Neural Architecture Search

Random Search and Reproducibility for Neural Architecture Search

The Evolved Transformer

Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation

Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions

Designing neural networks through neuroevolution

Automated Machine Learning: From Principles to Practices

ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware

MnasNet: Platform-Aware Neural Architecture Search for Mobile

Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search

Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing

Evolving simple programs for playing atari games

AutoAugment: Learning Augmentation Policies from Data

Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution

Meta-Learning Update Rules for Unsupervised Representation Learning

HOUDINI: Lifelong Learning as Program Synthesis

BLOCK-NORMALIZED GRADIENT METHOD: AN EMPIRICAL STUDY FOR TRAINING DEEP NEURAL NETWORK

Evolved Policy Gradients

Regularized Evolution for Image Classifier Architecture Search

Progressive Neural Architecture Search

Population Based Training of Neural Networks

Evolving Deep Convolutional Neural Networks for Image Classification

Mastering the game of Go without human knowledge

Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms

Practical Block-Wise Neural Network Architecture Generation

Neural Optimizer Search with Reinforcement Learning

A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets

Learning Transferable Architectures for Scalable Image Recognition

Program Synthesis

Towards Synthesizing Complex Programs From Input-Output Examples

A genetic programming approach to designing convolutional neural network architectures

RobustFill: Neural Program Learning under Noisy I/O

Learned Optimizers that Scale and Generalize

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Large-Scale Evolution of Image Classifiers

Evolving Deep Neural Networks

Towards Automatically-Tuned Neural Networks

The Power of Normalization: Faster Evasion of Saddle Points

Optimization as a Model for Few-Shot Learning

Designing Neural Network Architectures using Reinforcement Learning

DeepCoder: Learning to Write Programs

Neural Architecture Search with Reinforcement Learning

Neuro-Symbolic Program Synthesis

Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

The evolution of a generalized neural learning rule

Learning to learn by gradient descent by gradient descent

Learning to Optimize

Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

Mastering the game of Go with deep neural networks and tree search

Human-level concept learning through probabilistic program induction

Neural Programmer-Interpreters

Neural Programmer: Inducing Latent Programs with Gradient Descent

FlashMeta: a framework for inductive program synthesis

Beyond Convexity: Stochastic Quasi-Convex Optimization

An Empirical Exploration of Recurrent Network Architectures

Deep Clustered Convolutional Kernels

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Neural Turing Machines

Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

On the difficulty of training recurrent neural networks

Practical Recommendations for Gradient-Based Training of Deep Architectures

Practical Bayesian Optimization of Machine Learning Algorithms

Random Search for Hyper-Parameter Optimization

Sequential Model-Based Optimization for General Algorithm Configuration

Indirectly Encoding Neural Plasticity as a Pattern of Local Rules

Rectified Linear Units Improve Restricted Boltzmann Machines

Evolving Memory Cell Structures for Sequence Learning

Robust Reinforcement Learning

Parallelism and evolutionary algorithms

Optimal Ordered Problem Solver

Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms

Evolving Neural Networks through Augmenting Topologies

Evolution and design of distributed learning rules

Convolutional networks for images, speech, and time series

Long Short-Term Memory

Implementation of a reflective system

Use of genetic programming for the search of a new learning rule for neural networks

Acceleration of stochastic approximation by averaging

Learning representations by back-propagating errors

EURISKO: A Program That Learns New Heuristics and Domain Concepts

BEAGLE—A DARWINIAN APPROACH TO PATTERN RECOGNITION

Deep Neural Networks

Automated Machine Learning

Searching for Activation Functions

100

102

Dropout: a simple way to prevent neural networks from overfitting

103

Reading Digits in Natural Images with Unsupervised Feature Learning

104

Learning Multiple Layers of Features from Tiny Images

105

The mnist database of handwritten digits

106

Clinical applications of artificial neural networks: Evolving artificial neural networks

107

An evolutionary algorithm that constructs recurrent neural networks

108

Genetic Programming: On the Programming of Computers by Means of Natural Selection

109

The Evolution of Learning: An Experiment in Genetic Connectionism

110

A Comparative Analysis of Selection Schemes Used in Genetic Algorithms

111

The Cascade-Correlation Learning Architecture

112

Evolutionary principles in self-referential learning, or on learning how to learn: The meta-meta-. hook

113

Adaptation in Natural and Artificial Sys-tems: An Introductory Analysis with Applications to Biology

Field of Study

Computer ScienceMathematics

Venue Information

Name

International Conference on Machine Learning

Type

conference

URL

https://icml.cc/

Alternate Names

ICML
Int Conf Mach Learn

TL;DR

Abstract

Authors

References114 items

Multiplicative Interactions and Where to Find Them

Meta-learning curiosity algorithms

NAS evaluation is frustratingly hard

AtomNAS: Fine-Grained End-to-End Neural Architecture Search

Improving Generalization in Meta Reinforcement Learning using Learned Objectives

TL;DR

Abstract

Authors

References114 items

Multiplicative Interactions and Where to Find Them

Meta-learning curiosity algorithms

NAS evaluation is frustratingly hard

AtomNAS: Fine-Grained End-to-End Neural Architecture Search

Improving Generalization in Meta Reinforcement Learning using Learned Objectives

Randaugment: Practical automated data augmentation with a reduced search space

Towards modular and programmable architecture search

Weight Agnostic Neural Networks

SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition

NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection

ASAP: Architecture Search, Anneal and Prune

Exploring Randomly Wired Neural Networks for Image Recognition

NAS-Bench-101: Towards Reproducible Neural Architecture Search

Random Search and Reproducibility for Neural Architecture Search

The Evolved Transformer

Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation

Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions

Designing neural networks through neuroevolution

Designing neural networks through neuroevolution

Automated Machine Learning: From Principles to Practices

ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware

MnasNet: Platform-Aware Neural Architecture Search for Mobile

Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search

Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing

Evolving simple programs for playing atari games

AutoAugment: Learning Augmentation Policies from Data

Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution

Meta-Learning Update Rules for Unsupervised Representation Learning

HOUDINI: Lifelong Learning as Program Synthesis

BLOCK-NORMALIZED GRADIENT METHOD: AN EMPIRICAL STUDY FOR TRAINING DEEP NEURAL NETWORK

Evolved Policy Gradients

Regularized Evolution for Image Classifier Architecture Search

Progressive Neural Architecture Search

Population Based Training of Neural Networks

Evolving Deep Convolutional Neural Networks for Image Classification

Mastering the game of Go without human knowledge

Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms

Practical Block-Wise Neural Network Architecture Generation

Neural Optimizer Search with Reinforcement Learning

A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets

Learning Transferable Architectures for Scalable Image Recognition

Program Synthesis

Towards Synthesizing Complex Programs From Input-Output Examples

A genetic programming approach to designing convolutional neural network architectures

RobustFill: Neural Program Learning under Noisy I/O

Learned Optimizers that Scale and Generalize

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Genetic CNN

Large-Scale Evolution of Image Classifiers

Evolving Deep Neural Networks

Towards Automatically-Tuned Neural Networks

The Power of Normalization: Faster Evasion of Saddle Points

Optimization as a Model for Few-Shot Learning

Designing Neural Network Architectures using Reinforcement Learning

DeepCoder: Learning to Write Programs

Neural Architecture Search with Reinforcement Learning

Neuro-Symbolic Program Synthesis

Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

The evolution of a generalized neural learning rule

Learning to learn by gradient descent by gradient descent

Learning to Optimize

Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

Mastering the game of Go with deep neural networks and tree search

Human-level concept learning through probabilistic program induction

Neural Programmer-Interpreters

Neural Programmer: Inducing Latent Programs with Gradient Descent