AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020-03-06T00:00:00.000000Z)

TL;DR

It is shown that AutoML can go further: it is possible today to automatically discover complete machine learning algorithms just using basic mathematical operations as building blocks, by introducing a novel framework that significantly reduces human bias through a generic search space.

Abstract

Machine learning research has advanced in multiple aspects, including model structures and learning methods. The effort to automate such research, known as AutoML, has also made significant progress. However, this progress has largely focused on the architecture of neural networks, where it has relied on sophisticated expert-designed layers as building blocks---or similarly restrictive search spaces. Our goal is to show that AutoML can go further: it is possible today to automatically discover complete machine learning algorithms just using basic mathematical operations as building blocks. We demonstrate this by introducing a novel framework that significantly reduces human bias through a generic search space. Despite the vastness of this space, evolutionary search can still discover two-layer neural networks trained by backpropagation. These simple neural networks can then be surpassed by evolving directly on tasks of interest, e.g. CIFAR-10 variants, where modern techniques emerge in the top algorithms, such as bilinear interactions, normalized gradients, and weight averaging. Moreover, evolution adapts algorithms to different task types: e.g., dropout-like techniques appear when little data is available. We believe these preliminary successes in discovering machine learning algorithms from scratch indicate a promising new direction for the field.

Authors

Chen Liang

2 papers

David R. So

2 papers

Esteban Real

3 papers

TL;DR

Abstract

Authors

References114 items

Multiplicative Interactions and Where to Find Them

Meta-learning curiosity algorithms

NAS evaluation is frustratingly hard

AtomNAS: Fine-Grained End-to-End Neural Architecture Search

Improving Generalization in Meta Reinforcement Learning using Learned Objectives

Randaugment: Practical automated data augmentation with a reduced search space

Towards modular and programmable architecture search

Weight Agnostic Neural Networks

SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition

NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection

ASAP: Architecture Search, Anneal and Prune

Exploring Randomly Wired Neural Networks for Image Recognition

NAS-Bench-101: Towards Reproducible Neural Architecture Search

Random Search and Reproducibility for Neural Architecture Search

The Evolved Transformer

Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation

Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions

Designing neural networks through neuroevolution

Designing neural networks through neuroevolution

Automated Machine Learning: From Principles to Practices

ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware

MnasNet: Platform-Aware Neural Architecture Search for Mobile

Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search

Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing

Evolving simple programs for playing atari games

AutoAugment: Learning Augmentation Policies from Data

Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution

Meta-Learning Update Rules for Unsupervised Representation Learning

HOUDINI: Lifelong Learning as Program Synthesis

BLOCK-NORMALIZED GRADIENT METHOD: AN EMPIRICAL STUDY FOR TRAINING DEEP NEURAL NETWORK

Evolved Policy Gradients

Regularized Evolution for Image Classifier Architecture Search

Progressive Neural Architecture Search

Population Based Training of Neural Networks

Evolving Deep Convolutional Neural Networks for Image Classification

Mastering the game of Go without human knowledge

Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms

Practical Block-Wise Neural Network Architecture Generation

Neural Optimizer Search with Reinforcement Learning

A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets

Learning Transferable Architectures for Scalable Image Recognition

Program Synthesis

Towards Synthesizing Complex Programs From Input-Output Examples

A genetic programming approach to designing convolutional neural network architectures

RobustFill: Neural Program Learning under Noisy I/O

Learned Optimizers that Scale and Generalize

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Genetic CNN

Large-Scale Evolution of Image Classifiers

Evolving Deep Neural Networks

Towards Automatically-Tuned Neural Networks

The Power of Normalization: Faster Evasion of Saddle Points

Optimization as a Model for Few-Shot Learning

Designing Neural Network Architectures using Reinforcement Learning

DeepCoder: Learning to Write Programs

Neural Architecture Search with Reinforcement Learning

Neuro-Symbolic Program Synthesis

Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

The evolution of a generalized neural learning rule

Learning to learn by gradient descent by gradient descent

Learning to Optimize

Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

Mastering the game of Go with deep neural networks and tree search

Human-level concept learning through probabilistic program induction

Neural Programmer-Interpreters

Neural Programmer: Inducing Latent Programs with Gradient Descent

FlashMeta: a framework for inductive program synthesis

Beyond Convexity: Stochastic Quasi-Convex Optimization

An Empirical Exploration of Recurrent Network Architectures

Deep Clustered Convolutional Kernels

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Neural Turing Machines

Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

On the difficulty of training recurrent neural networks

Practical Recommendations for Gradient-Based Training of Deep Architectures