NAS-Bench-101: Towards Reproducible Neural Architecture Search (2019-02-25T00:00:00.000000Z)

TL;DR

This work introduces NAS-Bench-101, the first public architecture dataset for NAS research, which allows researchers to evaluate the quality of a diverse range of models in milliseconds by querying the pre-computed dataset.

Abstract

Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to researchers without access to large-scale computation. We aim to ameliorate these problems by introducing NAS-Bench-101, the first public architecture dataset for NAS research. To build NAS-Bench-101, we carefully constructed a compact, yet expressive, search space, exploiting graph isomorphisms to identify 423k unique convolutional architectures. We trained and evaluated all of these architectures multiple times on CIFAR-10 and compiled the results into a large dataset of over 5 million trained models. This allows researchers to evaluate the quality of a diverse range of models in milliseconds by querying the pre-computed dataset. We demonstrate its utility by analyzing the dataset as a whole and by benchmarking a range of architecture optimization algorithms.

Authors

K. Murphy

12 papers

Esteban Real

5 papers

F. Hutter

14 papers

TL;DR

Abstract

Authors

References54 items

Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization

Evaluating the Search Phase of Neural Architecture Search

Random Search and Reproducibility for Neural Architecture Search

Enumerating Unique Computational Graphs via an Iterative Graph Invariant

The Evolved Transformer

GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

BOHB: Robust and Efficient Hyperparameter Optimization at Scale

DARTS: Differentiable Architecture Search

Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution

Multi-objective Architecture Search for CNNs

Efficient Neural Architecture Search via Parameter Sharing

Regularized Evolution for Image Classifier Architecture Search

Progressive Neural Architecture Search

Decoupled Weight Decay Regularization

Squeeze-and-Excitation Networks

Improved Regularization of Convolutional Neural Networks with Cutout

Learning Transferable Architectures for Scalable Image Recognition

Efficient Architecture Search by Network Transformation

Large-Scale Evolution of Image Classifiers

Bayesian Optimization with Robust Bayesian Neural Networks

Designing Neural Network Architectures using Reinforcement Learning

Neural Architecture Search with Reinforcement Learning

Densely Connected Convolutional Networks

SGDR: Stochastic Gradient Descent with Warm Restarts

Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

Identity Mappings in Deep Residual Networks

Deep Residual Learning for Image Recognition

Rethinking the Inception Architecture for Computer Vision

Scalable Bayesian Optimization Using Deep Neural Networks

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Efficient Benchmarking of Hyperparameter Optimizers via Surrogates

Adam: A Method for Stochastic Optimization

Going deeper with convolutions

Sequence to Sequence Learning with Neural Networks

ImageNet classification with deep convolutional neural networks

Practical Bayesian Optimization of Machine Learning Algorithms

Random Search for Hyper-Parameter Optimization

Algorithms for Hyper-Parameter Optimization

2D Image Registration in CT Images Using Radial Image Descriptors

Sequential Model-Based Optimization for General Algorithm Configuration

ImageNet: A large-scale hierarchical image database

Landscapes and their correlation functions

Evolutionary Algorithms, Fitness Landscapes and Search

Simple statistical gradient-following algorithms for connectionist reinforcement learning

Correlated and uncorrelated fitness landscapes and how to tell the difference

GENERATIVE ADVERSARIAL NETS

Taking the Human Out of the Loop: A Review of Bayesian Optimization

Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters

Uci machine learning repository

Physicochemical properties of protein tertiary structure data set

Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude

The Million Song Dataset

Learning Multiple Layers of Features from Tiny Images

NAS-Bench-101

Field of Study

Venue Information

Name

Type

URL

Alternate Names