Progressive Neural Architecture Search (2017-12-02T00:00:00.000000Z)

TL;DR

This work proposes a new method for learning the structure of convolutional neural networks (CNNs) that is more efficient than recent state-of-the-art methods based on reinforcement learning and evolutionary algorithms using a sequential model-based optimization (SMBO) strategy.

Abstract

We propose a new method for learning the structure of convolutional neural networks (CNNs) that is more efficient than recent state-of-the-art methods based on reinforcement learning and evolutionary algorithms. Our approach uses a sequential model-based optimization (SMBO) strategy, in which we search for structures in order of increasing complexity, while simultaneously learning a surrogate model to guide the search through structure space. Direct comparison under the same search space shows that our method is up to 5 times more efficient than the RL method of Zoph et al. (2018) in terms of number of models evaluated, and 8 times faster in terms of total compute. The structures we discover in this way achieve state of the art classification accuracies on CIFAR-10 and ImageNet.

Authors

K. Murphy

12 papers

Li-Jia Li

8 papers

Li Fei-Fei

37 papers

TL;DR

Abstract

Authors

References53 items

DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures

PPP-Net: Platform-aware Progressive Search for Pareto-optimal Neural Architectures

Efficient Neural Architecture Search via Parameter Sharing

Regularized Evolution for Image Classifier Architecture Search

Simple And Efficient Architecture Search for Convolutional Neural Networks

Hierarchical Representations for Efficient Architecture Search

Squeeze-and-Excitation Networks

Practical Network Blocks Design with Q-Learning

SMASH: One-Shot Model Architecture Search through HyperNetworks

Improved Regularization of Convolutional Neural Networks with Cutout

Learning Transferable Architectures for Scalable Image Recognition

Proximal Policy Optimization Algorithms

Reinforcement Learning for Architecture Search by Network Transformation

Dual Path Networks

ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices

Efficient Architecture Search by Network Transformation

Learning Deep ResNet Blocks Sequentially using Boosting Theory

Accelerating Neural Architecture Search using Performance Prediction

DeepArchitect: Automatically Designing and Training Deep Architectures

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Genetic CNN

Large-Scale Evolution of Image Classifiers

Evolving Deep Neural Networks

Towards Automatically-Tuned Neural Networks

PolyNet: A Pursuit of Structural Diversity in Very Deep Networks

Aggregated Residual Transformations for Deep Neural Networks

Neural Architecture Search with Reinforcement Learning

Designing Neural Network Architectures using Reinforcement Learning

Densely Connected Convolutional Networks

SGDR: Stochastic Gradient Descent with Warm Restarts

AdaNet: Adaptive Structural Learning of Artificial Neural Networks

Deep Residual Learning for Image Recognition

Rethinking the Inception Architecture for Computer Vision

Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves

On a Family of Decomposable Kernels on Sequences

Adam: A Method for Stochastic Optimization

Going deeper with convolutions

Very Deep Convolutional Networks for Large-Scale Image Recognition

Microsoft COCO: Common Objects in Context

ImageNet classification with deep convolutional neural networks

Exploiting compositionality to explore a large space of model structures

Practical Bayesian Optimization of Machine Learning Algorithms

Sequential Model-Based Optimization for General Algorithm Configuration

ImageNet: A large-scale hierarchical image database

Evolving Neural Networks through Augmenting Topologies

AdaNet: Adaptive Structural Learning of Artificial Neural Networks

Neuroevolution: A different kind of deep learning, July 2017

SGDR: Stochastic Gradient Descent with Restarts

Taking the Human Out of the Loop: A Review of Bayesian Optimization

Learning Multiple Layers of Features from Tiny Images

Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

The total computational cost for AmoebaNet consists of an architecture search and a reranking phase

The search phase trains 20K models each for 25 epochs. The rest of the computation is the same as AmoebaNet-B

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names