DARTS: Differentiable Architecture Search (2018-06-24T00:00:00.000000Z)

TL;DR

The proposed algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques.

Abstract

This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. Extensive experiments on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques. Our implementation has been made publicly available to facilitate further research on efficient architecture search algorithms.

Authors

K. Simonyan

25 papers

Yiming Yang

11 papers

Hanxiao Liu

4 papers

TL;DR

Abstract

Authors

References48 items

Understanding and Simplifying One-Shot Architecture Search

Bilevel Programming for Hyperparameter Optimization and Meta-Learning

Differentiable Neural Network Architecture Search

Neural Architecture Search with Bayesian Optimisation and Optimal Transport

Efficient Neural Architecture Search via Parameter Sharing

Regularized Evolution for Image Classifier Architecture Search

Progressive Neural Architecture Search

Simple And Efficient Architecture Search for Convolutional Neural Networks

Breaking the Softmax Bottleneck: A High-Rank RNN Language Model

Hierarchical Representations for Efficient Architecture Search

Automatic differentiation in PyTorch

Connectivity Learning in Multi-Branch Networks

Dynamic Evaluation of Neural Sequence Models

Practical Block-Wise Neural Network Architecture Generation

SMASH: One-Shot Model Architecture Search through HyperNetworks

Improved Regularization of Convolutional Neural Networks with Cutout

Regularizing and Optimizing LSTM Language Models

Learning Transferable Architectures for Scalable Image Recognition

On the State of the Art of Evaluation in Neural Language Models

ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices

Efficient Architecture Search by Network Transformation

Learning Time/Memory-Efficient Deep Architectures with Budgeted Super Networks

Accelerating Neural Architecture Search using Performance Prediction

DeepArchitect: Automatically Designing and Training Deep Architectures

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Neural Architecture Search with Reinforcement Learning

Designing Neural Network Architectures using Reinforcement Learning

Unrolled Generative Adversarial Networks

Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling

Improving Neural Language Models with a Continuous Cache

Densely Connected Convolutional Networks

SGDR: Stochastic Gradient Descent with Warm Restarts

Recurrent Highway Networks

Convolutional Neural Fabrics

Hyperparameter optimization with approximate gradient

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters

Gradient-based Hyperparameter Optimization through Reversible Learning

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Adam: A Method for Stochastic Optimization

Going deeper with convolutions

An overview of bilevel optimization

Acceleration of stochastic approximation by averaging

Hierarchical optimization: An introduction

Marktform und Gleichgewicht

BPTT length 35, and weight decay

The training takes 12 days on a single GPU

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names