Learning Transferable Architectures for Scalable Image Recognition (2017-07-21T00:00:00.000000Z)

TL;DR

This paper proposes to search for an architectural building block on a small dataset and then transfer the block to a larger dataset and introduces a new regularization technique called ScheduledDropPath that significantly improves generalization in the NASNet models.

Abstract

Developing neural network image classification models often requires significant architecture engineering. In this paper, we study a method to learn the model architectures directly on the dataset of interest. As this approach is expensive when the dataset is large, we propose to search for an architectural building block on a small dataset and then transfer the block to a larger dataset. The key contribution of this work is the design of a new search space (which we call the "NASNet search space") which enables transferability. In our experiments, we search for the best convolutional layer (or "cell") on the CIFAR-10 dataset and then apply this cell to the ImageNet dataset by stacking together more copies of this cell, each with their own parameters to design a convolutional architecture, which we name a "NASNet architecture". We also introduce a new regularization technique called ScheduledDropPath that significantly improves generalization in the NASNet models. On CIFAR-10 itself, a NASNet found by our method achieves 2.4% error rate, which is state-of-the-art. Although the cell is not searched for directly on ImageNet, a NASNet constructed from the best cell achieves, among the published works, state-of-the-art accuracy of 82.7% top-1 and 96.2% top-5 on ImageNet. Our model is 1.2% better in top-1 accuracy than the best human-invented architectures while having 9 billion fewer FLOPS - a reduction of 28% in computational demand from the previous state-of-the-art model. When evaluated at different levels of computational cost, accuracies of NASNets exceed those of the state-of-the-art human-designed models. For instance, a small version of NASNet also achieves 74% top-1 accuracy, which is 3.1% better than equivalently-sized, state-of-the-art models for mobile platforms. Finally, the image features learned from image classification are generically useful and can be transferred to other computer vision problems. On the task of object detection, the learned features by NASNet used with the Faster-RCNN framework surpass state-of-the-art by 4.0% achieving 43.1% mAP on the COCO dataset.

References77 items

Squeeze-and-Excitation Networks

Improved Regularization of Convolutional Neural Networks with Cutout

Focal Loss for Dense Object Detection

Proximal Policy Optimization Algorithms

Dual Path Networks

ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices

DeepArchitect: Automatically Designing and Training Deep Architectures

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Learned Optimizers that Scale and Generalize

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Genetic CNN

Large-Scale Evolution of Image Classifiers

Evolving Deep Neural Networks

Learning to Optimize Neural Nets

Shake-Shake regularization of 3-branch residual networks

Beyond Skip Connections: Top-Down Modulation for Object Detection

Feature Pyramid Networks for Object Detection

Towards Automatically-Tuned Neural Networks

Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors

PolyNet: A Pursuit of Structural Diversity in Very Deep Networks

Learning to reinforcement learn

Aggregated Residual Transformations for Deep Neural Networks

Neural Architecture Search with Reinforcement Learning

Optimization as a Model for Few-Shot Learning

Designing Neural Network Architectures using Reinforcement Learning

RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning

Neural Machine Translation in Linear Time

Xception: Deep Learning with Depthwise Separable Convolutions

HyperNetworks

WaveNet: A Generative Model for Raw Audio

Densely Connected Convolutional Networks

SGDR: Stochastic Gradient Descent with Warm Restarts

Instance Normalization: The Missing Ingredient for Fast Stylization

Layer Normalization

Neural networks with differentiable structure

Learning to learn by gradient descent by gradient descent

Convolutional Neural Fabrics

FractalNet: Ultra-Deep Neural Networks without Residuals

Deep Networks with Stochastic Depth

Identity Mappings in Deep Residual Networks

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

Revisiting Distributed Synchronous SGD

PlaNet - Photo Geolocation with Convolutional Neural Networks

Deep Residual Learning for Image Recognition

Rethinking the Inception Architecture for Computer Vision

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

An Empirical Exploration of Recurrent Network Architectures

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

FaceNet: A unified embedding for face recognition and clustering

Scalable Bayesian Optimization Using Deep Neural Networks

Trust Region Policy Optimization

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Going deeper with convolutions

Very Deep Convolutional Networks for Large-Scale Image Recognition

Microsoft COCO: Common Objects in Context

DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition

Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures

ImageNet classification with deep convolutional neural networks

Practical Bayesian Optimization of Machine Learning Algorithms

Random Search for Hyper-Parameter Optimization

Algorithms for Hyper-Parameter Optimization

A High-Throughput Screening Approach to Discovering Good Forms of Biologically Inspired Visual Representation

ImageNet: A large-scale hierarchical image database

A Hypercube-Based Encoding for Evolving Large-Scale Neural Networks

Et al

Neuroevolution: from architectures to learning

Modeling systems with internal state using evolino

Long Short-Term Memory

Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position

Inceptionv4

Dropout: a simple way to prevent neural networks from overfitting

Metalearning

Learning Multiple Layers of Features from Tiny Images

Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

GradientBased Learning Applied to Document Recognition

Gradient-based learning applied to document recognition

Learning curves for stochastic gradient descent in linear feedforward networks