Parameter Prediction for Unseen Deep Architectures (2021-10-25T00:00:00.000000Z)

TL;DR

This work proposes a hypernetwork that can predict performant parameters in a single forward pass taking a fraction of a second, even on a CPU, and learns a strong representation of neural architectures enabling their analysis.

Abstract

Deep learning has been successful in automating the design of features in machine learning pipelines. However, the algorithms optimizing neural network parameters remain largely hand-designed and computationally inefficient. We study if we can use deep learning to directly predict these parameters by exploiting the past knowledge of training other networks. We introduce a large-scale dataset of diverse computational graphs of neural architectures - DeepNets-1M - and use it to explore parameter prediction on CIFAR-10 and ImageNet. By leveraging advances in graph neural networks, we propose a hypernetwork that can predict performant parameters in a single forward pass taking a fraction of a second, even on a CPU. The proposed model achieves surprisingly good performance on unseen and diverse networks. For example, it is able to predict all 24 million parameters of a ResNet-50 achieving a 60% accuracy on CIFAR-10. On ImageNet, top-5 accuracy of some of our networks approaches 50%. Our task along with the model and results can potentially lead to a new, more computationally efficient paradigm of training networks. Our model also learns a strong representation of neural architectures enabling their analysis.

Authors

M. Drozdzal

6 papers

Graham W. Taylor

6 papers

Adriana Romero-Soriano

3 papers

TL;DR

Abstract

Authors

References122 items

Data-driven Weight Initialization with Sylvester Solvers

Meta Learning Black-Box Population-Based Optimizers

GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training

High-Performance Large-Scale Image Recognition Without Normalization

Characterizing signal propagation to close the performance gap in unnormalized ResNets

Meta Learning Backpropagation And Improving It

Building LEGO Using Deep Generative Models of Graphs

Design Space for Graph Neural Networks

Are wider nets better given the same number of parameters?

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Neural Architecture Performance Prediction Using Graph Neural Networks

Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves

What is being transferred in transfer learning?

CATCH: Context-based Meta Reinforcement Learning for Transferrable Architecture Search

A Systematic Survey on Deep Generative Models for Graph Generation

Graph Structure of Neural Networks

The Computational Limits of Deep Learning

On the Bottleneck of Graph Neural Networks and its Practical Implications

The hardware lottery

Non-Local Graph Neural Networks

Language Models are Few-Shot Learners

Principled Weight Initialization for Hypernetworks

Towards Fast Adaptation of Neural Architectures with Meta Learning

Meta-Learning in Neural Networks: A Survey

Neural Graph Embedding for Neural Architecture Search

Designing Network Design Spaces

MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation

BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models

Geom-GCN: Geometric Graph Convolutional Networks

Deep graph similarity learning: a survey

Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild

Neural Predictor for Neural Architecture Search

SGAS: Sequential Greedy Architecture Search

Meta-Learning of Neural Architectures for Few-Shot Learning

A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark

Why ADAM Beats SGD for Attention Models

The Frechet Distance of training and test distribution predicts the generalization gap

CARS: Continuous Evolution for Efficient Neural Architecture Search

Once for All: Train One Network and Specialize it for Efficient Deployment

AutoML: A Survey of the State-of-the-Art

SPAGAN: Shortest Path Graph Attention Network

Faster Neural Network Training with Data Echoing

Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes

Energy and Policy Considerations for Deep Learning in NLP

Position-aware Graph Neural Networks

LGM-Net: Learning to Generate Matching Networks for Few-Shot Learning

Batch Normalization is a Cause of Adversarial Vulnerability

Searching for MobileNetV3

DAG-GNN: DAG Structure Learning with Graph Neural Networks

Knowledge Flow: Improve Upon Your Teachers

A survey on graph kernels

Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

Transfusion: Understanding Transfer Learning for Medical Imaging

HyperGAN: A Generative Model for Diverse, Performant Neural Networks

Fixup Initialization: Residual Learning Without Normalization

Graph HyperNetworks for Neural Architecture Search

SimGNN: A Neural Network Approach to Fast Graph Similarity Computation

Understanding and Simplifying One-Shot Architecture Search

Auto-Keras: An Efficient Neural Architecture Search System

DARTS: Differentiable Architecture Search

Do Better ImageNet Models Transfer Better?

Group Normalization

Regularized Evolution for Image Classifier Architecture Search

Progressive Neural Architecture Search

Graph Attention Networks

A Survey of Model Compression and Acceleration for Deep Neural Networks

Squeeze-and-Excitation Networks

SMASH: One-Shot Model Architecture Search through HyperNetworks

Learning Transferable Architectures for Scalable Image Recognition

FreezeOut: Accelerate Training by Progressively Freezing Layers

Attention is All you Need

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

Accelerating Neural Architecture Search using Performance Prediction

The Loss Surface of Deep and Wide Neural Networks

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Prototypical Networks for Few-shot Learning