Rethinking Neural Operations for Diverse Tasks
Published in
Neural Information Processing Systems(2021)External Links:
TL
TL;DR
This work introduces a search space of operations called XD-Operations that mimic the inductive bias of standard multi-channel convolutions while being much more expressive: it is proved that it includes many named operations across multiple application areas.
Abstract
An important goal of AutoML is to automate-away the design of neural networks on new tasks in under-explored domains. Motivated by this goal, we study the problem of enabling users to discover the right neural operations given data from their specific domain. We introduce a search space of operations called XD-Operations that mimic the inductive bias of standard multi-channel convolutions while being much more expressive: we prove that it includes many named operations across multiple application areas. Starting with any standard backbone such as ResNet, we show how to transform it into a search space over XD-operations and how to traverse the space using a simple weight-sharing scheme. On a diverse set of tasks -- solving PDEs, distance prediction for protein folding, and music modeling -- our approach consistently yields models with lower error than baseline networks and often even lower error than expert-designed domain-specific approaches.
References68 items
1
Highly accurate protein structure prediction with AlphaFold
2
Fourier Neural Operator for Parametric Partial Differential Equations
3
HiPPO: Recurrent Memory with Optimal Polynomial Projections
4
Sparse Linear Networks with a Fixed Butterfly Structure: Theory and Practice
5