AANG: Automating Auxiliary Learning (2022-05-27T00:00:00.000000Z)

TL;DR

This work presents an approach for automatically generating a suite of auxiliary objectives by deconstructing existing objectives within a novel unified taxonomy, identifying connections between them, and generating new ones based on the uncovered structure, and theoretically formalizes widely-held intuitions about how auxiliary learning improves generalization on the end-task.

Abstract

Auxiliary objectives, supplementary learning signals that are introduced to help aid learning on data-starved or highly complex end-tasks, are commonplace in machine learning. Whilst much work has been done to formulate useful auxiliary objectives, their construction is still an art which proceeds by slow and tedious hand-design. Intuition for how and when these objectives improve end-task performance has also had limited theoretical backing. In this work, we present an approach for automatically generating a suite of auxiliary objectives. We achieve this by deconstructing existing objectives within a novel unified taxonomy, identifying connections between them, and generating new ones based on the uncovered structure. Next, we theoretically formalize widely-held intuitions about how auxiliary learning improves generalization on the end-task. This leads us to a principled and efficient algorithm for searching the space of generated objectives to find those most useful to a specified end-task. With natural language processing (NLP) as our domain of study, we demonstrate that our automated auxiliary learning pipeline leads to strong improvements over competitive baselines across continued training experiments on a pre-trained model on 5 NLP tasks.

Authors

Graham Neubig

38 papers

Ameet Talwalkar

10 papers

Paul Michel

3 papers

TL;DR

Abstract

Authors

References67 items

Consecutive Pretraining: A Knowledge Transfer Learning Strategy with Relevant Unlabeled Data for Remote Sensing Domain

ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning

NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework

An Explanation of In-context Learning as Implicit Bayesian Inference

Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative

Auxiliary Task Update Decomposition: the Good, the Bad and the neutral

Rethinking Neural Operations for Diverse Tasks

Muppet: Massive Multi-task Representations with Pre-Finetuning

Auxiliary Task Reweighting for Minimum-data Learning

A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks

Self-training Improves Pre-training for Natural Language Understanding

Auxiliary Tasks Speed Up Learning PointGoal Navigation

Auxiliary Learning by Implicit Differentiation

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Language Models are Few-Shot Learners

Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps

Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks

Geometry-Aware Gradient Algorithms for Neural Architecture Search

Pre-trained models for natural language processing: A survey

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

A Simple Framework for Contrastive Learning of Visual Representations

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

RoBERTa: A Robustly Optimized BERT Pretraining Approach

SpanBERT: Improving Pre-training by Representing and Predicting Spans

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Searching for a Robust Neural Architecture in Four GPU Hours

SemEval-2019 Task 4: Hyperpartisan News Detection

Transfer Learning in Natural Language Processing

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Searching for MobileNetV3

SciBERT: Pretrained Contextualized Embeddings for Scientific Text

Self-Supervised Generalisation with Meta Auxiliary Learning

Learning Latent Dynamics for Planning from Pixels

Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction

Representation Learning with Contrastive Predictive Coding

DARTS: Differentiable Architecture Search

Measuring the Evolution of a Scientific Field through Citation Frames

Deep Contextualized Word Representations

Efficient Neural Architecture Search via Parameter Sharing

Decoupled Weight Decay Regularization

Data-Dependent Stability of Stochastic Gradient Descent

Neural Architecture Search with Reinforcement Learning

What makes ImageNet good for transfer learning?

Learning to Poke by Poking: Experiential Learning of Intuitive Physics

SemEval-2016 Task 6: Detecting Stance in Tweets

ChemProt-3.0: a global chemical biology diseases mapping

Train faster, generalize better: Stability of stochastic gradient descent

A Simple Named Entity Extractor using AdaBoost

Evolving Neural Networks through Augmenting Topologies

A Model of Inductive Bias Learning

Statistical Techniques for Natural Language Parsing

Multitask Learning

Speech recognition

Improved Fine-tuning by Leveraging Pre-training Data: Theory and Practice

S2ORC: The Semantic Scholar Open Research Corpus

Electra: Pre-training text encoders as discriminators rather than generators

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

Adaptive Auxiliary Task Weighting for Reinforcement Learning

Improving Language Understanding by Generative Pre-Training

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

meta-learning to adaptively weight the objectives in based on measuring each objective’s inﬂuence on E ’s validation set loss

GROWTH FUNCTIONS Lemma E.3 (Growth Recursion Under Dynamic Sampling). We consider the Stochastic Gradient update rule G

Stability and Generalization

we can get arbitrary improvements in this setting by whilst this might reduce the generalization error , it end-task which would result in large increase in the improvements on the dependence on T

Note that (¯ β = λ ∗ β ∗ ≤ β e ) always. So we get compared to Theorem E.2. 5. We can optimize λ e , λ a to minimize (cid:15)

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names