Model Zoo: A Growing Brain That Learns Continually (2021-06-06T00:00:00.000000Z)

TL;DR

This paper argues that continual learning methods can benefit by splitting the capacity of the learner across multiple models, and develops a method named Model Zoo which grows an ensemble of small models, each of which is trained during one episode of continual learning.

Abstract

This paper argues that continual learning methods can benefit by splitting the capacity of the learner across multiple models. We use statistical learning theory and experimental analysis to show how multiple tasks can interact with each other in a non-trivial fashion when a single model is trained on them. The generalization error on a particular task can improve when it is trained with synergistic tasks, but can also deteriorate when trained with competing tasks. This theory motivates our method named Model Zoo which, inspired from the boosting literature, grows an ensemble of small models, each of which is trained during one episode of continual learning. We demonstrate that Model Zoo obtains large gains in accuracy on a variety of continual learning benchmark problems. Code is available at https://github.com/grasp-lyrl/modelzoo_continual.

Authors

Rahul Ramesh

1 papers

P. Chaudhari

1 papers

TL;DR

Abstract

Authors

References79 items

Statistical Learning Theory

Omnidirectional Transfer for Quasilinear Lifelong Learning

Encoders and Ensembles for Task-Free Continual Learning

TAG: Task-based Accumulated Gradients for Lifelong learning

Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping

Linear Mode Connectivity in Multitask and Continual Learning

GDumb: A Simple Approach that Questions Our Progress in Continual Learning

Brain-inspired replay for continual learning with artificial neural networks

A No-Free-Lunch Theorem for MultiTask Learning

On the Theory of Transfer Learning: The Importance of Task Diversity

Understanding the Role of Training Regimes in Continual Learning

Continual Deep Learning by Functional Regularisation of Memorable Past

Generative Feature Replay For Class-Incremental Learning

Federated Continual Learning with Weighted Inter-client Transfer

Few-Shot Learning via Learning the Representation, Provably

BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning

On the Value of Target Data in Transfer Learning

AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning

Orthogonal Gradient Descent for Continual Learning

Improved Schemes for Episodic Memory-based Lifelong Learning

Three scenarios for continual learning

Branched Multi-Task Networks: Deciding what layers to share

Recurrence is required to capture the representational dynamics of the human visual system

On Tiny Episodic Memories in Continual Learning

Scalable and Order-robust Continual Learning with Additive Parameter Decomposition

A Unifying Bayesian View of Continual Learning

Functional Regularisation for Continual Learning using Gaussian Processes

Don't forget, there is more than forgetting: new metrics for Continual Learning

A System for Massively Parallel Hyperparameter Tuning

Efficient Lifelong Learning with A-GEM

Learning to Learn without Forgetting By Maximizing Transfer and Minimizing Interference

Tune: A Research Platform for Distributed Model Selection and Training

Towards Robust Evaluations of Continual Learning

Reinforced Continual Learning

Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence

Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights

Overcoming catastrophic forgetting with hard attention to the task

Memory Aware Synapses: Learning what (not) to forget

PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning

Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning

Variational Continual Learning

Gradient Episodic Memory for Continual Learning

Learning multiple visual domains with residual adapters

Continual Learning with Deep Generative Replay

Overcoming Catastrophic Forgetting by Incremental Moment Matching

Continual Learning Through Synaptic Intelligence

PathNet: Evolution Channels Gradient Descent in Super Neural Networks

Overcoming catastrophic forgetting in neural networks

iCaRL: Incremental Classifier and Representation Learning

Expert Gate: Lifelong Learning with a Network of Experts

Online Contrastive Divergence with Generative Replay: Experience Replay without Storing Data

Learning modular neural network policies for multi-task and multi-robot transfer

Progressive Neural Networks

Matching Networks for One Shot Learning

Wide Residual Networks

Cross-Stitch Networks for Multi-task Learning

Lifelong Learning with Non-i.i.d. Tasks

Learning Task Grouping and Overlap in Multi-task Learning

A theory of learning from different domains

Linear Algorithms for Online Multitask Classification

A notion of task relatedness yielding provable multiple-task learning guarantees

Learning from Multiple Sources

Bounds for Linear Multi-Task Learning

Learning Multiple Tasks with Kernel Methods

A Model of Inductive Bias Learning

Boosting Algorithms as Gradient Descent

Exploiting Generative Models in Discriminative Classifiers

Arcing classifier (with discussion and a rejoinder by the author)

Catastrophic Forgetting, Rehearsal and Pseudorehearsal

Learning how to learn

iii) To impose a strict constraint on the computational complexity of each episode

Nevergrad -A gradient-free optimization platform

Split-CIFAR100

Split-MNIST

has 20 tasks with each task consisting of 5 labels

Boosting: Foundations and Algorithms