Continual learning with hypernetworks (2019-06-03T00:00:00.000000Z)

TL;DR

Insight is provided into the structure of low-dimensional task embedding spaces (the input space of the hypernetwork) and it is shown that task-conditioned hypernetworks demonstrate transfer learning.

Abstract

Artificial neural networks suffer from catastrophic forgetting when they are sequentially trained on multiple tasks. To overcome this problem, we present a novel approach based on task-conditioned hypernetworks, i.e., networks that generate the weights of a target model based on task identity. Continual learning (CL) is less difficult for this class of models thanks to a simple key feature: instead of recalling the input-output relations of all previously seen data, task-conditioned hypernetworks only require rehearsing task-specific weight realizations, which can be maintained in memory using a simple regularizer. Besides achieving state-of-the-art performance on standard CL benchmarks, additional experiments on long task sequences reveal that task-conditioned hypernetworks display a very large capacity to retain previous memories. Notably, such long memory lifetimes are achieved in a compressive regime, when the number of trainable hypernetwork weights is comparable or smaller than target network size. We provide insight into the structure of low-dimensional task embedding spaces (the input space of the hypernetwork) and show that task-conditioned hypernetworks demonstrate transfer learning. Finally, forward information transfer is further supported by empirical results on a challenging CL benchmark based on the CIFAR-10/100 image datasets.

Authors

J. Sacramento

2 papers

J. Oswald

1 papers

Christian Henning

1 papers

TL;DR

Abstract

Authors

References66 items

Large Scale Adversarial Representation Learning

Task Agnostic Continual Learning via Meta Learning

Embedded Meta-Learning: Toward more flexible deep-learning models

Improving and Understanding Variational Continual Learning

Three scenarios for continual learning

High-Fidelity Image Generation With Fewer Labels

A Style-Based Generator Architecture for Generative Adversarial Networks

Approximating the Predictive Distribution via Adversarially-Trained Hypernetworks

Experience Replay for Continual Learning

Motor primitives in space and time via targeted gain modulation in cortical networks

Large Scale GAN Training for High Fidelity Natural Image Synthesis

Overcoming Catastrophic Forgetting for Continual Learning via Model Adaptation

Generative replay with feedback connections as a general strategy for continual learning

Memory Replay GANs: learning to generate images from new categories without forgetting

Self-Net: Lifelong Learning via Continual Self-Modeling

A Scalable Approach to Multi-Context Continual Learning via Lifelong Skill Encoding

Towards Robust Evaluations of Continual Learning

Measuring and regularizing networks in function space

Progress & Compress: A scalable framework for continual learning

Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting

Continual Lifelong Learning with Neural Networks: A Review

Overcoming Catastrophic Interference using Conceptor-Aided Backpropagation

Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization

Overcoming catastrophic forgetting with hard attention to the task

Note on the quadratic penalties in elastic weight consolidation

PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning

Implicit Weight Uncertainty in Neural Networks

Variational Continual Learning

Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations

Overcoming Catastrophic Interference by Conceptors

The Persistence and Transience of Memory

Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks

Continual Learning with Deep Generative Replay

Continual Learning Through Synaptic Intelligence

Multiplicative Normalizing Flows for Variational Bayesian Neural Networks

PathNet: Evolution Channels Gradient Descent in Super Neural Networks

Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks

Overcoming catastrophic forgetting in neural networks

Least Squares Generative Adversarial Networks

A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks

HyperNetworks

What Learning Systems do Intelligent Agents Need? Complementary Learning Systems Theory Updated

Learning without Forgetting

Learning feed-forward one-shot learners

Progressive Neural Networks

Dynamic Filter Networks

Deep Residual Learning for Image Recognition

Adam: A Method for Stochastic Optimization

Conditional Generative Adversarial Nets

Generative Adversarial Nets

A Normative Theory of Forgetting: Lessons from the Fruit Fly

Controlling Recurrent Neural Networks by Conceptors

Stochastic Backpropagation and Approximate Inference in Deep Generative Models

Auto-Encoding Variational Bayes

Context-dependent computation by recurrent dynamics in prefrontal cortex

Neuromodulation of Neuronal Circuits: Back to the Future

Catastrophic forgetting in connectionist networks

Catastrophic Forgetting, Rehearsal and Pseudorehearsal

Original Contribution: Multilayer feedforward networks with a nonpolynomial activation function can approximate any function

Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks

Small capacity target networks for the permuted

Recent developments in the GAN literature already allude

|Θtrgt| = 1.37

Neuroscience-Inspired Artiﬁcial Intelligence

2016)) and again produce the weights of this target network by a hypernetwork

Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names