Overcoming catastrophic forgetting in neural networks (2016-12-02T00:00:00.000000Z)

TL;DR

It is shown that it is possible to overcome the limitation of connectionist models and train networks that can maintain expertise on tasks that they have not experienced for a long time and selectively slowing down learning on the weights important for previous tasks.

Abstract

Significance Deep neural networks are currently the most successful machine-learning technique for solving a variety of tasks, including language translation, image classification, and image generation. One weakness of such models is that, unlike humans, they are unable to learn multiple tasks sequentially. In this work we propose a practical solution to train such models sequentially by protecting the weights important for previous tasks. This approach, inspired by synaptic consolidation in neuroscience, enables state of the art results on multiple reinforcement learning problems experienced sequentially. The ability to learn tasks in a sequential fashion is crucial to the development of artificial intelligence. Until now neural networks have not been capable of this and it has been widely thought that catastrophic forgetting is an inevitable feature of connectionist models. We show that it is possible to overcome this limitation and train networks that can maintain expertise on tasks that they have not experienced for a long time. Our approach remembers old tasks by selectively slowing down learning on the weights important for those tasks. We demonstrate our approach is scalable and effective by solving a set of classification tasks based on a hand-written digit dataset and by learning several Atari 2600 games sequentially.

Authors

References48 items

Computational principles of synaptic memory consolidation

What Learning Systems do Intelligent Agents Need? Complementary Learning Systems Theory Updated

Progressive Neural Networks

Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning

Policy Distillation

Deep Reinforcement Learning with Double Q-Learning

Labelling and optical erasure of synaptic memory traces in the motor cortex

Computational principles of biological memory

Weight Uncertainty in Neural Network

Weight Uncertainty in Neural Networks

Synaptic sampling: A connection between PSP variability and uncertainty explains neurophysiological observations

Branch-specific dendritic Ca2+ spikes cause persistent synaptic plasticity

Human-level control through deep reinforcement learning

Sleep promotes branch-specific formation of dendritic spines after learning

An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks

Compete to Compute

Context-dependent computation by recurrent dynamics in prefrontal cortex

ELLA: An Efficient Lifelong Learning Algorithm

Revisiting Natural Gradient for Deep Networks

ImageNet classification with deep convolutional neural networks

The Arcade Learning Environment: An Evaluation Platform for General Agents

Context Tree Switching

Synapses with short-term plasticity are optimal estimators of presynaptic membrane potentials

Stably maintained dendritic spines are associated with lifelong memories

Tag-Trigger-Consolidation: A Model of Early and Late Long-Term-Potentiation and Depression

Universal Intelligence: A Definition of Machine Intelligence

Making Working Memory Work: A Computational Model of Learning in the Prefrontal Cortex and Basal Ganglia

Cascade Models of Synaptically Stored Memories

Laplace Propagation

Using Noise to Compute Error Surfaces in Connectionist Networks: A Novel Means of Reducing Catastrophic Forgetting

Multiple Model-Based Reinforcement Learning

Catastrophic forgetting in connectionist networks

CHILD: A First Step Towards Continual Learning

Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory.

Original Contribution: Uniqueness of the weights for minimal feedforward nets with a given input-output map

A Practical Bayesian Framework for Backpropagation Networks

Connectionist models of recognition memory: constraints imposed by learning and forgetting functions.

The Forget-me-not Process

Deep Learning

Cognitive control over learning: creating, clustering, and generalizing task-set structure.

The mnist database of handwritten digits

URL http://papers.nips.cc/ paper/2444-laplace-propagation.pdf

The Frkhet Distance between Multivariate Normal Distributions

An integrative theory of prefrontal cortex function.

Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem

Theory of the backpropagation neural network

Modeling Brain Function

7500 The target network in Q-learning is updated to the policy network every 7500 step