Unsupervised Progressive Learning and the STAM Architecture (2019-04-03T00:00:00.000000Z)

TL;DR

The Self-Taught Associative Memory (STAM) architecture is proposed, and two recent continual learning models, Memory Aware Synapses and Gradient Episodic Memories (GEM), are compared after adapting them in the UPL setting.

Abstract

We first pose the Unsupervised Progressive Learning (UPL) problem: an online representation learning problem in which the learner observes a non-stationary and unlabeled data stream, learning a growing number of features that persist over time even though the data is not stored or replayed. To solve the UPL problem we propose the Self-Taught Associative Memory (STAM) architecture. Layered hierarchies of STAM modules learn based on a combination of online clustering, novelty detection, forgetting outliers, and storing only prototypical features rather than specific examples. We evaluate STAM representations using clustering and classification tasks. While there are no existing learning scenarios that are directly comparable to UPL, we compare the STAM architecture with two recent continual learning models, Memory Aware Synapses (MAS) and Gradient Episodic Memories (GEM), after adapting them in the UPL setting.

Authors

James Smith

1 papers

Cameron Taylor

1 papers

Seth Baer

1 papers

TL;DR

Abstract

Authors

References35 items

Unsupervised Continual Learning Via Pseudo Labels

Unsupervised Class-Incremental Learning Through Confusion

Continual Unsupervised Representation Learning

Unsupervised Progressive Learning and the STAM Architecture

Task-Free Continual Learning

A neural data structure for novelty detection

Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines

Meta-Learning: A Survey

Lifelong Learning of Spatiotemporal Representations With Dual-Memory Recurrent Self-Organization

Task Agnostic Continual Learning Using Online Variational Bayes

Unsupervised Representation Learning by Predicting Image Rotations

Memory Aware Synapses: Learning what (not) to forget

A clustering neural network model of insect olfaction

An Overview of Multi-Task Learning in Deep Neural Networks

Gradient Episodic Memory for Continual Learning

Continual Learning with Deep Generative Replay

EMNIST: an extension of MNIST to handwritten letters

iCaRL: Incremental Classifier and Representation Learning

A novel progressive learning technique for multi-class classification

What Learning Systems do Intelligent Agents Need? Complementary Learning Systems Theory Updated

Semi-supervised Learning with Deep Generative Models

An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks

Representation Learning: A Review and New Perspectives

An Analysis of Single-Layer Networks in Unsupervised Feature Learning

Perceptual learning

Proceedings of the 7th International Conference on Database Theory

When Is ''Nearest Neighbor'' Meaningful?

Annual Review of Psychology

Proceedings of the National Academy of Sciences

Recognition

In Advances in Neural Information Processing Systems 32

Frontiers in neurorobotics

In 2017 51st Asilomar Conference on Signals

Reading Digits in Natural Images with Unsupervised Feature Learning

Gradient-based learning applied to document recognition

Field of Study

Venue Information

Name

Type

URL

Alternate Names