Causes of Catastrophic Forgetting in Class-Incremental Semantic Segmentation (2022-09-16T00:00:00.000000Z)

TL;DR

This work demonstrates that the semantic shift of the background class and a bias towards new classes are the major causes of forgetting in CiSS and shows that both causes mostly manifest themselves in deeper classification layers of the network, while the early layer of the model are not affected.

Abstract

Class-incremental learning for semantic segmentation (CiSS) is presently a highly researched field which aims at updating a semantic segmentation model by sequentially learning new semantic classes. A major challenge in CiSS is overcoming the effects of catastrophic forgetting, which describes the sudden drop of accuracy on previously learned classes after the model is trained on a new set of classes. Despite latest advances in mitigating catastrophic forgetting, the underlying causes of forgetting specifically in CiSS are not well understood. Therefore, in a set of experiments and representational analyses, we demonstrate that the semantic shift of the background class and a bias towards new classes are the major causes of forgetting in CiSS. Furthermore, we show that both causes mostly manifest themselves in deeper classification layers of the network, while the early layers of the model are not affected. Finally, we demonstrate how both causes are effectively mitigated utilizing the information contained in the background, with the help of knowledge distillation and an unbiased cross-entropy loss.

Authors

Tobias Kalb

2 papers

J. Beyerer

2 papers

TL;DR

Abstract

Authors

References45 items

Probing Representation Forgetting in Supervised and Unsupervised Continual Learning

An Empirical Investigation of the Role of Pre-training in Lifelong Learning

Incremental Learning in Semantic Segmentation from Image Labels

Similarity and Matching of Neural Network Representations

RECALL: Replay-based Continual Learning in Semantic Segmentation

Continual Learning for Class- and Domain-Incremental Semantic Segmentation

Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation

Continual Learning in Deep Networks: an Analysis of the Last Layer

Self-Supervised Training Enhances Online Continual Learning

Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations

Self-Training for Class-Incremental Semantic Segmentation

PLOP: Learning without Forgetting for Continual Semantic Segmentation

Class-Incremental Learning: Survey and Performance Evaluation on Image Classification

Linear Mode Connectivity in Multitask and Continual Learning

What is being transferred in transfer learning?

Brain-inspired replay for continual learning with artificial neural networks

Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics

Class-Incremental Learning for Semantic Segmentation Re-Using Neither Old Data Nor Old Labels

Modeling the Background for Incremental Learning in Semantic Segmentation

Regularization Shortcomings for Continual Learning

REMIND Your Neural Network to Prevent Catastrophic Forgetting

A Continual Learning Survey: Defying Forgetting in Classification Tasks

Toward Understanding Catastrophic Forgetting in Continual Learning

Incremental Learning Techniques for Semantic Segmentation

Learning a Unified Classifier Incrementally via Rebalancing

Similarity of Neural Network Representations Revisited

Experience Replay for Continual Learning

Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines

Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data

Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation

Memory Aware Synapses: Learning what (not) to forget

Rethinking Atrous Convolution for Semantic Image Segmentation

Continual Learning with Deep Generative Replay

Continual Learning Through Synaptic Intelligence

Overcoming catastrophic forgetting in neural networks

iCaRL: Incremental Classifier and Representation Learning

Learning without Forgetting

Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks

U-Net: Convolutional Networks for Biomedical Image Segmentation

Distilling the Knowledge in a Neural Network

Catastrophic forgetting in connectionist networks

ERFNet: Efficient Residual Factorized ConvNet for Real-Time Semantic Segmentation

The Pascal Visual Object Classes Challenge 2006 ( VOC 2006 ) Results

Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem

The semantic shift of the background class can be addressed with the unbiased cross entropy loss (UNCE) [6]

Field of Study

Venue Information

Name

Type

URL

Alternate Names