Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation (2021-06-29T00:00:00.000000Z)

TL;DR

Local POD is proposed, a multi-scale pooling distillation scheme that preserves long- and short-range spatial relationships at feature level that significantly outperforms state-of-the-art methods in existing CSS scenarios, as well as in newly proposed challenging benchmarks.

Abstract

Deep learning approaches are nowadays ubiquitously used to tackle computer vision tasks such as semantic segmentation, requiring large datasets and substantial computational power. Continual learning for semantic segmentation (CSS) is an emerging trend that consists in updating an old model by sequentially adding new classes. However, continual learning methods are usually prone to catastrophic forgetting. This issue is further aggravated in CSS where, at each step, old classes from previous iterations are collapsed into the background. In this paper, we propose Local POD, a multi-scale pooling distillation scheme that preserves long- and short-range spatial relationships at feature level. Furthermore, we design an entropy-based pseudo-labelling of the background w.r.t. classes predicted by the old model to deal with background shift and avoid catastrophic forgetting of the old classes. Finally, we introduce a novel rehearsal method that is particularly suited for segmentation. Our approach, called PLOP, significantly outperforms state-of-the-art methods in existing CSS scenarios, as well as in newly proposed challenging benchmarks.

Authors

M. Cord

10 papers

Arthur Douillard

3 papers

Yifu Chen

2 papers

TL;DR

Abstract

Authors

References111 items

Understanding Continual Learning Settings with Data Distribution Drift Analysis

Half-Real Half-Fake Distillation for Class-Incremental Semantic Segmentation

DER: Dynamically Expandable Representation for Class Incremental Learning

Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations

MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks

Continuum: Simple Management of Complex Continual Learning Scenarios

ObjectAug: Object-level Data Augmentation for Semantic Image Segmentation

Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation

Self-Training for Class-Incremental Semantic Segmentation

A Few Guidelines for Incremental Few-Shot Segmentation

PLOP: Learning without Forgetting for Continual Semantic Segmentation

Knowledge Distillation for Semantic Segmentation Using Channel and Spatial Correlations and Adaptive Cross Entropy

DACS: Domain Adaptation via Cross-domain Mixed Sampling

ClassMix: Segmentation-Based Data Augmentation for Semi-Supervised Learning

ESL: Entropy-guided Self-supervised Learning for Domain Adaptation in Semantic Segmentation

Hierarchical Multi-Scale Attention for Semantic Segmentation

PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning

ResNeSt: Split-Attention Networks

Memory-Efficient Incremental Learning Through Feature Adaptation

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation

SuperMix: Supervising the Mixing Data Augmentation

On Feature Normalization and Data Augmentation

Mnemonics Training: Multi-Class Incremental Learning Without Forgetting

Modeling the Background for Incremental Learning in Semantic Segmentation

ScaIL: Classifier Weights Scaling for Class Incremental Learning

Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion

Maintaining Discrimination and Fairness in Class Incremental Learning

Compacting, Picking and Growing for Unforgetting Continual Learning

REMIND Your Neural Network to Prevent Catastrophic Forgetting

IL2M: Class Incremental Learning With Dual Memory

Object-Contextual Representations for Semantic Segmentation

InstaBoost: Boosting Instance Segmentation via Probability Map Guided Copy-Pasting

Incremental Learning Techniques for Semantic Segmentation

DisCoRL: Continual Reinforcement Learning via Policy Distillation

Rehearsal-Free Continual Learning over Small Non-I.I.D. Batches

Learning a Unified Classifier Incrementally via Rebalancing

Large Scale Incremental Learning

CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features

SEMEDA: Enhancing Segmentation Precision with Semantic Edge Aware Loss

Bidirectional Learning for Domain Adaptation of Semantic Segmentation

M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning

Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting

Continual Learning via Neural Pruning

On Tiny Episodic Memories in Continual Learning

Incremental Learning with Maximum Entropy Regularization: Rethinking Forgetting and Intransigence

ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation

CCNet: Criss-Cross Attention for Semantic Segmentation

Learning Without Memorizing

Extending pretrained segmentation networks with additional anatomical structures

Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data

Efficient Lifelong Learning with A-GEM

Dual Attention Network for Scene Segmentation

Unsupervised Domain Adaptation for Semantic Segmentation via Class-Balanced Self-training

PSANet: Point-wise Spatial Attention Network for Scene Parsing

OCNet: Object Context Network for Scene Parsing

End-to-End Incremental Learning

Manifold Mixup: Better Representations by Interpolating Hidden States

Learn the new, keep the old: Extending pretrained models with new anatomy and images

Context Encoding for Semantic Segmentation

ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation

Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence

Low-Shot Learning with Imprinted Weights

FearNet: Brain-Inspired Model for Incremental Learning

Memory Aware Synapses: Learning what (not) to forget

mixup: Beyond Empirical Risk Minimization

The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes

Measuring Catastrophic Forgetting in Neural Networks

Focal Loss for Dense Object Detection

Lifelong Learning with Dynamically Expandable Networks

Scene Parsing through ADE20K Dataset

Rethinking Atrous Convolution for Semantic Image Segmentation

Gradient Episodic Memory for Continual Learning

Continual Learning with Deep Generative Replay

Continual Learning Through Synaptic Intelligence