PLOP: Learning without Forgetting for Continual Semantic Segmentation (2020-11-23T00:00:00.000000Z)

TL;DR

Local POD is proposed, a multi-scale pooling distillation scheme that preserves long- and short-range spatial relationships at feature level that significantly outperforms state-of-the-art methods in existing CSS scenarios, as well as in newly proposed challenging benchmarks 1.

Abstract

Deep learning approaches are nowadays ubiquitously used to tackle computer vision tasks such as semantic segmentation, requiring large datasets and substantial computational power. Continual learning for semantic segmentation (CSS) is an emerging trend that consists in updating an old model by sequentially adding new classes. However, continual learning methods are usually prone to catastrophic forgetting. This issue is further aggravated in CSS where, at each step, old classes from previous iterations are collapsed into the background. In this paper, we propose Local POD, a multi-scale pooling distillation scheme that preserves long- and short-range spatial relationships at feature level. Furthermore, we design an entropy-based pseudo-labelling of the background w.r.t. classes predicted by the old model to deal with background shift and avoid catastrophic forgetting of the old classes. Our approach, called PLOP, significantly outperforms state-of-the-art methods in existing CSS scenarios, as well as in newly proposed challenging benchmarks 1.

Authors

M. Cord

10 papers

Arthur Douillard

3 papers

Yifu Chen

2 papers

TL;DR

Abstract

Authors

References85 items

Knowledge Distillation for Semantic Segmentation Using Channel and Spatial Correlations and Adaptive Cross Entropy

Bookworm continual learning: beyond zero-shot learning and continual learning

Insights from the Future for Continual Learning

ESL: Entropy-guided Self-supervised Learning for Domain Adaptation in Semantic Segmentation

Hierarchical Multi-Scale Attention for Semantic Segmentation

PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning

ResNeSt: Split-Attention Networks

Memory-Efficient Incremental Learning Through Feature Adaptation

Strip Pooling: Rethinking Spatial Pooling for Scene Parsing

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation

Mnemonics Training: Multi-Class Incremental Learning Without Forgetting

Modeling the Background for Incremental Learning in Semantic Segmentation

ScaIL: Classifier Weights Scaling for Class Incremental Learning

Maintaining Discrimination and Fairness in Class Incremental Learning

Compacting, Picking and Growing for Unforgetting Continual Learning

REMIND Your Neural Network to Prevent Catastrophic Forgetting

Zero-Shot Semantic Segmentation via Variational Mapping

IL2M: Class Incremental Learning With Dual Memory

Object-Contextual Representations for Semantic Segmentation

Incremental Learning Techniques for Semantic Segmentation

Rehearsal-Free Continual Learning over Small Non-I.I.D. Batches

Zero-Shot Semantic Segmentation

Learning a Unified Classifier Incrementally via Rebalancing

Large Scale Incremental Learning

Bidirectional Learning for Domain Adaptation of Semantic Segmentation

M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning

Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting

Continual Learning via Neural Pruning

On Tiny Episodic Memories in Continual Learning

Incremental Learning with Maximum Entropy Regularization: Rethinking Forgetting and Intransigence

ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation

CCNet: Criss-Cross Attention for Semantic Segmentation

Learning Without Memorizing

Extending pretrained segmentation networks with additional anatomical structures

Efficient Lifelong Learning with A-GEM

Dual Attention Network for Scene Segmentation

Unsupervised Domain Adaptation for Semantic Segmentation via Class-Balanced Self-training

PSANet: Point-wise Spatial Attention Network for Scene Parsing

OCNet: Object Context Network for Scene Parsing

End-to-End Incremental Learning

Learn the new, keep the old: Extending pretrained models with new anatomy and images

Context Encoding for Semantic Segmentation

ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation

Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence

Generalized Zero-Shot Learning via Synthesized Examples

FearNet: Brain-Inspired Model for Incremental Learning

Memory Aware Synapses: Learning what (not) to forget

Automatic differentiation in PyTorch

Measuring Catastrophic Forgetting in Neural Networks

Lifelong Learning with Dynamically Expandable Networks

Scene Parsing through ADE20K Dataset

Rethinking Atrous Convolution for Semantic Image Segmentation

Gradient Episodic Memory for Continual Learning

CORe50: a New Dataset and Benchmark for Continuous Object Recognition

Continual Learning with Deep Generative Replay

Continual Learning Through Synaptic Intelligence

PathNet: Evolution Channels Gradient Descent in Super Neural Networks

COCO-Stuff: Thing and Stuff Classes in Context

Pyramid Scene Parsing Network

Overcoming catastrophic forgetting in neural networks

iCaRL: Incremental Classifier and Representation Learning

Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer

Learning without Forgetting

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

The Cityscapes Dataset for Semantic Urban Scene Understanding

Deep Residual Learning for Image Recognition

SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation

U-Net: Convolutional Networks for Biomedical Image Segmentation

Learning Deconvolution Network for Semantic Segmentation

Distilling the Knowledge in a Neural Network

FitNets: Hints for Thin Deep Nets

Fully convolutional networks for semantic segmentation

The Pascal Visual Object Classes Challenge: A Retrospective

Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition