Representation Compensation Networks for Continual Semantic Segmentation (2022-03-10T00:00:00.000000Z)

TL;DR

This work proposes to use a structural re-parameterization mechanism, named representation compensation (RC) module, to decouple the representation learning of both old and new knowledge, and outperforms state-of-the-art performance.

Abstract

In this work, we study the continual semantic segmentation problem, where the deep neural networks are required to incorporate new classes continually without catastrophic forgetting. We propose to use a structural re-parameterization mechanism, named representation compensation (RC) module, to decouple the representation learning of both old and new knowledge. The RC module consists of two dynamically evolved branches with one frozen and one trainable. Besides, we design a pooled cube knowledge distillation strategy on both spatial and channel dimensions to further enhance the plasticity and stability of the model. We conduct experiments on two challenging continual semantic segmentation scenarios, continual class segmentation and continual domain segmentation. Without any extra computational overhead and parameters during inference, our method outperforms state-of-the-art performance. The code is available at https://github.com/zhangchbin/RCIL.

Authors

Mingg-Ming Cheng

6 papers

Xialei Liu

5 papers

Chang-Bin Zhang

1 papers

TL;DR

Abstract

Authors

References113 items

Continual Learning on Noisy Data Streams via Self-Purified Replay

Striking a Balance between Stability and Plasticity for Class-Incremental Learning

RECALL: Replay-based Continual Learning in Semantic Segmentation

An EM Framework for Online Incremental Learning of Semantic Segmentation

Co2L: Contrastive Continual Learning

Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning

Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning

Prototype Augmentation and Self-Supervision for Incremental Learning

SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers

Segmenter: Transformer for Semantic Segmentation

On Learning the Geodesic Path for Incremental Learning

Rehearsal revealed: The limits and merits of revisiting samples in continual learning

Few-Shot Incremental Learning with Continually Evolved Classifiers

Half-Real Half-Fake Distillation for Class-Incremental Semantic Segmentation

Rainbow Memory: Continual Learning with a Memory of Diverse Samples

DER: Dynamically Expandable Representation for Class Incremental Learning

Rectification-based Knowledge Retention for Continual Learning

CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification

Efficient Feature Transformations for Discriminative and Generative Continual Learning

Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations

Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning

Learning Statistical Texture for Semantic Segmentation

Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

Is Context-Aware CNN Ready for the Surroundings? Panoramic Semantic Segmentation in the Wild

RepVGG: Making VGG-style ConvNets Great Again

Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation

End-to-End Video Instance Segmentation with Transformers

Delving Deep Into Label Smoothing

PLOP: Learning without Forgetting for Continual Semantic Segmentation

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Adaptive Aggregation Networks for Class-Incremental Learning

Deformable DETR: Deformable Transformers for End-to-End Object Detection

Unsupervised Model Adaptation for Continual Semantic Segmentation

Imbalanced Continual Learning with Partitioning Reservoir Sampling

Online Class-Incremental Continual Learning with Adversarial Shapley Value

Topology-Preserving Class-Incremental Learning

Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference

Learning Joint Spatial-Temporal Transformations for Video Inpainting

Feature Pyramid Transformer

Interactive Knowledge Distillation

End-to-End Object Detection with Transformers

Continual Deep Learning by Functional Regularisation of Memorable Past

PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning

Dark Experience for General Continual Learning: a Strong, Simple Baseline

Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentation

Semantic Drift Compensation for Class-Incremental Learning

Memory-Efficient Incremental Learning Through Feature Adaptation

SS-IL: Separated Softmax for Incremental Learning

Conditional Channel Gated Networks for Task-Aware Continual Learning

Strip Pooling: Rethinking Spatial Pooling for Scene Parsing

Continual Learning with Node-Importance based Adaptive Group Sparse Regularization

Adversarial Continual Learning

Incremental Meta-Learning via Indirect Discriminant Alignment

BERT-of-Theseus: Compressing BERT by Progressive Module Replacing

Modeling the Background for Incremental Learning in Semantic Segmentation

Semantic Segmentation With Context Encoding and Multi-Path Decoding

Maintaining Discrimination and Fairness in Class Incremental Learning

REMIND Your Neural Network to Prevent Catastrophic Forgetting

Incremental Learning Using Conditional Adversarial Networks

IL2M: Class Incremental Learning With Dual Memory

Using Hindsight to Anchor Past Knowledge in Continual Learning

ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks

Continual Learning by Asymmetric Loss Approximation With Single-Side Overestimation

Expectation-Maximization Attention Networks for Semantic Segmentation

Incremental Learning Techniques for Semantic Segmentation

Learning a Unified Classifier Incrementally via Rebalancing

Large Scale Incremental Learning

Decoders Matter for Semantic Segmentation: Data-Dependent Decoding Enables Flexible Feature Aggregation

CCNet: Criss-Cross Attention for Semantic Segmentation

Learning Without Memorizing

Dual Attention Network for Scene Segmentation

Multi-scale Context Intertwining for Semantic Segmentation

DenseASPP for Semantic Segmentation in Street Scenes

Context Contrasted Feature and Gated Multi-scale Aggregation for Scene Segmentation

ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation