Unsupervised Multi-Target Domain Adaptation Through Knowledge Distillation (2020-07-14T00:00:00.000000Z)

TL;DR

This paper proposes a novel unsupervised MTDA approach to train a CNN that can generalize well across multiple target domains and relies on multi-teacher knowledge distillation (KD) to iteratively distill target domain knowledge from multiple teachers to a common student.

Abstract

Unsupervised domain adaptation (UDA) seeks to alleviate the problem of domain shift between the distribution of unlabeled data from the target domain w.r.t. labeled data from the source domain. While the single-target UDA scenario is well studied in the literature, Multi-Target Domain Adaptation (MTDA) remains largely unexplored despite its practical importance, e.g., in multi-camera video-surveillance applications. The MTDA problem can be addressed by adapting one specialized model per target domain, although this solution is too costly in many real-world applications. Blending multiple targets for MTDA has been proposed, yet this solution may lead to a reduction in model specificity and accuracy. In this paper, we propose a novel unsupervised MTDA approach to train a CNN that can generalize well across multiple target domains. Our Multi-Teacher MTDA (MT-MTDA) method relies on multi-teacher knowledge distillation (KD) to iteratively distill target domain knowledge from multiple teachers to a common student. The KD process is performed in a progressive manner, where the student is trained by each teacher on how to perform UDA for a specific target, instead of directly learning domain adapted features. Finally, instead of combining the knowledge from each teacher, MT-MTDA alternates between teachers that distill knowledge, thereby preserving the specificity of each target (teacher) when learning to adapt to the student. MT-MTDA is compared against state- of-the-art methods on several challenging UDA benchmarks, and empirical results show that our proposed model can provide a considerably higher level of accuracy across multiple target domains. Our code is available at: https://gi.thub.com/LIVIAETS/MT-MTDA.

TL;DR

Abstract

Authors

References30 items

Unsupervised Domain Adaptation in the Dissimilarity Space for Person Re-identification

Joint Progressive Knowledge Distillation and Unsupervised Domain Adaptation

Dual-Triplet Metric Learning for Unsupervised Domain Adaptation in Video-Based Face Recognition

Adversarial-Learned Loss for Domain Adaptation

Adversarial Domain Adaptation with Domain Mixup

Open Compound Domain Adaptation

Blending-Target Domain Adaptation by Adversarial Meta-Adaptation Networks

Unsupervised Domain Adaptation to Improve Image Segmentation Quality Both in the Source and Target Domain

Domain Agnostic Learning with Disentangled Representations

A Comprehensive Overhaul of Feature Distillation

Unsupervised Multi-Target Domain Adaptation: An Information Theoretic Approach

Geometry-Consistent Generative Adversarial Networks for One-Sided Unsupervised Domain Mapping

Multi-target Unsupervised Domain Adaptation without Exactly Shared Categories

Conditional Generative Adversarial Network for Structured Domain Adaptation

Deep Unsupervised Domain Adaptation for Face Recognition

Improving face recognition with domain adaptation

Domain Adaptive Faster R-CNN for Object Detection in the Wild

Deep Visual Domain Adaptation: A Survey

A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning

Conditional Adversarial Domain Adaptation

Learning from Noisy Labels with Distillation

Adversarial Discriminative Domain Adaptation

Knowledge Adaptation: Teaching to Adapt

Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks

Revisiting Batch Normalization For Practical Domain Adaptation

Distilling the Knowledge in a Neural Network

Learning Transferable Features with Deep Adaptation Networks

FitNets: Hints for Thin Deep Nets

Unsupervised Domain Adaptation by Backpropagation

Deep Learning

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names