KNOT: Knowledge Distillation Using Optimal Transport for Solving NLP Tasks (2021-10-06T00:00:00.000000Z)

TL;DR

A new metric, Semantic Distance (SD), is introduced, that measures semantic closeness between the predicted and ground truth label distributions, and shows improvements in the global model’s SD performance over the baseline across three NLP tasks while performing on par with Entropy-based distillation on standard accuracy and F1 metrics.

Abstract

We propose a new approach, Knowledge Distillation using Optimal Transport (KNOT), to distill the natural language semantic knowledge from multiple teacher networks to a student network. KNOT aims to train a (global) student model by learning to minimize the optimal transport cost of its assigned probability distribution over the labels to the weighted sum of probabilities predicted by the (local) teacher models, under the constraints that the student model does not have access to teacher models’ parameters or training data. To evaluate the quality of knowledge transfer, we introduce a new metric, Semantic Distance (SD), that measures semantic closeness between the predicted and ground truth label distributions. The proposed method shows improvements in the global model’s SD performance over the baseline across three NLP tasks while performing on par with Entropy-based distillation on standard accuracy and F1 metrics. The implementation pertaining to this work is publicly available at https://github.com/declare-lab/KNOT.

Authors

Soujanya Poria

37 papers

Rishabh Bhardwaj

5 papers

Tushar Vaidya

1 papers

TL;DR

Abstract

Authors

References47 items

CICERO: A Dataset for Contextualized Commonsense Inference in Dialogues

Scaling Federated Learning for Fine-tuning of Large Language Models

Ensemble Distillation for Robust Model Fusion in Federated Learning

Knowledge Distillation: A Survey

Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research

Distributed Gradient Methods for Convex Machine Learning Problems in Networks: Distributed Optimization

Federated pretraining and fine tuning of BERT using clinical notes from multiple silos

Adversarial NLI: A New Benchmark for Natural Language Understanding

Contrastive Representation Distillation

Conversational transfer learning for emotion recognition

FEED: Feature-level Ensemble for Knowledge Distillation

A Survey on Bias and Fairness in Machine Learning

Fair Resource Allocation in Federated Learning

Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances

Knowledge Flow: Improve Upon Your Teachers

Agnostic Federated Learning

Federated Optimization in Heterogeneous Networks

Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data

Interpolating between Optimal Transport and MMD using Sinkhorn Divergences

MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations

Don't Use Large Mini-Batches, Use Local SGD

Differential Properties of Sinkhorn Approximation for Learning with Wasserstein Distance

Algorithms and Theory for Multiple-Source Adaptation

Born Again Neural Networks

DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset

Learning from Multiple Teacher Networks

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Federated Optimization: Distributed Machine Learning for On-Device Intelligence

Communication-Efficient Learning of Deep Networks from Decentralized Data

Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering

A large annotated corpus for learning natural language inference

Learning with a Wasserstein Loss

Distilling the Knowledge in a Neural Network

FitNets: Hints for Thin Deep Nets

The Monge-Kantorovich problem: achievements, connections, and perspectives

IEMOCAP: interactive emotional dyadic motion capture database

Modeling Semantic Containment and Exclusion in Natural Language Inference

Model compression

Rademacher and Gaussian Complexities: Risk Bounds and Structural Results

The effect of class distribution on classifier learning: an empirical study

Probability in Banach Spaces: Isoperimetry and Processes

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Computational optimal transport: With applications to data science

(Frogner et al., 2015)) Let the transport cost matrix be C and the constant CM = max

Machine learning - a probabilistic perspective

Asymmetrical Facial Expressions based on an Advanced Interpretation of Two-dimensional Russells Emotional Model

Overfitting in Neural Nets: Backpropagation, Conjugate Gradient, and Early Stopping

Field of Study

Venue Information

Name

Type

URL

Alternate Names