Simplified TinyBERT: Knowledge Distillation for Document Retrieval
TL
TL;DR
Evaluation on MS MARCO document re-ranking task confirms the effectiveness of the proposed simplifications of TinyBERT, and investigates the applications of knowledge distillation models on document ranking task.
Abstract
Despite the effectiveness of utilizing BERT for document ranking, the computational cost of such approaches is non-negligible when compared to other retrieval methods. To this end, this paper first empirically investigates the applications of knowledge distillation models on document ranking task. In addition, on top of the recent TinyBERT, two simplifications are proposed. Evaluation on MS MARCO document re-ranking task confirms the effectiveness of the proposed simplifications.
References21 items
1
Understanding BERT Rankers Under Distillation
2
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
3
Overview of the TREC 2019 deep learning track
4
Document Ranking with a Pretrained Sequence-to-Sequence Model
5