Texts

Demetr

Introduced in DEMETR: Diagnosing Evaluation Metrics for Translation2022

About this Dataset

Demetr is a diagnostic dataset with 31K English examples (translated from 10 source languages) for evaluating the sensitivity of MT evaluation metrics to 35 different linguistic perturbations spanning semantic, syntactic, and morphological error categories.

Source: DEMETR: Diagnosing Evaluation Metrics for Translation

Image Source: https://arxiv.org/pdf/2210.13746v1.pdf

Source: DEMETR: Diagnosing Evaluation Metrics for Translation

Dataset Variants

Demetr

Papers1

DEMETR: Diagnosing Evaluation Metrics for Translation

DEMETR is a diagnostic dataset with 31K English examples for evaluating the sensitivity of MT evaluation metrics to 35 different linguistic perturbations spanning semantic, syntactic, and morphological error categories and it is found that learned metrics perform substantially better than string-based metrics on DEMETR.

Tasks

EDIT

Machine Translation Translation

Similar Datasets