Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction
TL
TL;DR
ERRANT, a grammatical ERRor ANnotation Toolkit designed to automatically extract edits from parallel original and corrected sentences and classify them according to a new, dataset-agnostic, rule-based framework, which facilitates error type evaluation at different levels of granularity.
Abstract
Until now, error type performance for Grammatical Error Correction (GEC) systems could only be measured in terms of recall because system output is not annotated. To overcome this problem, we introduce ERRANT, a grammatical ERRor ANnotation Toolkit designed to automatically extract edits from parallel original and corrected sentences and classify them according to a new, dataset-agnostic, rule-based framework. This not only facilitates error type evaluation at different levels of granularity, but can also be used to reduce annotator workload and standardise existing GEC datasets. Human experts rated the automatic edits as “Good” or “Acceptable” in at least 95% of cases, so we applied ERRANT to the system output of the CoNLL-2014 shared task to carry out a detailed error type analysis for the first time.
References23 items
1
Automatic Extraction of Learner Errors in ESL Sentences Using Linguistically Enhanced Alignments
2
Compositional Sequence Labeling Models for Error Detection in Learner Writing
3
Reassessing the Goals of Grammatical Error Correction: Fluency Instead of Grammaticality
4
Human Evaluation of Grammatical Error Correction Systems
5