BigNLI: Native Language Identification with Big Bird Embeddings
TL
TL;DR
This work shows input size is a limiting factor, and that classifiers trained using Big Bird embeddings outperform linguistic feature engineering models by a large margin on the Reddit-L2 dataset, and offers further insight into input length dependencies.
Abstract
Native Language Identification (NLI) intends to classify an author’s native language based on their writing in another language. Historically, the task has heavily relied on time-consuming linguistic feature engineering, and NLI transformer models have thus far failed to offer effective, practical alternatives. The current work shows input size is a limiting factor, and that classifiers trained using Big Bird embeddings outperform linguistic feature engineering models (for which we reproduce previous work) by a large margin on the Reddit-L2 dataset. Additionally, we provide further insight into input length dependencies, show consistent out-of-sample (Europe subreddit) and out-of-domain (TOEFL-11) performance, and qualitatively analyze the embedding space. Given the effectiveness and computational efficiency of this method, we believe it offers a promising avenue for future NLI work.
References39 items
1
Scaling Transformer to 1M tokens and beyond with RMT
2
A Deep Generative Approach to Native Language Identification
3
Big Bird: Transformers for Longer Sequences
4
Language Models are Few-Shot Learners
5