BigGreen at SemEval-2021 Task 1: Lexical Complexity Prediction with Assembly Models (2021-04-19T00:00:00.000000Z)

TL;DR

This paper assembles a feature engineering-based model with a deep neural network model founded on BERT to predict the lexical complexity of English words in a given context and demonstrates how they can be harnessed to perform well on the multi word expression subtask.

Abstract

This paper describes a system submitted by team BigGreen to LCP 2021 for predicting the lexical complexity of English words in a given context. We assemble a feature engineering-based model with a deep neural network model founded on BERT. While BERT itself performs competitively, our feature engineering-based model helps in extreme cases, eg. separating instances of easy and neutral difficulty. Our handcrafted features comprise a breadth of lexical, semantic, syntactic, and novel phonological measures. Visualizations of BERT attention maps offer insight into potential features that Transformers models may learn when fine-tuned for lexical complexity prediction. Our ensembled predictions score reasonably well for the single word subtask, and we demonstrate how they can be harnessed to perform well on the multi word expression subtask too.

Authors

Soroush Vosoughi

6 papers

Weicheng Ma

2 papers

A. Islam

1 papers

TL;DR

Abstract

Authors

References41 items

SemEval-2021 Task 1: Lexical Complexity Prediction

CompLex — A New Corpus for Lexical Complexity Prediction from Likert Scale Data

Variable Selection

What Does BERT Look at? An Analysis of BERT’s Attention

Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned

Multi-Task Deep Neural Networks for Natural Language Understanding

NILC at CWI 2018: Exploring Feature Engineering and Feature Learning

A Report on the Complex Word Identification Shared Task 2018

Deep Contextualized Word Representations

SMOGN: a Pre-processing Approach for Imbalanced Regression

Complex Word Identification: Challenges in Data Annotation and System Performance

Attention is All you Need

Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

SV000gg at SemEval-2016 Task 11: Heavy Gauge Complex Word Identification with System Voting

SemEval 2016 Task 11: Complex Word Identification

TALN at SemEval-2016 Task 11: Modelling Complex Words by Contextual, Lexical and Semantic Features

XGBoost: A Scalable Tree Boosting System

GloVe: Global Vectors for Word Representation

The Stanford CoreNLP Natural Language Processing Toolkit

Out in the Open: Finding and Categorising Errors in the Lexical Simplification Pipeline

A Comparison of Techniques to Automatically Identify Complex Words.

Scikit-learn: Machine Learning in Python

Steven Bird, Ewan Klein and Edward Loper: Natural Language Processing with Python, Analyzing Text with the Natural Language Toolkit

Moving beyond Kučera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English

A Text Corpora-Based Estimation of the Familiarity of Health Terminology

The British national corpus

Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from an ice cream cone

Lexical complexity and fixation times in reading: Effects of word frequency, verb complexity, and lexical ambiguity

PhraseFinder – search millions of books for language use

K-nearest neighbors algorithm — Wikipedia, the free encyclopedia

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Deep Learning Architecture for ComplexWord Identification. In Thirteenth Workshop of Innovative Use of NLP for Building Educational Applications, pages 328–334

SemEval-2016 Task 11 : Identifying Complex Words in a Sentence

English Gigaword Fifth Edition LDC2011T07 (tech

WordNet

Moving beyond Ku era and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English

Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods

The use of a psycholinguistic database in the simplification of text for aphasic readers

Regression Shrinkage and Selection via the Lasso

SMOG Grading - A New Readability Formula.

A formula for predicting readability: Instructions

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names