UPB at SemEval-2020 Task 6: Pretrained Language Models for Definition Extraction (2020-09-11T00:00:00.000000Z)

TL;DR

This work uses various pretrained language models (i.e., BERT, XLNet, RoBERTa, SciBERT, and ALBERT) to solve each of the three subtasks of the DeftEval competition, and explores a multi-task architecture that was trained to jointly predict the outputs for the second and the third subtasks.

Abstract

This work presents our contribution in the context of the 6th task of SemEval-2020: Extracting Definitions from Free Text in Textbooks (DeftEval). This competition consists of three subtasks with different levels of granularity: (1) classification of sentences as definitional or non-definitional, (2) labeling of definitional sentences, and (3) relation classification. We use various pretrained language models (i.e., BERT, XLNet, RoBERTa, SciBERT, and ALBERT) to solve each of the three subtasks of the competition. Specifically, for each language model variant, we experiment by both freezing its weights and fine-tuning them. We also explore a multi-task architecture that was trained to jointly predict the outputs for the second and the third subtasks. Our best performing model evaluated on the DeftEval dataset obtains the 32nd place for the first subtask and the 37th place for the second subtask. The code is available for further research at: https://github.com/avramandrei/DeftEval

Authors

Andrei-Marius Avram

3 papers

Dumitru-Clementin Cercel

1 papers

Costin-Gabriel Chiru

1 papers

TL;DR

Abstract

Authors

References33 items

A Joint Model for Definition Extraction with Syntactic Connection and Semantic Consistency

Findings of the NLP4IF-2019 Shared Task on Fine-Grained Propaganda Detection

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Introducing RONEC - the Romanian Named Entity Corpus

DEFT: A corpus for definition extraction in free- and semi-structured text

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Learning Emphasis Selection for Written Text in Visual Media from Crowd-Sourced Label Distributions

Complex Word Identification as a Sequence Labelling Task

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Bi-LSTM-CRF Sequence Labeling for Keyphrase Extraction from Scholarly Documents

To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks

SciBERT: A Pretrained Language Model for Scientific Text

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

A Character Prediction Approach in a Security Context using a Recurrent Neural Network

Language Identification and Analysis of Code-Switched Social Media Text

Neural Metaphor Detecting with CNN-LSTM Model

Syntactically Aware Neural Architectures for Definition Extraction

Large Scaled Relation Extraction With Reinforcement Learning

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Empower Sequence Labeling with Task-Aware Neural Language Model

Attention is All you Need

End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

Adam: A Method for Stochastic Optimization

Learning Word-Class Lattices for Definition and Hypernym Extraction

Automatic extraction of definitions

Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

Long Short-Term Memory

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Viterbi Algorithm

Learning to Identify Definitions using Syntactic Features

Evaluation of the DEFINDER system for fully automatic glossary construction

Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Visual Pattern Recognition

Field of Study

Venue Information

Name

Type

URL

Alternate Names