A Little Pretraining Goes a Long Way: A Case Study on Dependency Parsing Task for Low-resource Morphologically Rich Languages (2021-02-12T00:00:00.000000Z)

TL;DR

This work proposes simple auxiliary tasks for pretraining for dependency parsing for morphological rich languages (MRLs) in a low-resource setting to address the challenges to get morphological information for MRLs.

Abstract

Neural dependency parsing has achieved remarkable performance for many domains and languages. The bottleneck of massive labelled data limits the effectiveness of these approaches for low resource languages. In this work, we focus on dependency parsing for morphological rich languages (MRLs) in a low-resource setting. Although morphological information is essential for the dependency parsing task, the morphological disambiguation and lack of powerful analyzers pose challenges to get this information for MRLs. To address these challenges, we propose simple auxiliary tasks for pretraining. We perform experiments on 10 MRLs in low-resource settings to measure the efficacy of our proposed pretraining method and observe an average absolute gain of 2 points (UAS) and 3.6 points (LAS).

Authors

Pawan Goyal

11 papers

Amrith Krishna

3 papers

Ashim Gupta

2 papers

TL;DR

Abstract

Authors

References50 items

A Graph-Based Framework for Structured Prediction Tasks in Sanskrit

Evaluating Neural Morphological Taggers for Sanskrit

The State and Fate of Linguistic Diversity and Inclusion in the NLP World

Deep Contextualized Self-training for Low Resource Dependency Parsing

Deep Contextualized Word Embeddings in Transition-Based and Graph-Based Dependency Parsing - A Tale of Two Parsers Revisited

Pāṇinian Syntactico-Semantic Relation Labels

75 Languages, 1 Model: Parsing Universal Dependencies Universally

Joint Transition-Based Models for Morpho-Syntactic Parsing: Parsing Strategies for MRLs and a Case Study from Modern Hebrew

Parameter-Efficient Transfer Learning for NLP

A Framework for Understanding the Role of Morphology in Universal Dependency Parsing

CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies

Semi-Supervised Sequence Modeling with Cross-View Training

What do character-level models learn about morphology? The case of dependency parsing

An Improved Neural Network Model for Joint POS Tagging and Dependency Parsing

The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing

NCRF++: An Open-source Neural Sequence Labeling Toolkit

Stack-Pointer Networks for Dependency Parsing

Learning Word Vectors for 157 Languages

Deep Contextualized Word Representations

Universal Language Model Fine-tuning for Text Classification

Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm

Deep Biaffine Attention for Neural Dependency Parsing

Design and analysis of a lean interface for Sanskrit corpus annotation

Universal Dependencies v1: A Multilingual Treebank Collection

Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations

Character-level Convolutional Networks for Text Classification

Bidirectional LSTM-CRF Models for Sequence Tagging

A Graph-based Lattice Dependency Parser for Joint Morphological Segmentation and Syntactic Analysis

Transition-Based Dependency Parsing with Stack Long Short-Term Memory

Universal Dependency Annotation for Multilingual Parsing

Introduction to case, animacy and semantic roles

Designing a Constraint Based Parser for Sanskrit

On the interaction of structural and semantic case

Catastrophic forgetting in connectionist networks

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

BERT and PALs: Projected attention layers for efficient adaptation in multi-task learning

TranSeq-DL: In this setting, we use discriminative learning (DL) rate (Howard and Ruder, 2018) for pre-trained layers, i.e., decreasing the learning rate

Adversarial Training for Cross-Domain Universal Dependency Parsing

Stanford’s Graph-based Neural Dependency Parser at the CoNLL 2017 Shared Task

decreasing the learning rate as we move from top-to-bottom layers. TranSeq-FT: We fine-tune (FT), pre-trained layers with default learning rate used by

labelled data by integrating base parser with representation models

A Fast and Accurate Dependency Parser using Neural Networks

Design of a lean interface for Sanskrit corpus annotation

Dcs-the digital corpus of sanskrit. Heidelberg (2010-2020)

Case: abstract vs. morphological

Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem

Stickland and Murray, 2019) in between two consecutive pre-trained layers. TranSeq-UF: Gradually Unfreeze (UF) these three pre-trained layers in the top to down fashion (Felbo et al

Optimum branchings

Edinburgh Research Explorer A systematic comparison of methods for low-resource dependency parsing on genuinely low-resource languages

these automatically-generated trees are transformed into one or more sequence tagging schemes

Field of Study

Venue Information

Name

Type

URL

Alternate Names