Linguistically-Informed Self-Attention for Semantic Role Labeling (2018-04-23T00:00:00.000000Z)

TL;DR

LISA is a neural network model that combines multi-head self-attention with multi-task learning across dependency parsing, part-of-speech tagging, predicate detection and SRL, and can incorporate syntax using merely raw tokens as input.

Abstract

Current state-of-the-art semantic role labeling (SRL) uses a deep neural network with no explicit linguistic features. However, prior work has shown that gold syntax trees can dramatically improve SRL decoding, suggesting the possibility of increased accuracy from explicit modeling of syntax. In this work, we present linguistically-informed self-attention (LISA): a neural network model that combines multi-head self-attention with multi-task learning across dependency parsing, part-of-speech tagging, predicate detection and SRL. Unlike previous models which require significant pre-processing to prepare linguistic features, LISA can incorporate syntax using merely raw tokens as input, encoding the sequence only once to simultaneously perform parsing, predicate detection and role labeling for all predicates. Syntax is incorporated by training one attention head to attend to syntactic parents for each token. Moreover, if a high-quality syntactic parse is already available, it can be beneficially injected at test time without re-training our SRL model. In experiments on CoNLL-2005 SRL, LISA achieves new state-of-the-art performance for a model using predicted predicates and standard word embeddings, attaining 2.5 F1 absolute higher than the previous state-of-the-art on newswire and more than 3.5 F1 on out-of-domain data, nearly 10% reduction in error. On ConLL-2012 English SRL we also show an improvement of more than 2.5 F1. LISA also out-performs the state-of-the-art with contextually-encoded (ELMo) word representations, by nearly 1.0 F1 on news and more than 2.0 F1 on out-of-domain text.

Authors

Emma Strubell

6 papers

A. McCallum

17 papers

D. Andor

5 papers

TL;DR

Abstract

Authors

References78 items

Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling

Deep Contextualized Word Representations

Deep Semantic Role Labeling with Self-Attention

End-to-end Neural Coreference Resolution

Deep Semantic Role Labeling: What Works and What’s Next

Frame-Semantic Parsing with Softmax-Margin Segmental RNNs and a Syntactic Scaffold

Attention is All you Need

Learning Structured Text Representations

Deep Multitask Learning for Semantic Dependency Parsing

Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling

Identifying beneficial task relations for multi-task learning in deep neural networks

A Simple and Accurate Syntax-Agnostic Neural Model for Dependency-based Semantic Role Labeling

When is multitask learning effective? Semantic sequence prediction under varying data conditions

A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks

Deep Biaffine Attention for Neural Dependency Parsing

Deep multi-task learning with low level tasks supervised at lower layers

Layer Normalization

Neural Semantic Role Labeling with Dependency Path Embeddings

Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations

Training with Exploration Improves a Greedy Stack LSTM Parser

Stack-propagation: Improved Representation Learning for Syntax

Incorporating Nesterov Momentum into Adam

Deep Residual Learning for Image Recognition

Multi-task Sequence to Sequence Learning

Joint A* CCG Parsing and Semantic Role Labelling

Semantic Role Labeling with Neural Network Factors

Machine Comprehension with Syntax, Frames, and Semantics

End-to-end learning of semantic role labeling using recurrent neural networks

It Depends: Dependency Parser Comparison Using A Web-based Evaluation Tool

Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Learning to Search Better than Your Teacher

Efficient Inference and Structured Learning for Semantic Role Labeling

Adam: A Method for Stochastic Optimization

Modeling Biological Processes for Reading Comprehension

GloVe: Global Vectors for Word Representation

FrameNet: A Knowledge Base for Natural Language Processing

Unsupervised induction and filling of semantic slots for spoken dialogue systems using frame-semantic parsing

Towards Robust Linguistic Analysis using OntoNotes

Semantic Roles for String to Tree Machine Translation

A Dynamic Oracle for Arc-Eager Dependency Parsing

On the difficulty of training recurrent neural networks

Fast and Robust Part-of-Speech Tagging Using Dynamic Model Selection

Getting the Most out of Transition-based Dependency Parsing

Natural Language Processing (Almost) from Scratch

A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

Semantic Role Features for Machine Translation

Search-based structured prediction

Dependency-based Semantic Role Labeling of PropBank

The Stanford Typed Dependencies Representation

The Importance of Syntactic Parsing and Inference in Semantic Role Labeling

A Global Joint Model for Semantic Role Labeling

Combination Strategies for Semantic Role Labeling

OntoNotes: The 90% Solution

Introduction to the CoNLL-2005 Shared Task: Semantic Role Labeling

Joint Parsing and Semantic Role Labeling

Semantic Role Labeling Using Different Syntactic Views

The Proposition Bank: An Annotated Corpus of Semantic Roles

Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network

Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions

Learning long-term dependencies with gradient descent is difficult

English Verb Classes and Alternations: A Preliminary Investigation

Multitask Learning: A Knowledge-Based Source of Inductive Bias

Building a Large Annotated Corpus of English: The Penn Treebank

A Learning Algorithm for Continually Running Fully Recurrent Neural Networks

CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies

Semisu-pervised classiﬁcation with graph convolutional networks

CONDITIONAL RANDOM FIELDS

Tensorﬂow: Large-scale machine learning on heterogeneous systems, 2015

Dropout: a simple way to prevent neural networks from overfitting

Rectifier Nonlinearities Improve Neural Network Acoustic Models

2018) to evaluate our models, which uses the annotations from here9 but the subset of those documents from the CoNLL-2012 co-reference split described here10

Guidelines for the clear style constituent to dependency conversion

Very high accuracy and fast dependency parsing is not a contradiction

Semi-supervised learning for spoken language understanding semantic role labeling