SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis

Published in

Annual Meeting of the Association for Computati...(2020)

External Links:

Generate Graph DownloadPDF

TL;DR

Sentiment Knowledge Enhanced Pre-training (SKEP) is introduced in order to learn a unified sentiment representation for multiple sentiment analysis tasks, and significantly outperforms strong pre-training baseline, and achieves new state-of-the-art results on most of the test datasets.

Abstract

Recently, sentiment analysis has seen remarkable advance with the help of pre-training approaches. However, sentiment knowledge, such as sentiment words and aspect-sentiment pairs, is ignored in the process of pre-training, despite the fact that they are widely used in traditional sentiment analysis approaches. In this paper, we introduce Sentiment Knowledge Enhanced Pre-training (SKEP) in order to learn a unified sentiment representation for multiple sentiment analysis tasks. With the help of automatically-mined knowledge, SKEP conducts sentiment masking and constructs three sentiment knowledge prediction objectives, so as to embed sentiment information at the word, polarity and aspect level into pre-trained sentiment representation. In particular, the prediction of aspect-sentiment pairs is converted into multi-label classification, aiming to capture the dependency between words in a pair. Experiments on three kinds of sentiment tasks show that SKEP significantly outperforms strong pre-training baseline, and achieves new state-of-the-art results on most of the test datasets. We release our code at https://github.com/baidu/Senta.

Authors

Bolei He

2 papers

Xinyan Xiao

4 papers

Hua Wu

12 papers

References44 items

A Knowledge Regularized Hierarchical Approach for Emotion Cause Analysis

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification

SenseBERT: Driving Some Sense into BERT

SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis

Published in

Annual Meeting of the Association for Computati...(2020)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Bolei He

2 papers

Xinyan Xiao

4 papers

Hua Wu

12 papers

References44 items

A Knowledge Regularized Hierarchical Approach for Emotion Cause Analysis

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification

SenseBERT: Driving Some Sense into BERT

Haifeng Wang

11 papers

GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge

What Does BERT Learn about the Structure of Language?

RoBERTa: A Robustly Optimized BERT Pretraining Approach

SpanBERT: Improving Pre-training by Representing and Predicting Spans

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Modeling Sentiment Dependencies with Graph Convolutional Networks for Aspect-level Sentiment Classification

ERNIE: Enhanced Language Representation with Informative Entities

Unsupervised Data Augmentation

ERNIE: Enhanced Representation through Knowledge Integration

A Variational Approach to Weakly Supervised Document-Level Multi-Aspect Sentiment Classification

A Multi-sentiment-resource Enhanced Attention Network for Sentiment Classification

SRL4ORL: Improving Opinion Role Labeling Using Multi-Task Learning with Semantic Role Labeling

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Deep Contextualized Word Representations

Deep learning for sentiment analysis: A survey

Deep Multi-Task Learning for Aspect Term Extraction with Memory Interaction

A Question Answering Approach for Emotion Cause Extraction

Attention is All you Need

Recurrent Neural Networks with Auxiliary Labels for Cross-Domain Opinion Target Extraction

Linguistically Regularized LSTM for Sentiment Classification

Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks

Lexicon Integrated CNN Models with Attention for Sentiment Analysis

Character-level Convolutional Networks for Text Classification

Target-Dependent Twitter Sentiment Classification with Rich Automatic Features

SemEval-2014 Task 4: Aspect Based Sentiment Analysis

Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Sentiment Analysis and Opinion Mining

Sentiment Analysis of Twitter Data

Lexicon-Based Methods for Sentiment Analysis

Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Lexicon information in neural sentiment analysis: a multi-task learning approach

Improving language understanding with unsupervised learning

Opinion Mining and Sentiment Analysis

sentiment seed words as follows. We ﬁrst count the 9,750 items of Qian et al.

Fine-grained subjectivity and sentiment analysis: recognizing the intensity, polarity, and attitudes of private states

Annotating Expressions of Opinions and Emotions in Language

fun, sure, pretty, negativeword too, little, bad, game long, hard, waste, problem, try, poor

Field of Study

Computer Science

Venue Information

Name

Annual Meeting of the Association for Computational Linguistics

Type

conference

URL

https://www.aclweb.org/anthology/venues/acl/

Alternate Names

Annu Meet Assoc Comput Linguistics
Meeting of the Association for Computational Linguistics
ACL
Meet Assoc Comput Linguistics