Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation (2019-01-02T00:00:00.000000Z)

TL;DR

A large-scale, systematic study to evaluate the existing evaluation methods for natural language generation in the context of generating online product reviews finds lexical diversity an intriguing metric that is indicative of the assessments of different evaluators.

Abstract

We conduct a large-scale, systematic study to evaluate the existing evaluation methods for natural language generation in the context of generating online product reviews. We compare human-based evaluators with a variety of automated evaluation procedures, including discriminative evaluators that measure how well machine-generated text can be distinguished from human-written text, as well as word overlap metrics that assess how similar the generated text compares to human-written references. We determine to what extent these different evaluators agree on the ranking of a dozen of state-of-the-art generators for online product reviews. We find that human evaluators do not correlate well with discriminative evaluators, leaving a bigger question of whether adversarial accuracy is the correct objective for natural language generation. In general, distinguishing machine-generated text is challenging even for human evaluators, and human decisions correlate better with lexical overlaps. We find lexical diversity an intriguing metric that is indicative of the assessments of different evaluators. A post-experiment survey of participants provides insights into how to evaluate and improve the quality of natural language generation systems.

Authors

Cristina Garbacea

2 papers

Samuel Carton

1 papers

Shiyan Yan

1 papers

TL;DR

Abstract

Authors

References134 items

Intelligent sentence completion based on global context dependent recurrent neural network language model

Jointly Measuring Diversity and Quality in Text Generation Models

Unifying Human and Statistical Evaluation for Natural Language Generation

Language GANs Falling Short

Evaluating Text GANs as Language Models

Language Modeling for Morphologically Rich Languages: Character-Aware Modeling for Word-Level Prediction

On Accurate Evaluation of GANs for Language Generation

Toward Diverse Text Generation with Inverse Reinforcement Learning

Towards Diverse Text Generation with Inverse Reinforcement Learning

Eval all, trust a few, do wrong to none: Comparing sentence generation models

Neural Text Generation: Past, Present and Beyond

Texygen: A Benchmarking Platform for Text Generation Models

MaskGAN: Better Text Generation via Filling in the ______

Hierarchical Text Generation and Planning for Strategic Dialogue

Generating Sentences by Editing Prototypes

Long Text Generation via Adversarial Training with Leaked Information

Adversarial evaluation for open-domain dialogue generation

Controlling Linguistic Style Aspects in Neural Language Generation

Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation

Adversarial Feature Matching for Text Generation

Adversarial Ranking for Language Generation

Adversarial Generation of Natural Language

VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning

Learning to Generate Product Reviews from Attributes

Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation

Semi and Weakly Supervised Semantic Segmentation Using Generative Adversarial Network

Deep Photo Style Transfer

Towards Diverse and Natural Image Descriptions via a Conditional GAN

Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features

Maximum-Likelihood Augmented Discrete Generative Adversarial Networks

Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses

Adversarial Evaluation of Dialogue Models

Adversarial Learning for Neural Dialogue Generation

NIPS 2016 Tutorial: Generative Adversarial Networks

Context-aware Natural Language Generation with Recurrent Neural Networks

Semantic Segmentation using Adversarial Networks

Unsupervised Cross-Domain Image Generation

Categorical Reparameterization with Gumbel-Softmax

Professor Forcing: A New Algorithm for Training Recurrent Networks

Neural Photo Editing with Introspective Adversarial Networks

SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient

Generative Visual Manipulation on the Natural Image Manifold

Improved Techniques for Training GANs

How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

XGBoost: A Scalable Tree Boosting System

Practical Black-Box Attacks against Machine Learning

Exploring the Limits of Language Modeling

Generating Sentences from a Continuous Space

How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?

A note on the evaluation of generative models

A Neural Attention Model for Abstractive Sentence Summarization

The proof and measurement of association between two things.

Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems

Data Generation as Sequential Decision Making

Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks

Highway Networks

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Deep visual-semantic alignments for generating image descriptions

CIDEr: Consensus-based image description evaluation

Testing for Significance of Increased Correlation with Human Judgment

Chinese Poetry Generation with Recurrent Neural Networks

Sequence to Sequence Learning with Neural Networks

Neural Machine Translation by Jointly Learning to Align and Translate

Convolutional Neural Networks for Sentence Classification

Auto-Encoding Variational Bayes

One billion word benchmark for measuring progress in statistical language modeling

Text-based measures of document diversity

Generating Sequences With Recurrent Neural Networks

Context dependent recurrent neural network language model

Putting Human Assessments of Machine Translation Systems in Order

A Survey of Monte Carlo Tree Search Methods

Random Search for Hyper-Parameter Optimization

RNNLM - Recurrent Neural Network Language Modeling Toolkit

Finding Deceptive Opinion Spam by Any Stretch of the Imagination

Extensions of recurrent neural network language model

Why Does Unsupervised Pre-training Help Deep Learning?