Retrieve, Caption, Generate: Visual Grounding for Enhancing Commonsense in Text Generation Models (2021-09-08T00:00:00.000000Z)

TL;DR

Comprehensive evaluation and analysis demonstrate that VisCTG noticeably improves model performance while successfully addressing several issues of the baseline generations, including poor commonsense, fluency, and specificity.

Abstract

We investigate the use of multimodal information contained in images as an effective method for enhancing the commonsense of Transformer models for text generation. We perform experiments using BART and T5 on concept-to-text generation, specifically the task of generative commonsense reasoning, or CommonGen. We call our approach VisCTG: Visually Grounded Concept-to-Text Generation. VisCTG involves captioning images representing appropriate everyday scenarios, and using these captions to enrich and steer the generation process. Comprehensive evaluation and analysis demonstrate that VisCTG noticeably improves model performance while successfully addressing several issues of the baseline generations, including poor commonsense, fluency, and specificity.

Authors

E. Hovy

15 papers

T. Mitamura

5 papers

Malihe Alikhani

6 papers

TL;DR

Abstract

Authors

References49 items

SAPPHIRE: Approaches for Enhanced Concept-to-Text Generation

Retrieval Enhanced Model for Commonsense Generation

A Survey of Data Augmentation Approaches for NLP

NAREOR: The Narrative Reordering Problem

Learning Transferable Visual Models From Natural Language Supervision

Do Neural Language Models Overcome Reporting Bias?

An Enhanced Knowledge Injection Model for Commonsense Generation

Denoising Pre-Training and Data Augmentation Strategies for Enhanced RDF Verbalization with Transformers

COSMIC: COmmonSense knowledge for eMotion Identification in Conversations

GenAug: Data Augmentation for Finetuning Text Generators

KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning

A Survey on Deep Learning for Multimodal Data Fusion

CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning

oLMpics-On What Language Model Pre-training Captures

A Good Sample is Hard to Find: Noise Injection Sampling and Self-Training for Neural Language Generation Models

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

ALOHA: Artificial Learning of Human Attributes for Dialogue Agents

Keep Calm and Switch On! Preserving Sentiment and Fluency in Semantic Text Exchange

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Boosting Dialog Response Generation

COMET: Commonsense Transformers for Automatic Knowledge Graph Construction

BERTScore: Evaluating Text Generation with BERT

From Recognition to Cognition: Visual Commonsense Reasoning

CGMH: Constrained Sentence Generation by Metropolis-Hastings Sampling

Findings of the E2E NLG Challenge

Discriminability Objective for Training Descriptive Captions

The WebNLG Challenge: Generating Text from RDF Data

Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

Multimodal Machine Learning: A Survey and Taxonomy

Self-Critical Sequence Training for Image Captioning

SPICE: Semantic Propositional Image Caption Evaluation

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Deep Residual Learning for Image Recognition

A Diversity-Promoting Objective Function for Neural Conversation Models

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

CIDEr: Consensus-based image description evaluation

Microsoft COCO: Common Objects in Context

Reporting bias and knowledge acquisition

Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics

Bleu: a Method for Automatic Evaluation of Machine Translation

‘Bottom-up’ and ‘top-down’

Human Language Technologies, 110–119

If an annotator answers the quality-check question wrong (e.g. they choose the obviously bad output), their two remaining actual example annotations

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

TNT-NLG , System 2 : Data Repetition and Meaning Representation Manipulation to Improve Neural Generation

Significance Tests Which May be Applied to Samples from Any Populations

of the Association for Computational Linguistics

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names