Whether you can locate or not? Interactive Referring Expression Generation (2023-08-19T00:00:00.000000Z)

TL;DR

An Interactive REG (IREG) model is proposed that can interact with a real REC model, utilizing signals indicating whether the object is located and the visual region located by the REC model to gradually modify REs.

Abstract

Referring Expression Generation (REG) aims to generate unambiguous Referring Expressions (REs) for objects in a visual scene, with a dual task of Referring Expression Comprehension (REC) to locate the referred object. Existing methods construct REG models independently by using only the REs as ground truth for model training, without considering the potential interaction between REG and REC models. In this paper, we propose an Interactive REG (IREG) model that can interact with a real REC model, utilizing signals indicating whether the object is located and the visual region located by the REC model to gradually modify REs. Our experimental results on three RE benchmark datasets, RefCOCO, RefCOCO+, and RefCOCOg show that IREG outperforms previous state-of-the-art methods on popular evaluation metrics. Furthermore, a human evaluation shows that IREG generates better REs with the capability of interaction.

Authors

Xiaojie Wang

4 papers

Fangxiang Feng

2 papers

Fulong Ye

2 papers

TL;DR

Abstract

Authors

References32 items

Towards Unifying Reference Expression Generation and Comprehension

OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning

Iterative Shrinking for Referring Expression Grounding Using Deep Reinforcement Learning

Learning Transferable Visual Models From Natural Language Supervision

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

Unifying Vision-and-Language Tasks via Text Generation

UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning

CoNAN: A Complementary Neighboring-based Attention Network for Referring Expression Generation

Referring Expression Generation via Visual Dialogue

Ref-NMS: Breaking Proposal Bottlenecks in Two-Stage Referring Expression Grounding

Improving One-stage Visual Grounding by Recursive Sub-query Construction

Attribute-Guided Attention for Referring Expression Generation and Comprehension

UNITER: UNiversal Image-TExt Representation Learning

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

A Fast and Accurate One-Stage Approach to Visual Grounding

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Learning to Compose and Reason with Language Tree Structures for Visual Grounding

Learning to Assemble Neural Module Tree Networks for Visual Grounding

Generating Easy-to-Understand Referring Expressions for Target Identifications

Bundled Object Context for Referring Expressions

MAttNet: Modular Attention Network for Referring Expression Comprehension

Referring Expression Generation and Comprehension via Attributes

A Joint Speaker-Listener-Reinforcer Model for Referring Expressions

Modeling Context in Referring Expressions

Generation and Comprehension of Unambiguous Object Descriptions

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

ReferItGame: Referring to Objects in Photographs of Natural Scenes

Microsoft COCO: Common Objects in Context

A Proposal-Free One-Stage Framework for Referring Expression Comprehension and Generation via Dense Cross-Attention

Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

MM ’23, October 29–November 3, 2023, Ottawa, ON, Canada

Field of Study

Journal Information

Name

Venue Information

Name

Type

URL

Alternate Names