Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Dataset (2023-02-28T00:00:00.000000Z)

TL;DR

This work annotate Instruction Clarification Requests in CoDraw, an existing dataset of interactions in a multimodal collaborative dialogue game, and shows that it contains lexically and semantically diverse iCRs being produced self-motivatedly by players deciding to clarify in order to solve the task successfully.

Abstract

In visual instruction-following dialogue games, players can engage in repair mechanisms in face of an ambiguous or underspecified instruction that cannot be fully mapped to actions in the world. In this work, we annotate Instruction Clarification Requests (iCRs) in CoDraw, an existing dataset of interactions in a multimodal collaborative dialogue game. We show that it contains lexically and semantically diverse iCRs being produced self-motivatedly by players deciding to clarify in order to solve the task successfully. With 8.8k iCRs found in 9.9k dialogues, CoDraw-iCR (v1) is a large spontaneous iCR corpus, making it a valuable resource for data-driven research on clarification in dialogue. We then formalise and provide baseline models for two tasks: Determining when to make an iCR and how to recognise them, in order to investigate to what extent these tasks are learnable from data.

Authors

David Schlangen

4 papers

Brielen Madureira

1 papers

TL;DR

Abstract

Authors

References67 items

SemEval-2022 Task 7: Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts

Collecting Interactive Multi-modal Datasets for Grounded Language Understanding

Dialog Acts for Task Driven Embodied Agents

IGLU 2022: Interactive Grounded Language Understanding in a Collaborative Environment at NeurIPS 2022

Learning to Execute Actions or Ask Clarification Questions

DialFRED: Dialogue-Enabled Agents for Embodied Instruction Following

LatteGAN: Visually Guided Language Attention for Multi-Turn Text-Conditioned Image Manipulation

NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environment

TEACh: Task-driven Embodied Agents that Chat

Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions

How Should Agents Ask Questions For Situated Learning? An Annotated Dialogue Corpus

A recipe for annotating grounded clarifications

Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge

Interactive Question Clarification in Dialogue via Reinforcement Learning

IR-GAN: Image Manipulation with Linguistic Instruction by Increment Reasoning

Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoning

Text as Neural Operator:Image Manipulation by Text Instruction

ClarQ: A large-scale and diverse dataset for Clarification Question Generation

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Asking Clarifying Questions in Open-Domain Information-Seeking Conversations

Vision-and-Dialog Navigation

Collaborative Dialogue in Minecraft

Language-Capable Robots may Inadvertently Weaken Human Moral Norms

Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science

Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instruction

Robot: Asker of questions and changer of norms?

Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information

CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication

Modeling the clarification potential of instructions: Predicting clarification requests and other reactions

What Do You Mean Exactly?: Analyzing Clarification Questions in CQA

Deep Residual Learning for Image Recognition

Miscommunication Recovery in Physically Situated Dialogue

Clarifying Intentions in Dialogue: A Corpus Study

The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets

Adam: A Method for Stochastic Optimization

Learning the Visual Interpretation of Sentences

Modelling Human Clarification Strategies

Bringing Semantics into Focus Using Visual Abstraction

Clarifying commands with information-theoretic human-robot dialog

Exploring Miscommunication and Collaborative Behaviour in Human-Robot Interaction

Clarification Potential of Instructions

Using Machine Learning to Explore Human Multimodal Clarification Strategies

An Analysis of Clarification Dialogue for Question Answering

On the Means for Clarification in Dialogue

A Coefficient of Agreement for Nominal Scales

On Using Language

Dialogue Collection for Recording the Process of Building Common Ground in a Collaborative Task

Visual Thinking of Neural Networks: Interactive Text to Image Synthesis

Requesting clarifications with speech and gestures

Virtual world context encoding for grounded dialogue in minecraft

Towards Natural Clariﬁcation Questions in Dialogue Systems

Clariﬁca-tion requests at the level of uptake

Clarification Questions with Feedback

Speaking through a noisy channel - experiments on inducing clarification behaviour in human-human dialogue

Beyond Repair: Testing the Limits of the Conversational Repair System

Grounding and Crification

A Corpus Collection and Annotation Framework for Learning Multimodal Clarification Strategies

Sorry and I Didn’t Catch That! - An Investigation of Non-understanding Errors and Recovery Strategies

Form, Intonation and Function of Clarification Requests in German Task-Oriented Spoken Dialogues

Causes and Strategies for Requesting Clarification in Dialogue

Experimenting with Clarification in Dialogue

Clarification in Spoken Dialogue Systems

An activity-based approach to pragmatics

Edinburgh Research Explorer Implications for Generating Clarification Requests in Task-Oriented Dialogues

Requesting clariﬁcations with speech and gestures

(cid:66) Frequency : relative and absolute occurrence of iCRs should be large enough for data-driven methods and statistical purposes

of the Association for Computational Linguistics

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names