ViQuAE, a Dataset for Knowledge-based Visual Question Answering about Named Entities (2022-07-06T00:00:00.000000Z)

TL;DR

This work addresses KVQAE as a two-stage problem: Information Retrieval and Reading Comprehension, with both zero- and few-shot learning methods, and paves the way for better multimodal entity representations and question answering.

Abstract

Whether to retrieve, answer, translate, or reason, multimodality opens up new challenges and perspectives. In this context, we are interested in answering questions about named entities grounded in a visual context using a Knowledge Base (KB). To benchmark this task, called KVQAE (Knowledge-based Visual Question Answering about named Entities), we provide ViQuAE, a dataset of 3.7K questions paired with images. This is the first KVQAE dataset to cover a wide range of entity types (e.g. persons, landmarks, and products). The dataset is annotated using a semi-automatic method. We also propose a KB composed of 1.5M Wikipedia articles paired with images. To set a baseline on the benchmark, we address KVQAE as a two-stage problem: Information Retrieval and Reading Comprehension, with both zero- and few-shot learning methods. The experiments empirically demonstrate the difficulty of the task, especially when questions are not about persons. This work paves the way for better multimodal entity representations and question answering. The dataset, KB, code, and semi-automatic annotation pipeline are freely available at https://github.com/PaulLerner/ViQuAE.

Authors

Paul Lerner

1 papers

Olivier Ferret

1 papers

C. Guinaudeau

1 papers

TL;DR

Abstract

Authors

References60 items

MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding

Datasets: A Community Library for Natural Language Processing

WebQA: Multihop and Multimodal QA

On the Opportunities and Risks of Foundation Models

MultiModalQA: Complex Question Answering over Text, Tables and Images

A Replication Study of Dense Passage Retriever

Learning Transferable Visual Models From Natural Language Supervision

Zero-Shot Text-to-Image Generation

ConceptBert: Concept-Aware Representation for Visual Question Answering

Findings of the Association for Computational Linguistics: EMNLP 2020

KILT: a Benchmark for Knowledge Intensive Language Tasks

Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets

Language Models are Few-Shot Learners

Visuo-Lingustic Question Answering (VLQA) Challenge

Dense Passage Retrieval for Open-Domain Question Answering

PyTorch: An Imperative Style, High-Performance Deep Learning Library

HuggingFace's Transformers: State-of-the-art Natural Language Processing

Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering

Natural Questions: A Benchmark for Question Answering Research

KVQA: Knowledge-Aware Visual Question Answering

OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge

HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering

Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking

ArcFace: Additive Angular Margin Loss for Deep Face Recognition

Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks

Simple and Effective Multi-Paragraph Reading Comprehension

Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

Are You Smarter Than a Sixth Grader? Textbook Question Answering for Multimodal Machine Comprehension

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

Reading Wikipedia to Answer Open-Domain Questions

Billion-Scale Similarity Search with GPUs

MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition

FVQA: Fact-Based Visual Question Answering

SQuAD: 100,000+ Questions for Machine Comprehension of Text

Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks

Deep Residual Learning for Image Recognition

Explicit Knowledge-based Reasoning for Visual Question Answering

VQA: Visual Question Answering

Microsoft COCO: Common Objects in Context

CNN Features Off-the-Shelf: An Astounding Baseline for Recognition

TAGME: on-the-fly annotation of short text fragments (by wikipedia entities)

ImageNet: A large-scale hierarchical image database

A comparison of statistical significance tests for information retrieval evaluation

The CLEF Cross Language Image Retrieval Track (ImageCLEF) 2004

Building a question answering test collection

Intelligent Indexing and Semantic Retrieval of Multimodal Documents

Generating Referring Expressions Involving Relations

ranx: A Blazing-Fast Python Library for Ranking Evaluation and Comparison

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Computational Generation of Referring Expressions: An Updated Survey

Adam: AMethod for Stochastic Optimization

Okapi at TREC-3

Measuring nominal scale agreement among many raters.

The design of experiments. The design of experiments

Adam optimizer with a constant learning rate of 10 − 5 , 𝛽 1 = 0 . 9, = 0 . 999. Gradients’ norms are clipped at 1.

articles in the KB

B.1 Information Retrieval For training DPR, we use the same hyperparameters

We train DPR using 4 V100 GPUs of 32GB, allowing total batch size of 256 (32 questions × 2 passages each × 4 GPUs

Rishi Bommasani

A KNOWLEDGE BASE As explained in Section 5,

Field of Study

Journal Information

Name

Venue Information

Name

Type

URL

Alternate Names