A Survey on Interpretable Cross-modal Reasoning (2023-09-05T00:00:00.000000Z)

TL;DR

This survey delves into the realm of interpretable cross-modal reasoning (I-CMR), where the objective is not only to achieve high predictive performance but also to provide human-understandable explanations for the results.

Abstract

In recent years, cross-modal reasoning (CMR), the process of understanding and reasoning across different modalities, has emerged as a pivotal area with applications spanning from multimedia analysis to healthcare diagnostics. As the deployment of AI systems becomes more ubiquitous, the demand for transparency and comprehensibility in these systems' decision-making processes has intensified. This survey delves into the realm of interpretable cross-modal reasoning (I-CMR), where the objective is not only to achieve high predictive performance but also to provide human-understandable explanations for the results. This survey presents a comprehensive overview of the typical methods with a three-level taxonomy for I-CMR. Furthermore, this survey reviews the existing CMR datasets with annotations for explanations. Finally, this survey summarizes the challenges for I-CMR and discusses potential future directions. In conclusion, this survey aims to catalyze the progress of this emerging research area by providing researchers with a panoramic and comprehensive perspective, illuminating the state of the art and discerning the opportunities. The summarized methods, datasets, and other resources are available at https://github.com/ZuyiZhou/Awesome-Interpretable-Cross-modal-Reasoning.

Authors

Dizhan Xue

5 papers

Shengsheng Qian

3 papers

Changsheng Xu

3 papers

TL;DR

Abstract

Authors

References117 items

Decision-Making Driven by Driver Intelligence and Environment Reasoning for High-Level Autonomous Vehicles: A Survey

Decoding the Underlying Meaning of Multimodal Hateful Memes

Interpretable Multimodal Misinformation Detection with Logic Reasoning

T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Large Language Model Signals for Science Question Answering

Zero-Shot Everything Sketch-Based Image Retrieval, and in Explainable Style

Video-Text as Game Players: Hierarchical Banzhaf Interaction for Cross-Modal Representation Learning

NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations

LLaMA: Open and Efficient Foundation Language Models

Multimodal Chain-of-Thought Reasoning in Language Models

UnICLAM: Contrastive Representation Learning with Adversarial Masking for Unified and Interpretable Medical Vision Question Answering

From Images to Textual Prompts: Zero-shot Visual Question Answering with Frozen Large Language Models

Relation-Aware Image Captioning for Explainable Visual Question Answering

SLAN: Self-Locator Aided Network for Cross-Modal Understanding

Visual Programming: Compositional visual reasoning without training

MMT: Image-guided Story Ending Generation with Multimodal Memory Transformer

Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

Explainable AI (XAI): Core Ideas, Techniques, and Solutions

Reasoning about Actions over Visual and Linguistic Modalities: A Survey

Integrating Multi-Label Contrastive Learning With Dual Adversarial Graph Neural Networks for Cross-Modal Retrieval

Cross-modal Representation Learning and Relation Reasoning for Bidirectional Adaptive Manipulation

MultiViz: Towards Visualizing and Understanding Multimodal Models

Multimodal Learning With Transformers: A Survey

Query and Attention Augmentation for Knowledge-Based Explainable Reasoning

End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Models

X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrieval

MuKEA: Multimodal Knowledge Extraction and Accumulation for Knowledge-based Visual Question Answering

REX: Reasoning-aware and Grounded Explanation

Cross-modal Graph Matching Network for Image-text Retrieval

DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations

A Review of Emerging Research Directions in Abstract Visual Reasoning

A survey of multimodal deep generative models

Grounding Answers for Visual Questions Asked by Visually Impaired People

Deconfounded Visual Grounding

SPACES: Explainable Multimodal AI for Active Surveillance, Diagnosis, and Management of Adverse Childhood Experiences (ACEs)

Evidential reasoning for preprocessing uncertain categorical data for trustworthy decisions: An application on healthcare and finance

Multi-Modal Sarcasm Detection with Interactive In-Modal and Cross-Modal Graphs

Exploring Logical Reasoning for Referring Expression Comprehension

Coarse-to-Fine Reasoning for Visual Question Answering

ProTo: Program-Guided Transformer for Program-Guided Tasks

Cross-modal Relational Reasoning Network for Visual Question Answering

Hierarchical Cross-Modal Graph Consistency Learning for Video-Text Retrieval

LayerCAM: Exploring Hierarchical Class Activation Maps for Localization

Interpretable visual reasoning: A survey

Step-Wise Hierarchical Alignment Network for Image-Text Matching

Learning Relation Alignment for Calibrated Cross-modal Retrieval

Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval

GLM: General Language Model Pretraining with Autoregressive Blank Infilling

Multi-Level Knowledge Injecting for Visual Commonsense Reasoning

Comparative analysis on cross-modal information retrieval: A review

Explaining the black-box model: A survey of local interpretation methods for deep neural networks

Beyond VQA: Generating Multi-word Answers and Rationales to Visual Questions

Semantics of the Black-Box: Can Knowledge Graphs Help Make Deep Learning Systems More Interpretable and Explainable?

Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs

Visual Grounding Via Accumulated Attention

KBGN: Knowledge-Bridge Graph Network for Adaptive Vision-Text Reasoning in Visual Dialogue

A Survey on Performance Metrics for Object-Detection Algorithms

2020 International Conference on Systems, Signals and Image Processing (IWSSIP)

Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answering

Language Models are Few-Shot Learners

Robust and Interpretable Grounding of Spatial References with Relation Networks

Improving Vision-and-Language Navigation with Image-Text Pairs from the Web

Program Guided Agent

Linguistically Driven Graph Capsule Network for Visual Question Reasoning

VQA-LOL: Visual Question Answering under the Lens of Logic

Reasoning on the Relation: Enhancing Visual Representation for Visual Question Answering and Cross-Modal Retrieval

Integrating Non-monotonic Logical Reasoning and Inductive Learning With Deep Learning for Explainable Visual Question Answering

Learning Cross-modal Context Graph for Visual Grounding

Meta Module Network for Compositional Visual Reasoning

Explainable High-order Visual Question Reasoning: A New Benchmark and Knowledge-routed Network

Visual Semantic Reasoning for Image-Text Matching

KVQA: Knowledge-Aware Visual Question Answering

Multimodal Logical Inference System for Visual-Textual Entailment

OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge

Full-Gradient Representation for Neural Network Visualization

Relation-Aware Graph Attention Network for Visual Question Answering

GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering