Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment

Published in

International Workshop on the Semantic Web(2023)

External Links:

Generate Graph DownloadPDF

TL;DR

UMAEA is introduced, a robust multi-modal entity alignment approach designed to tackle uncertainly missing and ambiguous visual modalities, which consistently achieves SOTA performance across all 97 benchmark splits, significantly surpassing existing baselines with limited parameters and time consumption, while effectively alleviating the identified limitations of other models.

Abstract

As a crucial extension of entity alignment (EA), multi-modal entity alignment (MMEA) aims to identify identical entities across disparate knowledge graphs (KGs) by exploiting associated visual information. However, existing MMEA approaches primarily concentrate on the fusion paradigm of multi-modal entity features, while neglecting the challenges presented by the pervasive phenomenon of missing and intrinsic ambiguity of visual images. In this paper, we present a further analysis of visual modality incompleteness, benchmarking latest MMEA models on our proposed dataset MMEA-UMVM, where the types of alignment KGs covering bilingual and monolingual, with standard (non-iterative) and iterative training paradigms to evaluate the model performance. Our research indicates that, in the face of modality incompleteness, models succumb to overfitting the modality noise, and exhibit performance oscillations or declines at high rates of missing modality. This proves that the inclusion of additional multi-modal data can sometimes adversely affect EA. To address these challenges, we introduce UMAEA , a robust multi-modal entity alignment approach designed to tackle uncertainly missing and ambiguous visual modalities. It consistently achieves SOTA performance across all 97 benchmark splits, significantly surpassing existing baselines with limited parameters and time consumption, while effectively alleviating the identified limitations of other models. Our code and benchmark data are available at https://github.com/zjukg/UMAEA.

Authors

Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment

Published in

International Workshop on the Semantic Web(2023)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Hua-zeng Chen

10 papers

Y. Li

3 papers

Zhuo Chen

10 papers

Lingbing Guo

6 papers

Yin Fang

5 papers

Yichi Zhang

3 papers

Jiaoyan Chen

17 papers

Jeff Z. Pan

7 papers

Wen Zhang

9 papers

References63 items

Revisiting Embedding-Based Entity Alignment: A Robust and Adaptive Method

Revisit and Outstrip Entity Alignment: A Perspective of Generative Models

Knowledge graph-enhanced molecular contrastive learning with functional prompt

mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality

A Multi-Modal Entity Alignment Method with Inter-Modal Enhancement

Deep Active Alignment of Knowledge Graph Entities and Schemata

Probing the Impacts of Visual Context in Multimodal Entity Alignment

Vision, Deduction and Alignment: An Empirical Study on Multi-Modal Knowledge Graph Alignment

MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid

Multi-modal Contrastive Representation Learning for Entity Alignment

Multi-modal Siamese Network for Entity Alignment

LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injection

Facing Changes: Continual Entity Alignment for Growing Knowledge Graphs

TL;DR

Abstract

Authors

TL;DR

Abstract

Authors

References63 items

Revisiting Embedding-Based Entity Alignment: A Robust and Adaptive Method

Revisit and Outstrip Entity Alignment: A Perspective of Generative Models

Knowledge graph-enhanced molecular contrastive learning with functional prompt

mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality

A Multi-Modal Entity Alignment Method with Inter-Modal Enhancement

Deep Active Alignment of Knowledge Graph Entities and Schemata

Probing the Impacts of Visual Context in Multimodal Entity Alignment

Vision, Deduction and Alignment: An Empirical Study on Multi-Modal Knowledge Graph Alignment

MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid

Multi-modal Contrastive Representation Learning for Entity Alignment

Multi-modal Siamese Network for Entity Alignment

LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injection

Facing Changes: Continual Entity Alignment for Growing Knowledge Graphs

DUET: Cross-modal Semantic Grounding for Contrastive Zero-shot Learning

ClusterEA: Scalable Entity Alignment with Stochastic Training and Normalized Mini-batch Similarities

Entity Alignment with Reliable Path Reasoning and Relation-Aware Heterogeneous Graph Transformer

Semantics Driven Embedding Learning for Effective Entity Alignment

Informed Multi-context Entity Alignment

Zero-Shot and Few-Shot Learning With Knowledge Graphs: A Comprehensive Survey

Molecular Contrastive Learning with Chemical Element Knowledge Graph

Entity and Relation Matching Consensus for Entity Alignment

Zero-shot Visual Question Answering using Knowledge Graph

Unsupervised Knowledge Graph Alignment by Probabilistic Reasoning and Semantic Embedding

Learning Transferable Visual Models From Natural Language Supervision

Knowledge-aware Zero-Shot Learning: Survey and Perspective

OntoZSL: Ontology-enhanced Zero-shot Learning

Richpedia: A Large-Scale, Comprehensive Multi-Modal Knowledge Graph

Incomplete Cross-modal Retrieval with Dual-Aligned Variational Autoencoders

Exploring and Evaluating Attributes, Values, and Structure for Entity Alignment

Visual Pivoting for (Unsupervised) Entity Alignment

MMEA: Entity Alignment for Multi-modal Knowledge Graph

BERT-INT: A BERT-based Interaction Model For Knowledge Graph Alignment

Neighborhood Matching Network for Entity Alignment

Meta Dropout: Learning to Perturb Latent Features for Generalization

A benchmarking study of embedding-based entity alignment for knowledge graphs

A Simple Framework for Contrastive Learning of Visual Representations

Knowledge Graph Alignment Network with Gated Multi-hop Neighborhood Aggregation

Semi-supervised Entity Alignment via Joint Knowledge Embedding Model and Cross-graph Model

TransEdge: Translating Relation-Contextualized Embeddings for Knowledge Graphs

Aligning Cross-Lingual Entities with Multi-Aspect Information

Neighborhood-Aware Attentional Representation for Multilingual Knowledge Graphs

Entity Alignment between Knowledge Graphs Using Attribute Embeddings

Multi-Channel Graph Neural Network for Entity Alignment

Multi-view Knowledge Graph Embedding for Entity Alignment

Relation-Aware Entity Alignment for Heterogeneous Knowledge Graphs

MMKG: Multi-Modal Knowledge Graphs

Bootstrapping Entity Alignment with Knowledge Graph Embedding

Graph Attention Networks

Cross-Lingual Entity Alignment via Joint Attribute-Preserving Embedding

Attention is All you Need

Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics

Deep Residual Learning for Image Recognition

Learning Structured Output Representation using Deep Conditional Generative Models

Embedding Entities and Relations for Learning and Inference in Knowledge Bases

Translating Embeddings for Modeling Multi-relational Data

PARIS: Probabilistic Alignment of Relations, Instances, and Schema

LogMap: Logic-Based and Scalable Ontology Matching

Multi-Modal Entity Alignment Using Uncertainty Quantification for Modality Importance

Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities

AUTO-ENCODING VARIATIONAL BAYES

Unpaired image-to-image translation using cycle-consistent adversarial networks

DBpedia - A large-scale, multilingual knowledge base extracted from Wikipedia

Title Suppressed Due to Excessive Length 19

Field of Study

Venue Information

Name

Type

URL

Alternate Names