Towards Semantic Consistency: Dirichlet Energy Driven Robust Multi-Modal Entity Alignment (2024-01-31T00:00:00.000000Z)

TL;DR

DESAlign is proposed, a robust method addressing the over-smoothing caused by semantic inconsistency and interpolating missing semantics using existing modalities using existing modalities, and a training strategy for multi-modal knowledge graph learning based on the proposed generalizable theoretical principle.

Abstract

Multi-Modal Entity Alignment (MMEA) is a pivotal task in Multi-Modal Knowledge Graphs (MMKGs), seeking to identify identical entities by leveraging associated modal attributes. However, real-world MMKGs confront the challenges of semantic inconsistency arising from diverse and incomplete data sources. This inconsistency is predominantly caused by the absence of specific modal attributes, manifesting in two distinct forms: disparities in attribute counts or the absence of certain modalities. Current methods address these issues through attribute interpolation, but their reliance on predefined distributions introduces modality noise, compromising original semantic information. Furthermore, the absence of a generalizable theoretical principle hampers progress towards achieving semantic consistency. In this work, we propose a generalizable theoretical principle by examining semantic consistency from the perspective of Dirichlet energy. Our research reveals that, in the presence of semantic inconsistency, models tend to overfit to modality noise, leading to over-smoothing and performance oscillations or declines, particularly in scenarios with a high rate of missing modality. To overcome these challenges, we propose DESAlign, a robust method addressing the over-smoothing caused by semantic inconsistency and interpolating missing semantics using existing modalities. Specifically, we devise a training strategy for multi-modal knowledge graph learning based on our proposed principle. Then, we introduce a propagation strategy that utilizes existing features to provide interpolation solutions for missing semantic features. DESAlign outperforms existing approaches across 60 benchmark splits, encompassing both monolingual and bilingual scenarios, achieving state-of-the-art performance. Experiments on splits with high missing modal attributes demonstrate its effectiveness, providing a robust MMEA solution to semantic inconsistency in real-world MMKGs.

References63 items

Gradient Flow of Energy: A General and Efficient Approach for Entity Alignment Decoding

Cross-Graph Embedding With Trainable Proximity for Graph Alignment

Revisiting Embedding-Based Entity Alignment: A Robust and Adaptive Method

Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment

Multi-order Matched Neighborhood Consistent Graph Alignment in a Union Vector Space

Multi-Modal Knowledge Hypergraph for Diverse Image Retrieval

A Fractional Graph Laplacian Approach to Oversmoothing

Data Imputation from the Perspective of Graph Dirichlet Energy

Attribute-Consistent Knowledge Graph Representation Learning for Multi-Modal Entity Alignment

MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid

Guiding Neural Entity Alignment with Compatibility

Multi-modal Contrastive Representation Learning for Entity Alignment

Multi-modal Siamese Network for Entity Alignment

Semantic consistency for graph representation learning

SelfKG: Self-Supervised Entity Alignment in Knowledge Graphs

Multi-Modal Knowledge Graph Construction and Application: A Survey

On the Unreasonable Effectiveness of Feature propagation in Learning on Graphs with Missing Node Features

Dirichlet Energy Constrained Learning for Deep Graph Neural Networks

Multi-modal Entity Alignment in Hyperbolic Space

DeepViT: Towards Deeper Vision Transformer

Skeleton Aware Multi-modal Sign Language Recognition

Probabilistic Embeddings for Cross-Modal Retrieval

Entity Alignment for Knowledge Graphs With Multi-Order Convolutional Networks

Multi-modal Knowledge Graphs for Recommender Systems

Exploring and Evaluating Attributes, Values, and Structure for Entity Alignment

Visual Pivoting for (Unsupervised) Entity Alignment

MMEA: Entity Alignment for Multi-modal Knowledge Graph

Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review

COTSAE: CO-Training of Structure and Attribute Embeddings for Entity Alignment

A Simple Framework for Contrastive Learning of Visual Representations

Collective Entity Alignment via Adaptive Features

Knowledge Graph Alignment Network with Gated Multi-hop Neighborhood Aggregation

Cross-Modal Image-Text Retrieval with Semantic Consistency

Aligning Cross-Lingual Entities with Multi-Aspect Information

Neighborhood-Aware Attentional Representation for Multilingual Knowledge Graphs

Multi-Modal Question Answering System Driven by Domain Knowledge Graph

Entity Alignment between Knowledge Graphs Using Attribute Embeddings

Multi-Channel Graph Neural Network for Entity Alignment

Deep Supervised Cross-Modal Retrieval

Semi-Supervised Entity Alignment via Knowledge Graph Embedding with Awareness of Degree Difference

Unsupervised Entity Alignment Using Attribute Triples and Relation Triples

MMKG: Multi-Modal Knowledge Graphs

Spectral graph theory

Bootstrapping Entity Alignment with Knowledge Graph Embedding

Co-training Embeddings of Knowledge Graphs and Entity Descriptions for Cross-lingual Entity Alignment

Graph Attention Networks

Iterative Entity Alignment via Joint Knowledge Embeddings

Cross-Lingual Entity Alignment via Joint Attribute-Preserving Embedding

Attention is All you Need

Multilingual Knowledge Graph Embeddings for Cross-lingual Knowledge Alignment

Deep Residual Learning for Image Recognition

Embedding Entities and Relations for Learning and Inference in Knowledge Bases

Very Deep Convolutional Networks for Large-Scale Image Recognition

Translating Embeddings for Modeling Multi-relational Data

PARIS: Probabilistic Alignment of Relations, Instances, and Schema

LogMap: Logic-Based and Scalable Ontology Matching

Understanding the difficulty of training deep feedforward neural networks

Freebase: a collaboratively created graph database for structuring human knowledge

Spectral Graph Theory

MultiJAF: Multi-modal joint entity alignment framework for multi-modal knowledge graph

Cross-lingual Knowledge Graph Alignment via Graph Convolutional Networks

DBpedia - A large-scale, multilingual knowledge base extracted from Wikipedia

license agreement with IEEE. Restrictions apply