Assessing Knowledge Editing in Language Models via Relation Perspective
Published in
arXiv.org(2023)External Links:
TL
TL;DR
A new benchmark named RaKE is constructed, which focuses on Relation based Knowledge Editing, and confirms that knowledge related to relations is not only stored in the FFN network but also in the attention layers, which provides experimental support for future relation-based knowledge editing methods.
Abstract
Knowledge Editing (KE) for modifying factual knowledge in Large Language Models (LLMs) has been receiving increasing attention. However, existing knowledge editing methods are entity-centric, and it is unclear whether this approach is suitable for a relation-centric perspective. To address this gap, this paper constructs a new benchmark named RaKE, which focuses on Relation based Knowledge Editing. In this paper, we establish a suite of innovative metrics for evaluation and conduct comprehensive experiments involving various knowledge editing baselines. We notice that existing knowledge editing methods exhibit the potential difficulty in their ability to edit relations. Therefore, we further explore the role of relations in factual triplets within the transformer. Our research results confirm that knowledge related to relations is not only stored in the FFN network but also in the attention layers. This provides experimental support for future relation-based knowledge editing methods.
References25 items
1
EX-FEVER: A Dataset for Multi-hop Explainable Fact Verification
2
Memory Injections: Correcting Multi-Hop Reasoning Failures During Inference in Transformer-Based Language Models
3
MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions
4
Can We Edit Factual Knowledge by In-Context Learning?
5