Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models (2023-01-10T00:00:00.000000Z)

TL;DR

This paper shows that localization conclusions from representation denoising do not provide any insight into which model MLP layer would be best to edit in order to override an existing stored fact with a new one, and finds that which layer the authors edit is a far better predictor of performance.

Abstract

Language models learn a great quantity of factual information during pretraining, and recent work localizes this information to specific model weights like mid-layer MLP weights. In this paper, we find that we can change how a fact is stored in a model by editing weights that are in a different location than where existing methods suggest that the fact is stored. This is surprising because we would expect that localizing facts to specific model parameters would tell us where to manipulate knowledge in models, and this assumption has motivated past work on model editing methods. Specifically, we show that localization conclusions from representation denoising (also known as Causal Tracing) do not provide any insight into which model MLP layer would be best to edit in order to override an existing stored fact with a new one. This finding raises questions about how past work relies on Causal Tracing to select which model layers to edit. Next, we consider several variants of the editing problem, including erasing and amplifying facts. For one of our editing problems, editing performance does relate to localization results from representation denoising, but we find that which layer we edit is a far better predictor of performance. Our results suggest, counterintuitively, that better mechanistic understanding of how pretrained language models work may not always translate to insights about how to best change their behavior. Our code is available at https://github.com/google/belief-localization

Authors

Mohit Bansal

47 papers

Asma Ghandeharioun

4 papers

Been Kim

4 papers

TL;DR

Abstract

Authors

References61 items

Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspaces

Finding Skill Neurons in Pre-trained Transformer-based Language Models

Mass-Editing Memory in a Transformer

Local Relighting of Real Scenes

Memory-Based Model Editing at Scale

Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space

Natural Language Descriptions of Deep Visual Features

Sparse Interventions in Language Models with Differentiable Masking

Editing a classifier by rewriting its prediction rules

Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs

Fast Model Editing at Scale

Knowledge Neurons in Pretrained Transformers

Editing Factual Knowledge in Language Models

An Interpretability Illusion for BERT

Of Non-Linearity and Commutativity in BERT

Transformer Feed-Forward Layers Are Key-Value Memories

Modifying Memories in Transformer Models

Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks

Understanding the role of individual units in a deep neural network

Rewriting a Deep Generative Model

Invertible Concept-based Explanations for CNN Models with Non-negative Concept Activation Vectors

Compositional Explanations of Neurons

Mechanisms for handling nested dependencies in neural-network language models and humans

Explaining Neural Networks by Decoding Layer Activations

Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias

A Primer in BERTology: What We Know About How BERT Works

Language Models as Knowledge Bases?

The emergence of number and syntax units in LSTM language models

Towards Automatic Concept-based Explanations

Interpretable Basis Decomposition for Visual Explanation

The Building Blocks of Interpretability

Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)

Interpreting Deep Visual Representations via Network Dissection

Zero-Shot Relation Extraction via Reading Comprehension

Learning to Generate Reviews and Discovering Sentiment

Visualizing and Understanding Convolutional Networks

Efficient Estimation of Word Representations in Vector Space

Locating and Editing Factual Knowledge in GPT

mesh-transformer-jax, May 2021

A mathematical framework for transformer circuits

GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model

Constrained Finetuning (window size 1)

Language Models are Unsupervised Multitask Learners

To reconnect tracing localization conclusions with editing performance

s ∗ is a “neighboring” entity to the subject s (similar to s ) for which ( s ∗ , r, o ) is a true fact like ( s, r, o )

it makes little difference whether we edit at one layer or ﬁve layers in order to match the number of representations restored by Causal Tracing.

Fact Forcing

that model edit success is essentially unrelated to where factual information is stored in models, as measured by Causal Tracing

Graphical clusterability and local specialization in deep neural networks

s noise is a noised representation of the subject s . We add Gaussian noise to the token embeddings of s , following Meng et al. (2022a)

o false is an object that incorrectly completes the tuple ( s, r, · ) . CounterFact contains an o false for each datapoint, intended to be the new model output when evaluating model editing methods

amount of probability mass to be erased

The edit method from Sec. 4, ROME edits a single MLP layer’s down-projection weight

Neighborhood Score

of the Association for Computational Linguistics

Fact Ampliﬁcation

Fact Erasure . FT-1: norm constraint of 1e-5

r ∗ is a paraphrase of the relation r , such as “is situated in” for “is located in.”

Tracing Reversal . FT-1: Norm constraint of 1e-5. FT-5: Norm constraint of 2e-5. FT-5: 2e-5. ROME: default parameters

Does Localization Inform Editing? Surprising Differences in Causality-based Localization vs. Knowledge Editing in LMs

Tracing Reversal . We maximize p θ ( o noise | s, r ) , aiming to change the model output from o true back to the output for the “original” noised input P = ( s noise , r ) in

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names