Explaining black box text modules in natural language with language models (2023-05-17T00:00:00.000000Z)

TL;DR

This work introduces Summarize and Score (SASC), a method that takes in a text module and returns a natural language explanation of the module's selectivity along with a score for how reliable the explanation is, and shows that SASC can generate explanations for the response of individual fMRI voxels to language stimuli.

Abstract

Large language models (LLMs) have demonstrated remarkable prediction performance for a growing array of tasks. However, their rapid proliferation and increasing opaqueness have created a growing need for interpretability. Here, we ask whether we can automatically obtain natural language explanations for black box text modules. A"text module"is any function that maps text to a scalar continuous value, such as a submodule within an LLM or a fitted model of a brain region."Black box"indicates that we only have access to the module's inputs/outputs. We introduce Summarize and Score (SASC), a method that takes in a text module and returns a natural language explanation of the module's selectivity along with a score for how reliable the explanation is. We study SASC in 3 contexts. First, we evaluate SASC on synthetic modules and find that it often recovers ground truth explanations. Second, we use SASC to explain modules found within a pre-trained BERT model, enabling inspection of the model's internals. Finally, we show that SASC can generate explanations for the response of individual fMRI voxels to language stimuli, with potential applications to fine-grained brain mapping. All code for using SASC and reproducing results is made available on Github.

Authors

Jianfeng Gao

21 papers

Chandan Singh

7 papers

Shailee Jain

2 papers

TL;DR

Abstract

Authors

References105 items

Tree Prompting: Efficient Task Adaptation without Fine-Tuning

A Multi-Task Embedder For Retrieval Augmented LLMs

The Cortical Representation of Language Timescales is Shared between Reading and Listening

Llama 2: Open Foundation and Fine-Tuned Chat Models

Lexical semantic content, not syntactic structure, is the main contributor to ANN-brain similarity of fMRI responses in the language network

Explaining Language Models' Predictions with High-Impact Concepts

Finding Neurons in a Haystack: Case Studies with Sparse Probing

Towards Automated Circuit Discovery for Mechanistic Interpretability

N2G: A Scalable Approach for Quantifying Interpretable Neuron Representations in Large Language Models

Driving and suppressing the human language network using large language models

Can Large Language Models Transform Computational Social Science?

Goal Driven Discovery of Distributional Differences via Language Descriptions

Information-Restricted Neural Language Models Reveal Different Brain Regions’ Sensitivity to Semantics, Syntax, and Context

LLaMA: Open and Efficient Foundation Language Models

Guiding Large Language Models via Directional Stimulus Prompting

Computational Language Modeling and the Promise of In Silico Experimentation

Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?

One Embedder, Any Task: Instruction-Finetuned Text Embeddings

Joint processing of linguistic properties in brains and language models

Large Language Models Are Human-Level Prompt Engineers

Semantic reconstruction of continuous language from non-invasive brain recordings

Deep language algorithms predict semantic comprehension from brain activity

Augmenting interpretable models with large language models during training

A natural language fMRI dataset for voxelwise encoding models

A Review of Sparse Expert Models in Deep Learning

GSCLIP : A Framework for Explaining Distribution Shifts in Natural Language

OPT: Open Pre-trained Transformer Language Models

Predictability and stability testing to assess clinical decision instrument performance for children after blunt torso trauma

Shared computational principles for language processing in humans and deep language models

Describing Differences between Text Distributions with Natural Language

Natural Language Descriptions of Deep Visual Features

Neurocomputational Models of Language Processing

Visual and linguistic semantic representations are aligned at the border of human visual cortex

Toward a Visual Concept Vocabulary for GAN Latent Space

Parallel and distributed encoding of speech across human auditory cortex

Adaptive wavelet distillation from neural networks through interpretations

Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses

Knowledge Neurons in Pretrained Transformers

Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections

Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors

Disentangling syntax and semantics in the brain with deep networks

Learning from the Best: Rationalizing Prediction by Adversarial Information Calibration

Interpretable multi-timescale models for predicting fMRI responses to continuous natural speech

Understanding the role of individual units in a deep neural network

The neural architecture of language: Integrative modeling converges on predictive processing

Can fMRI reveal the representation of syntactic structure in the brain?

DeBERTa: Decoding-enhanced BERT with Disentangled Attention

Language Models are Few-Shot Learners

WT5?! Training Text-to-Text Models to Explain their Predictions

A Primer in BERTology: What We Know About How BERT Works

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Artificial Intelligence, Values, and Alignment

Explain Yourself! Leveraging Language Models for Commonsense Reasoning

Incorporating Priors with Feature Attribution on Text Classification

Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)

Explainable AI for Trees: From Local Explanations to Global Understanding

BERTScore: Evaluating Text Generation with BERT

Discovery of Natural Language Concepts in Individual Units of CNNs

What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models

e-SNLI: Natural Language Inference with Natural Language Explanations

From Recognition to Cognition: Visual Commonsense Reasoning

GAN Dissection: Visualizing and Understanding Generative Adversarial Networks

Interpretable Textual Neuron Representations for NLP

Hierarchical interpretations for neural network predictions

Incorporating Context into Language Encoding Models for fMRI

What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties

On the importance of single directions for generalization

Multimodal Explanations: Justifying Decisions and Pointing to the Evidence

Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)

Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation

Detecting Statistical Interactions from Neural Network Weights

Network Dissection: Quantifying Interpretability of Deep Visual Representations

Eye movement-invariant representations in the human visual system

Pointer Sentinel Mixture Models

Deep learning for computational biology

European Union Regulations on Algorithmic Decision-Making and a "Right to Explanation"