Extending Context Window of Large Language Models via Positional Interpolation (2023-06-27T00:00:00.000000Z)

TL;DR

Position Interpolation linearly down-scales the input position indices to match the original context window size, rather than extrapolating beyond the trained context length which may lead to catastrophically high attention scores that completely ruin the self-attention mechanism.

Abstract

We present Position Interpolation (PI) that extends the context window sizes of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal fine-tuning (within 1000 steps), while demonstrating strong empirical results on various tasks that require long context, including passkey retrieval, language modeling, and long document summarization from LLaMA 7B to 65B. Meanwhile, the extended model by Position Interpolation preserve quality relatively well on tasks within its original context window. To achieve this goal, Position Interpolation linearly down-scales the input position indices to match the original context window size, rather than extrapolating beyond the trained context length which may lead to catastrophically high attention scores that completely ruin the self-attention mechanism. Our theoretical study shows that the upper bound of interpolation is at least $\sim 600 \times$ smaller than that of extrapolation, further demonstrating its stability. Models extended via Position Interpolation retain its original architecture and can reuse most pre-existing optimization and infrastructure.

Authors

Shouyuan Chen

1 papers

Sherman Wong

1 papers

Liangjian Chen

1 papers

TL;DR

Abstract

Authors

References47 items

Landmark Attention: Random-Access Infinite Context Length for Transformers

PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

Learning to Compress Prompts with Gist Tokens

CoLT5: Faster Long-Range Transformers with Conditional Computation

LLaMA: Open and Efficient Foundation Language Models

Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformer

Few-shot Learning with Retrieval Augmented Language Models

Recurrent Memory Transformer

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

OPT: Open Pre-trained Transformer Language Models

Transformer Language Models without Positional Encodings Still Learn Positional Information

Memorizing Transformers

SCROLLS: Standardized CompaRison Over Long Language Sequences

ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

Combiner: Full Attention Transformer with Sparse Computation Cost

LoRA: Low-Rank Adaptation of Large Language Models

RoFormer: Enhanced Transformer with Rotary Position Embedding

Efficient Attentions for Long Document Summarization

The Pile: An 800GB Dataset of Diverse Text for Language Modeling

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Memformer: A Memory-Augmented Transformer for Sequence Modeling

Rethinking Attention with Performers

Big Bird: Transformers for Longer Sequences

Relevance-guided Supervision for OpenQA with ColBERT

Linformer: Self-Attention with Linear Complexity

Dense Passage Retrieval for Open-Domain Question Answering

Longformer: The Long-Document Transformer

REALM: Retrieval-Augmented Language Model Pre-Training

Reformer: The Efficient Transformer

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Compressive Transformers for Long-Range Sequence Modelling

Generating Long Sequences with Sparse Transformers

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

Decoupled Weight Decay Regularization

Attention is All you Need

Together

ROUGE: A Package for Automatic Evaluation of Summaries

Redpajama: An open source recipe to reproduce llama training dataset

Proof-pile, 2022. URL https:// github.com/zhangir-azerbayev/proof-pile

Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length-extrapolatable transformer

on scaling fully sharded

mags = (xfreq.sin().cumsum(dim=1).pow(2) + xfreq.cos plt.plot(mags.sum(dim=1)/d)plt.axhline(1.0,color=’k’, linestyle="--") plt.xlabel("Positional difference $s$") plt.ylabel("$B(s)/d$")

r") for i in range(25,75): plt.axvline(i, color="k

Things i ´m learning while training superhot

length-extrapolatable

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names