Reinforced Self-Training (ReST) for Language Modeling (2023-08-17T00:00:00.000000Z)

TL;DR

The results show that ReST can substantially improve translation quality, as measured by automated metrics and human evaluation on machine translation benchmarks in a compute and sample-efficient manner.

Abstract

Reinforcement learning from human feedback (RLHF) can improve the quality of large language model's (LLM) outputs by aligning them with human preferences. We propose a simple algorithm for aligning LLMs with human preferences inspired by growing batch reinforcement learning (RL), which we call Reinforced Self-Training (ReST). Given an initial LLM policy, ReST produces a dataset by generating samples from the policy, which are then used to improve the LLM policy using offline RL algorithms. ReST is more efficient than typical online RLHF methods because the training dataset is produced offline, which allows data reuse. While ReST is a general approach applicable to all generative learning settings, we focus on its application to machine translation. Our results show that ReST can substantially improve translation quality, as measured by automated metrics and human evaluation on machine translation benchmarks in a compute and sample-efficient manner.

Authors

Orhan Firat

6 papers

Aditya Siddhant

3 papers

Caglar Gulcehre

2 papers

TL;DR

Abstract

Authors

References69 items

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

The Curse of Recursion: Training on Generated Data Makes Models Forget

Impossible Distillation for Paraphrasing and Summarization: How to Make High-quality Lemonade out of Small, Low-quality Model

RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment

Sparks of Artificial General Intelligence: Early experiments with GPT-4

Solving math word problems with process- and outcome-based feedback

Scaling Laws for Reward Model Overoptimization

Improving alignment of dialogue agents via targeted human judgements

Defining and Characterizing Reward Hacking

MAD for Robust Reinforcement Learning in Machine Translation

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

When does return-conditioned supervised learning work for offline reinforcement learning?

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

STaR: Bootstrapping Reasoning With Reasoning

Training language models to follow instructions with human feedback

Competition-level code generation with AlphaCode

Red Teaming Language Models with Language Models

Scaling Language Models: Methods, Analysis & Insights from Training Gopher

Recursively Summarizing Books with Human Feedback

Scaling Laws for Neural Machine Translation

Highly accurate protein structure prediction with AlphaFold

Launchpad: A Programming Model for Distributed Machine Learning Research

Decision Transformer: Reinforcement Learning via Sequence Modeling

Machine Translation Decoding beyond Beam Search

Regularized Behavior Value Estimation

Supervised Seeded Iterated Learning for Interactive Language Learning

COMET: A Neural Framework for MT Evaluation

Learning to summarize from human feedback

Language Models are Few-Shot Learners

D4RL: Datasets for Deep Data-Driven Reinforcement Learning

Self-Training for Unsupervised Neural Machine Translation in Unbalanced Training Data Scenarios

BLEURT: Learning Robust Metrics for Text Generation

Countering Language Drift with Seeded Iterated Learning

Self-Training With Noisy Student Improves ImageNet Classification

Revisiting Self-Training for Neural Sequence Generation

V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control

Self-Imitation Learning

Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

Proximal Policy Optimization Algorithms

Attention is All you Need

Exploiting Source-side Monolingual Data in Neural Machine Translation

Asynchronous Methods for Deep Reinforcement Learning

Iterated learning and the evolution of language

Sequence to Sequence Learning with Neural Networks

Bootstrapping POS-taggers using unlabelled data

Unsupervised Word Sense Disambiguation Rivaling Supervised Methods

Probability of error of some adaptive pattern-recognition machines

A Markovian Decision Process

Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust

Should I Run Offline Reinforcement Learning or Behavioral Cloning?

Beyond Tabula Rasa: Reincarnating Reinforcement Learning

An empirical analysis of compute-optimal large language model training

On Multi-objective Policy Optimization as a Tool for Reinforcement Learning

Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain

2021) we did not use the V-trace as it did not improve over the vanilla advantage

The BVMPO approach is similar to DIME

Expert iteration

When using it in Improve step, we start with a BC policy

Findings of the WMT 2020 Shared Task on Parallel Corpus Filtering and Alignment

The DeepMind Chinese–English Document Translation System at WMT2020

Text Generation by Learning from Demonstrations

The DeepMind JAX Ecosystem, 2020

In each Grow step, we generate 25 candidates for each source sentence in the training

ALVINN, an autonomous land vehicle in a neural network

Report on the 11th IWSLT evaluation campaign

Batch Reinforcement Learning

Reinforced Self-Training ( ReST ) for Language Modeling

do not share or forward externally

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type