Galactica: A Large Language Model for Science (2022-11-16T00:00:00.000000Z)

TL;DR

Galactica is introduced: a large language model that can store, combine and reason about scientific knowledge, and sets a new state-of-the-art on downstream tasks such as PubMedQA and MedMCQA dev of 77.6% and 52.9%.

Abstract

Information overload is a major obstacle to scientific progress. The explosive growth in scientific literature and data has made it ever harder to discover useful insights in a large mass of information. Today scientific knowledge is accessed through search engines, but they are unable to organize scientific knowledge alone. In this paper we introduce Galactica: a large language model that can store, combine and reason about scientific knowledge. We train on a large scientific corpus of papers, reference material, knowledge bases and many other sources. We outperform existing models on a range of scientific tasks. On technical knowledge probes such as LaTeX equations, Galactica outperforms the latest GPT-3 by 68.2% versus 49.0%. Galactica also performs well on reasoning, outperforming Chinchilla on mathematical MMLU by 41.3% to 35.7%, and PaLM 540B on MATH with a score of 20.4% versus 8.8%. It also sets a new state-of-the-art on downstream tasks such as PubMedQA and MedMCQA dev of 77.6% and 52.9%. And despite not being trained on a general corpus, Galactica outperforms BLOOM and OPT-175B on BIG-bench. We believe these results demonstrate the potential for language models as a new interface for science. We open source the model for the benefit of the scientific community.

Authors

Marcin Kardas

4 papers

Ross Taylor

3 papers

Robert Stojnic

3 papers

TL;DR

Abstract

Authors

References107 items

Scaling Instruction-Finetuned Language Models

Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them

Few-shot Learning with Retrieval Augmented Language Models

Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?

Solving Quantitative Reasoning Problems with Language Models

ProGen2: Exploring the Boundaries of Protein Language Models

Memory-Based Model Editing at Scale

Large Language Models are Zero-Shot Reasoners

Scaling Laws and Interpretability of Learning from Repeated Data

OPT: Open Pre-trained Transformer Language Models

Flamingo: a Visual Language Model for Few-Shot Learning

GPT-NeoX-20B: An Open-Source Autoregressive Language Model

Training Compute-Optimal Large Language Models

LinkBERT: Pretraining Language Models with Document Links

MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering

Impact of Pretraining Term Frequencies on Few-Shot Reasoning

Chain of Thought Prompting Elicits Reasoning in Large Language Models

LaMDA: Language Models for Dialog Applications

Improving language models by retrieving from trillions of tokens

ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning

DOCKSTRING: Easy Molecular Docking Yields Better Benchmarks for Ligand Design

Training Verifiers to Solve Math Word Problems

Multitask Prompted Training Enables Zero-Shot Task Generalization

TruthfulQA: Measuring How Models Mimic Human Falsehoods

Finetuned Language Models Are Zero-Shot Learners

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

Transformer-based artificial neural networks for the conversion between chemical notations

Chemformer: a pre-trained transformer for computational chemistry

Highly accurate protein structure prediction with AlphaFold

Deduplicating Training Data Makes Language Models Better

PonderNet: Learning to Ponder

TagRec: Automated Tagging of Questions with Hierarchical Learning Taxonomy

Overview of BioASQ 2021: The ninth BioASQ challenge on Large-Scale Biomedical Semantic Indexing and Question Answering

Societal Biases in Language Generation: Progress and Challenges

A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers

Measuring Mathematical Problem Solving With the MATH Dataset

STOUT: SMILES to IUPAC names using neural machine translation

Bio-Megatron: Larger Biomedical Domain Language Model

CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models

RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models

Measuring Massive Multitask Language Understanding

Language (Technology) is Power: A Critical Survey of “Bias” in NLP

Language Models are Few-Shot Learners

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

UnifiedQA: Crossing Format Boundaries With a Single QA System

TLDR: Extreme Summarization of Scientific Documents

StereoSet: Measuring stereotypical bias in pretrained language models

Scaling Laws for Neural Language Models

SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery

GORC: A large contextual citation graph of academic papers

QASC: A Dataset for Question Answering via Sentence Composition

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

The Woman Worked as a Babysitter: On Biases in Language Generation

Language Models as Knowledge Bases?

PubMedQA: A Dataset for Biomedical Research Question Answering

On Measuring and Mitigating Biased Inferences of Word Embeddings

Reasoning Over Paragraph Effects in Situations

Measuring Bias in Contextualized Word Representations

BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

SciBERT: Pretrained Contextualized Embeddings for Scientific Text

Wizard of Wikipedia: Knowledge-Powered Conversational agents

Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

SciTaiL: A Textual Entailment Dataset from Science Question Answering

Fixing Weight Decay Regularization in Adam

Towards Human-level Machine Reading Comprehension: Reasoning and Inference with Multiple Strategies

Are You Smarter Than a Sixth Grader? Textbook Question Answering for Multimodal Machine Comprehension

Attention is All you Need

MMseqs2: sensitive protein sequence searching for the analysis of massive data sets

Billion-Scale Similarity Search with GPUs

Bag of Tricks for Efficient Text Classification

Gaussian Error Linear Units (GELUs)

BioCreative V CDR task corpus: a resource for chemical disease relation extraction

Adaptive Computation Time for Recurrent Neural Networks

Big Challenges? Big Data …

Frank-Wolfe Bayesian Quadrature: Probabilistic Integration with Theoretical Guarantees