Generative power of a protein language model trained on multiple sequence alignments (2022-04-14T00:00:00.000000Z)

TL;DR

This work proposes and test an iterative method that directly employs the masked language modeling objective to generate sequences using MSA Transformer and demonstrates that the resulting sequences score as well as natural sequences, for homology, coevolution and structure-based measures.

Abstract

Computational models starting from large ensembles of evolutionarily related protein sequences capture a representation of protein families and learn constraints associated to protein structure and function. They thus open the possibility for generating novel sequences belonging to protein families. Protein language models trained on multiple sequence alignments, such as MSA Transformer, are highly attractive candidates to this end. We propose and test an iterative method that directly employs the masked language modeling objective to generate sequences using MSA Transformer. We demonstrate that the resulting sequences score as well as natural sequences, for homology, coevolution and structure-based measures. For large protein families, our synthetic sequences have similar or better properties compared to sequences generated by Potts models, including experimentally-validated ones. Moreover, for small protein families, our generation method based on MSA Transformer outperforms Potts models. Our method also more accurately reproduces the higher-order statistics and the distribution of sequences in sequence space of natural data than Potts models. MSA Transformer is thus a strong candidate for protein sequence generation and protein design.

Authors

Damiano Sgarbossa

1 papers

Umberto Lupo

1 papers

Anne-Florence Bitbol

1 papers

TL;DR

Abstract

Authors

References88 items

ProtGPT2 is a deep unsupervised language model for protein design

Efficient evolution of human antibodies from general protein language models and sequence information alone

Protein language models trained on multiple sequence alignments learn phylogenetic relationships

Extracting phylogenetic dimensions of coevolution reveals hidden functional signals

Interpreting Potts and Transformer Protein Models Through the Lens of Simplified Attention

Correlations from structure and phylogeny combine constructively in the inference of protein partners from sequences

Single-sequence protein structure prediction using language models from deep learning

Faculty Opinions recommendation of Accurate prediction of protein structures and interactions using a three-track neural network.

Deep neural language modeling enables functional protein generation across families

Highly accurate protein structure prediction with AlphaFold

Language models enable zero-shot prediction of the effects of mutations on protein function

Accurate prediction of protein structures and interactions using a 3-track neural network

Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis-Hastings

On the effect of phylogenetic correlations in coevolution-based contact prediction in proteins

Protein design and variant prediction using autoregressive generative models

Large-scale discovery of protein interactions at residue resolution using co-evolution calculated from genomic sequences

MSA Transformer

Improving sequence-based modeling of protein families using secondary-structure quality assessment

Generating novel protein sequences using Gibbs sampling of masked language models

Single Layers of Attention Suffice to Predict Protein Contacts

Transformer protein language models are unsupervised structure learners

The generative capacity of probabilistic protein sequence models

Pfam: The protein families database in 2021

What are “tippy” and “stemmy” phylogenies? Resolving a phylogenetic terminological tangle

De novo protein design by deep network hallucination

An evolution-based model for designing chorismate mutase enzymes

ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing

BERTology Meets Biology: Interpreting Attention in Protein Language Models

Generating functional protein variants with variational autoencoders

ProGen: Language Modeling for Protein Generation

Epistatic contributions promote the unification of incompatible models of neutral molecular evolution

Toward Inferring Potts Models for Phylogenetically Correlated Sequence Data

Unified rational protein engineering with sequence-based deep representation learning

Protein interaction networks revealed by proteome coevolution

Phylogenetic correlations can suffice to infer protein partners from sequences

Using deep learning to annotate the protein universe

Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Information

BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model

Synthetic protein alignments by CCMgen quantify noise in residue-residue contact prediction

Learning protein constitutive motifs from sequence data

How Pairwise Coevolutionary Models Capture the Collective Residue Variability in Proteins?

Coevolutionary Landscape of Kinase Family Proteins: Sequence Probabilities and Functional Motifs.

Power law tails in phylogenetic systems

Directed Evolution: Bringing New Chemistry to Life

Global analysis of protein folding using massively parallel design, synthesis, and testing

Attention is All you Need

Inverse statistical physics of protein sequences: a key issues review

Connecting the Sequence-Space of Bacterial Signaling Proteins to Phenotypes Using Coevolutionary Landscapes

A Metric on Phylogenetic Tree Shapes

Simultaneous identification of specifically interacting paralogs and interprotein contacts by direct coupling analysis

Inferring interaction partners from protein sequences

Quantifying Synergistic Information Using Intermediate Stochastic Variables

Coevolutionary Landscape Inference and the Context-Dependence of Mutations in Beta-Lactamase TEM-1

Understanding Interdependency Through Complex Information Sharing

Characterizing and comparing phylogenies from their Laplacian spectrum

Large-Scale Conformational Transitions and Dimerization Are Encoded in the Amino-Acid Sequences of Hsp70 Chaperones

Evolvability as a Function of Purifying Selection in TEM-1 β-Lactamase

Neural Machine Translation by Jointly Learning to Align and Translate

Synergy, redundancy, and multivariate information measures: an experimentalist’s perspective

Toward rationally redesigning bacterial two-component signaling systems using coevolutionary information

Coevolutionary signals across protein lineages help capture multiple protein conformations

Predicting Functionally Informative Mutations in Escherichia coli BamA Using Evolutionary Covariance Analysis

Improved contact prediction in proteins: using pseudolikelihoods to infer Potts models.

Biophysics: Searching for Principles

The spatial architecture of protein function and adaptation

Genomics-aided structure prediction

Protein 3D Structure Computed from Evolutionary Sequence Variation

Direct-coupling analysis of residue coevolution captures native contacts across many protein families

Nonnegative Decomposition of Multivariate Information

FastTree 2 – Approximately Maximum-Likelihood Trees for Large Alignments

Identification of direct residue contacts in protein–protein interaction by message passing

Mutual information without the influence of phylogeny or entropy dramatically improves residue contact prediction

Rediscovering the power of pairwise interactions

Evolutionary information for specifying a protein fold

Design of a Novel Globular Protein Fold with Atomic-Level Accuracy