ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing (2020-07-12T00:00:00.000000Z)

Abstract

Computational biology and bioinformatics provide vast data gold-mines from protein sequences, ideal for Language Models (LMs) taken from Natural Language Processing (NLP). These LMs reach for new prediction frontiers at low inference costs. Here, we trained two auto-regressive language models (Transformer-XL, XLNet) and two auto-encoder models (Bert, Albert) on data from UniRef and BFD containing up to 393 billion amino acids (words) from 2.1 billion protein sequences (22- and 112-times the entire English Wikipedia). The LMs were trained on the Summit supercomputer at Oak Ridge National Laboratory (ORNL), using 936 nodes (total 5616 GPUs) and one TPU Pod (V3-512 or V3-1024). We validated the advantage of up-scaling LMs to larger models supported by bigger data by predicting secondary structure (3-states: Q3=76-84, 8-states: Q8=65-73), sub-cellular localization for 10 cellular compartments (Q10=74) and whether a protein is membrane-bound or water-soluble (Q2=89). Dimensionality reduction revealed that the LM-embeddings from unlabeled data (only protein sequences) captured important biophysical properties governing protein shape. This implied learning some of the grammar of the language of life realized in protein sequences. The successful up-scaling of protein LMs through HPC to larger data sets slightly reduced the gap between models trained on evolutionary information and LMs.

Authors

References113 items

Embeddings from protein language models predict conservation and variant effects

Light attention predicts protein location from the language of life

PredictProtein - Predicting Protein Structure and Function for 29 Years

MSA Transformer

Transformer protein language models are unsupervised structure learners

Embeddings from deep learning transfer GO annotations beyond homology

Big Bird: Transformers for Longer Sequences

Transforming the Language of Life: Transformer Neural Networks for Protein Prediction Tasks

Language Models are Few-Shot Learners

Language modelling for biological sequences – curated datasets and baselines

ProGen: Language Modeling for Protein Generation

Reformer: The Efficient Transformer

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Modeling aspects of the language of life through transfer-learning protein sequences

Pre-Training of Deep Bidirectional Protein Sequence Representations With Structural Information

Improved protein structure prediction using predicted interresidue orientations

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Unified rational protein engineering with sequence-based deep representation learning

ZeRO: Memory Optimization Towards Training A Trillion Parameter Models

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Attention Interpretability Across NLP Tasks

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

On the Variance of the Adaptive Learning Rate and Beyond

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Evaluating Protein Transfer Learning with TAPE

A Multiscale Visualization of Attention in the Transformer Model

Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

Generating Long Sequences with Sparse Transformers

Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

Generative Models for Graph-Based Protein Design

Learning protein sequence embeddings using information from structure

ElemCor: accurate data analysis and enrichment calculation for high-resolution LC-MS stable isotope labeling experiments

ProteinNet: a standardized data set for machine learning of protein structure

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

SCOPe: classification of large macromolecular structures in the structural classification of proteins—extended database

UniProt: a worldwide hub of protein knowledge

Dark Proteins Important for Cellular Function

Evolutionary couplings and sequence variation effect predict protein binding sites

A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation

Protein-level assembly increases protein sequence recovery from metagenomic samples manyfold

TFLMS: Large Model Support in TensorFlow by Graph Rewriting

Probabilistic variable-length segmentation of protein sequences for discriminative motif discovery (DiMotif) and sequence embedding (ProtVecX)

NetSurfP-2.0: improved prediction of protein structural features by integrated deep learning

Promotion of virus assembly and organization by the measles virus matrix protein

Assessment of hard target modeling in CASP12 reveals an emerging role of alignment‐based contact prediction methods

Horovod: fast and easy distributed deep learning in TensorFlow

Deep Contextualized Word Representations

End-to-end differentiable learning of protein structure

Evolutionary adaptations to new environments generally reverse plastic phenotypic changes

Universal Language Model Fine-tuning for Text Classification

DeepLoc: prediction of protein subcellular localization using deep learning

Quantitative firing pattern phenotyping of hippocampal neuron types

MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets

Capturing non‐local interactions by long short‐term memory bidirectional recurrent neural networks for improving prediction of protein secondary structure, backbone angles, contact numbers and solvent accessibility

PowerAI DDL

Attention is All you Need

In-datacenter performance analysis of a tensor processing unit

Clustering huge protein sequence sets in linear time

Sixty-five years of the long march in protein secondary structure prediction: the final stretch?

FastText.zip: Compressing text classification models

TMSEG: Novel prediction of transmembrane helices

Extreme Scale-out SuperMUC Phase 2 - lessons learned

Announcing Supercomputer Summit

RaptorX-Property: a web server for protein structure property prediction

TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

Protein Secondary Structure Prediction Using Deep Convolutional Neural Fields

Unexpected features of the dark proteome

Accurate contact predictions using covariation techniques and machine learning

JPred4: a protein secondary structure prediction server

UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches

GloVe: Global Vectors for Word Representation

Neural Machine Translation by Jointly Learning to Align and Translate

One billion word benchmark for measuring progress in statistical language modeling

Distributed Representations of Words and Phrases and their Compositionality

LocTree2 predicts localization for all domains of life

Approximate Nearest Neighbor: Towards Removing the Curse of Dimensionality

Three-Dimensional Structures of Membrane Proteins from Genomic Sequencing

Hydrophobic forces and the length limit of foldable protein domains

Protein 3D Structure Computed from Evolutionary Sequence Variation

What is the best multi-stage architecture for object recognition?

PSI-2: structural genomics to cover protein domain family space.

NVIDIA cuda software and gpu parallel computing architecture

Improving fold recognition without folds.

Protein flexibility and intrinsic disorder

PISCES: a protein sequence culling server

Enhanced protein domain discovery by using language modeling techniques from speech recognition

Evaluation and improvement of multiple sequence methods for protein secondary structure prediction

High-resolution structures of variant Zif268-DNA complexes: implications for understanding zinc finger-DNA recognition.

Pitfalls of protein sequence analysis.

Epitope Mapping by Label-Free Biomolecular Interaction Analysis

Combining evolutionary information and neural networks to predict protein secondary structure

Improved prediction of protein secondary structure by use of sequence profiles and neural networks.

Prediction of protein secondary structure at better than 70% accuracy.

Amino acid substitution matrices from protein blocks.

The SWISS-PROT protein sequence data bank.

Enzyme Nomenclature

Studies on the reduction and re-formation of protein disulfide bonds.

Supplementary Material of Optimal Gradient Checkpoint Search for Arbitrary Computation Graphs

100

Continuous

101

Human Mutation

102

Electra: Pre-training text encoders as discriminators rather than generators

103

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

104

Anaconda software distribution

105

Visualizing Data using t-SNE

106

Approximate Nearest Neighbors

107

The ENZYME database in 2000

108

The Protein Data Bank

109

Bridging the protein sequence-structure gap by structure predictions.

110

PHD: predicting one-dimensional protein structure by profile-based neural networks.

111

Characteristics of Sentence Length in Running Text

112

Enzyme nomenclature 1992. Recommendations of the Nomenclature Committee of the International Union of Biochemistry and Molecular Biology on the Nomenclature and Classification of Enzymes.

113

Press release announcing Supercomputer Fugaku. Technical report, RIKEN