Structure-aware protein self-supervised learning (2022-04-06T00:00:00.000000Z)

TL;DR

A novel structure-aware protein self-supervised learning method to effectively capture structural information of proteins via a novel pseudo bi-level optimization scheme using a graph neural network model pretrained on protein sequences.

Abstract

Abstract Motivation Protein representation learning methods have shown great potential to many downstream tasks in biological applications. A few recent studies have demonstrated that the self-supervised learning is a promising solution to addressing insufficient labels of proteins, which is a major obstacle to effective protein representation learning. However, existing protein representation learning is usually pretrained on protein sequences without considering the important protein structural information. Results In this work, we propose a novel structure-aware protein self-supervised learning method to effectively capture structural information of proteins. In particular, a graph neural network model is pretrained to preserve the protein structural information with self-supervised tasks from a pairwise residue distance perspective and a dihedral angle perspective, respectively. Furthermore, we propose to leverage the available protein language model pretrained on protein sequences to enhance the self-supervised learning. Specifically, we identify the relation between the sequential information in the protein language model and the structural information in the specially designed graph neural network model via a novel pseudo bi-level optimization scheme. We conduct experiments on three downstream tasks: the binary classification into membrane/non-membrane proteins, the location classification into 10 cellular compartments, and the enzyme-catalyzed reaction classification into 384 EC numbers, and these experiments verify the effectiveness of our proposed method. Availability and implementation The Alphafold2 database is available in https://alphafold.ebi.ac.uk/. The PDB files are available in https://www.rcsb.org/. The downstream tasks are available in https://github.com/phermosilla/IEConv\_proteins/tree/master/Datasets. The code of the proposed method is available in https://github.com/GGchen1997/STEPS_Bioinformatics.

References41 items

Bidirectional Learning for Offline Model-based Biological Sequence Design

Bidirectional Learning for Offline Infinite-width Model-based Optimization

Gradient-based Bi-level Optimization for Deep Learning: A Survey

Unbiased Implicit Feedback via Bi-level Optimization

Multi-Scale Representation Learning on Proteins

Protein Representation Learning by Geometric Structure Pretraining

Generalized Data Weighting via Class-level Gradient Manipulation

Geometric Graph Representation Learning on Protein Structure Prediction

Highly accurate protein structure prediction with AlphaFold

Learning the protein language: Evolution, structure, and function.

Geometry-enhanced molecular representation learning for property prediction

Structure-based protein function prediction using graph convolutional networks

Transformer protein language models are unsupervised structure learners

ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing

BERTology Meets Biology: Interpreting Attention in Protein Language Models

Meta-Learning in Neural Networks: A Survey

Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping

Revolutionary cryo-EM is taking over structural biology

Evaluating Protein Transfer Learning with TAPE

Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

Learning protein sequence embeddings using information from structure

Machine-learning-guided directed evolution for protein engineering

Dataset Distillation

How Powerful are Graph Neural Networks?

End-to-End Learning on 3D Protein Structure for Interface Prediction

DeepLoc: prediction of protein subcellular localization using deep learning

DeepSF: deep convolutional neural network for mapping protein sequences to folds

f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization

Convolutional LSTM Networks for Subcellular Localization of Proteins

A large-scale evaluation of computational protein function prediction

Four Distances between Pairs of Amino Acids Provide a Precise Description of their Interaction

Principles that govern the folding of protein chains.

On Pre-training Language Model for Antibody

Protein Representation Learning via Knowledge Enhanced Primary Structure Reasoning

Nature Machine Intelligence

Intrinsic-Extrinsic Convolution and Pooling for Learning on 3D Protein Structures

In KDD

and Stefanie Jegelka

Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros

In AICoB

Electronic Reprint Biological Crystallography the Protein Data Bank Biological Crystallography the Protein Data Bank