BioT5+: Towards Generalized Biological Understanding with IUPAC Integration and Multi-task Tuning (2024-02-27T00:00:00.000000Z)

TL;DR

BioT5+ is introduced, an extension of the BioT5 framework, tailored to enhance biological research and drug discovery, and stands out for its ability to capture intricate relationships in biological data, thereby contributing significantly to bioinformatics and computational biology.

Abstract

Recent research trends in computational biology have increasingly focused on integrating text and bio-entity modeling, especially in the context of molecules and proteins. However, previous efforts like BioT5 faced challenges in generalizing across diverse tasks and lacked a nuanced understanding of molecular structures, particularly in their textual representations (e.g., IUPAC). This paper introduces BioT5+, an extension of the BioT5 framework, tailored to enhance biological research and drug discovery. BioT5+ incorporates several novel features: integration of IUPAC names for molecular understanding, inclusion of extensive bio-text and molecule data from sources like bioRxiv and PubChem, the multi-task instruction tuning for generality across tasks, and a numerical tokenization technique for improved processing of numerical data. These enhancements allow BioT5+ to bridge the gap between molecular representations and their textual descriptions, providing a more holistic understanding of biological entities, and largely improving the grounded reasoning of bio-text and bio-sequences. The model is pre-trained and fine-tuned with a large number of experiments, including \emph{3 types of problems (classification, regression, generation), 15 kinds of tasks, and 21 total benchmark datasets}, demonstrating the remarkable performance and state-of-the-art results in most cases. BioT5+ stands out for its ability to capture intricate relationships in biological data, thereby contributing significantly to bioinformatics and computational biology. Our code is available at \url{https://github.com/QizhiPei/BioT5}.

Authors

References89 items

Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Survey

InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discovery

The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4

MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter

BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations

BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine

GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Text

Prot2Text: Multimodal Protein's Function Generation with GNNs and Transformers

Llama 2: Open Foundation and Fine-Tuned Chat Models

Empowering Molecule Discovery for Molecule-Caption Translation With Large Language Models: A ChatGPT Perspective

MolFM: A Multimodal Molecular Foundation Model

Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks

MolXPT: Wrapping Molecules with Text for Generative Pre-training

LLaMA: Open and Efficient Foundation Language Models

Multilingual translation for zero-shot biomedical classification using BioTranslator

A text-guided protein design framework

Unifying Molecular and Textual Representations via Multi-task Language Modelling

Galactica: A Large Language Model for Science

GLM-130B: An Open Bilingual Pre-trained Model

A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language

Interpretable bilinear attention network with domain adaptation improves drug–target prediction

PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding

Translation between Molecules and Natural Language

A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals

BERN2: an advanced neural biomedical named entity recognition and normalization tool

Pre-training Molecular Graph Representation with 3D Geometry

Motif-based Graph Self-Supervised Learning for Molecular Property Prediction

Machine Learning in Drug Discovery: A Review

Geometry-enhanced molecular representation learning for property prediction

Investigating the Limitations of Transformers with Simple Arithmetic Tasks

STOUT: SMILES to IUPAC names using neural machine translation

ZINC20 - A Free Ultralarge-Scale Chemical Database for Ligand Discovery

Graph Contrastive Learning with Augmentations

ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction

PubMed 2.0

Language Models are Few-Shot Learners

TransformerCPI: improving compound-protein interaction prediction by sequence-based deep learning with self-attention mechanism and label reversal experiments

MolTrans: Molecular Interaction Transformer for drug–target interaction prediction

bioRxiv: the preprint server for biology

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Self-referencing embedded strings (SELFIES): A 100% robust molecular string representation

Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

DeepConv-DTI: Prediction of drug-target interactions via deep learning with convolution on protein sequences

PubChem 2019 update: improved access to chemical data

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

Fréchet ChemNet Distance: A Metric for Generative Models for Molecules in Drug Discovery

Decoupled Weight Decay Regularization

Attention is All you Need

MoleculeNet: a benchmark for molecular machine learning

Deep Residual Learning for Image Recognition

An Introduction to Convolutional Neural Networks

Get Your Atoms in Order - An Open-Source Implementation of a Novel and Robust Molecular Canonicalization Algorithm

Harnessing Computational Biology for Exact Linear B-Cell Epitope Prediction: A Novel Amino Acid Composition-Based Feature Descriptor

Improving compound–protein interaction prediction by building up highly credible negative samples

PubMed: The Bibliographic Database

Extended-Connectivity Fingerprints

Levenshtein Distance: Information theory, Computer science, String (computer science), String metric, Damerau?Levenshtein distance, Spell checker, Hamming distance

Prediction of drug–target interaction networks from the integration of chemical and genomic spaces

Detection of IUPAC and IUPAC-like chemical names

BindingDB: a web-accessible database of experimentally determined protein–ligand binding affinities

METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments

ROUGE: A Package for Automatic Evaluation of Summaries

Reoptimization of MDL Keys for Use in Drug Discovery

Bleu: a Method for Automatic Evaluation of Machine Translation

Prediction of Membrane Protein Types Based on the Hydrophobic Index of Amino Acids

Long Short-Term Memory

Support-Vector Networks

SMILES. 2. Algorithm for generation of unique SMILES notation

Improved tools for biological sequence comparison.

SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules

Rapid and sensitive protein similarity searches.

Uni-Mol: A Universal 3D Molecular Representation Learning Framework

The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023

International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA

Text2Mol: Cross-Modal Molecule Retrieval with Natural Language Queries

Prottrans: Toward understanding the language of life through self-supervised learning

Rdkit: Open-source cheminfor-matics software

GraphDTA: Predicting drug-target binding affinity with graph neural networks

Biosnap datasets: Stanford biomedical network dataset collection

Random decision forests

2023. Mol-instructions: A large-scale biomolecular instruction dataset for large language models

2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

2023a. Empowering ai drug discovery with explicit and implicit knowledge

2023. Stanford alpaca: An instruction-following llama model

2023a. Baize: An open-source chat model with parameter-efficient tuning on self-chat data

2022. Molecular contrastive learning of representations via graph neural networks

BIOINFORMATICS ORIGINAL PAPER doi:10.1093/bioinformatics/btm098 Databases and ontologies UniRef: comprehensive and non-redundant UniProt reference

2023. GPT-4 technical report

2022. Language models of protein sequences at the scale of evolution enable accurate structure prediction