BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations (2023-10-11T00:00:00.000000Z)

TL;DR

A comprehensive pre-training framework that enriches cross-modal integration in biology with chemical knowledge and natural language associations, and distinguishes between structured and unstructured knowledge, leading to more effective utilization of information.

Abstract

Recent advancements in biological research leverage the integration of molecules, proteins, and natural language to enhance drug discovery. However, current models exhibit several limitations, such as the generation of invalid molecular SMILES, underutilization of contextual information, and equal treatment of structured and unstructured knowledge. To address these issues, we propose $\mathbf{BioT5}$, a comprehensive pre-training framework that enriches cross-modal integration in biology with chemical knowledge and natural language associations. $\mathbf{BioT5}$ utilizes SELFIES for $100%$ robust molecular representations and extracts knowledge from the surrounding context of bio-entities in unstructured biological literature. Furthermore, $\mathbf{BioT5}$ distinguishes between structured and unstructured knowledge, leading to more effective utilization of information. After fine-tuning, BioT5 shows superior performance across a wide range of tasks, demonstrating its strong capability of capturing underlying relations and properties of bio-entities. Our code is available at $\href{https://github.com/QizhiPei/BioT5}{Github}$.

Authors

Jinhua Zhu

5 papers

Lijun Wu

6 papers

Qizhi Pei

2 papers

TL;DR

Abstract

Authors

References100 items

Geometry-enhanced pretraining on interatomic potentials

Empowering Molecule Discovery for Molecule-Caption Translation With Large Language Models: A ChatGPT Perspective

MolXPT: Wrapping Molecules with Text for Generative Pre-training

Multilingual translation for zero-shot biomedical classification using BioTranslator

A text-guided protein design framework

ProtST: Multi-Modality Learning of Protein Sequences and Biomedical Texts

UniProt: the Universal Protein Knowledgebase in 2023

Galactica: A Large Language Model for Science

BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining

A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language

Interpretable bilinear attention network with domain adaptation improves drug–target prediction

ProtGPT2 is a deep unsupervised language model for protein design

PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding

Translation between Molecules and Natural Language

SELFIES and the future of molecular string representations

A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals

BERN2: an advanced neural biomedical named entity recognition and normalization tool

Hierarchical Clustering Split for Low-Bias Evaluation of Drug-Target Interaction Prediction

MolGPT: Molecular Generation Using a Transformer-Decoder Model

Pre-training Molecular Graph Representation with 3D Geometry

Motif-based Graph Self-Supervised Learning for Molecular Property Prediction

Finetuned Language Models Are Zero-Shot Learners

Machine Learning in Drug Discovery: A Review

Geometry-enhanced molecular representation learning for property prediction

Making Pre-trained Language Models Better Few-shot Learners

ZINC20 - A Free Ultralarge-Scale Chemical Database for Ligand Discovery

ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

Self-Supervised Graph Transformer on Large-Scale Molecular Data

Language Models are Few-Shot Learners

TransformerCPI: improving compound-protein interaction prediction by sequence-based deep learning with self-attention mechanism and label reversal experiments

MolTrans: Molecular Interaction Transformer for drug–target interaction prediction

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

PUG-View: programmatic access to chemical annotations integrated in PubChem

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Self-referencing embedded strings (SELFIES): A 100% robust molecular string representation

Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

SciBERT: A Pretrained Language Model for Scientific Text

BioBERT: a pre-trained biomedical language representation model for biomedical text mining

DeepConv-DTI: Prediction of drug-target interactions via deep learning with convolution on protein sequences

DeepSMILES: An Adaptation of SMILES for Use in Machine-Learning of Chemical Structures

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

DeepSol: a deep learning framework for sequence‐based protein solubility prediction

Fréchet ChemNet Distance: A Metric for Generative Models for Molecules in Drug Discovery

Decoupled Weight Decay Regularization

DrugBank 5.0: a major update to the DrugBank database for 2018

DeepLoc: prediction of protein subcellular localization using deep learning

Attention is All you Need

MoleculeNet: a benchmark for molecular machine learning

Clustering huge protein sequence sets in linear time

Semi-Supervised Classification with Graph Convolutional Networks

Deep Residual Learning for Image Recognition

An Introduction to Convolutional Neural Networks

ZINC 15 – Ligand Discovery for Everyone

Get Your Atoms in Order - An Open-Source Implementation of a Novel and Robust Molecular Canonicalization Algorithm

ChEBI in 2016: Improved services and an expanding collection of metabolites

Harnessing Computational Biology for Exact Linear B-Cell Epitope Prediction: A Novel Amino Acid Composition-Based Feature Descriptor

NCBI Viral Genomes Resource

propy: a tool to generate various modes of Chou's PseAAC

PubMed: The Bibliographic Database

InChI - the worldwide chemical structure identifier standard

Large-scale prediction of human protein-protein interactions from amino acid sequence based on latent topic features.

Extended-Connectivity Fingerprints

Levenshtein Distance: Information theory, Computer science, String (computer science), String metric, Damerau?Levenshtein distance, Spell checker, Hamming distance

The Probabilistic Relevance Framework: BM25 and Beyond

Using support vector machine combined with auto covariance to predict protein–protein interactions from protein sequences

BindingDB: a web-accessible database of experimentally determined protein–ligand binding affinities

METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments

ROUGE: A Package for Automatic Evaluation of Summaries

Development of human protein reference database as an initial platform for approaching systems biology in humans.

Reoptimization of MDL Keys for Use in Drug Discovery

Bleu: a Method for Automatic Evaluation of Machine Translation

Medical Subject Headings (MeSH).

Prediction of Membrane Protein Types Based on the Hydrophobic Index of Amino Acids

Unsupervised Data Base Clustering Based on Daylight's Fingerprint and Tanimoto Similarity: A Fast and Automated Way To Cluster Small and Large Data Sets

Long Short-Term Memory