InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discovery (2023-11-27T00:00:00.000000Z)

TL;DR

InstructMol, a multi-modal LLM, effectively aligns molecular structures with natural language via an instruction-tuning approach, utilizing a two-stage training strategy that adeptly combines limited domain-specific data with molecular and textual information.

Abstract

The rapid evolution of artificial intelligence in drug discovery encounters challenges with generalization and extensive training, yet Large Language Models (LLMs) offer promise in reshaping interactions with complex molecular data. Our novel contribution, InstructMol, a multi-modal LLM, effectively aligns molecular structures with natural language via an instruction-tuning approach, utilizing a two-stage training strategy that adeptly combines limited domain-specific data with molecular and textual information. InstructMol showcases substantial performance improvements in drug discovery-related molecular tasks, surpassing leading LLMs and significantly reducing the gap with specialized models, thereby establishing a robust foundation for a versatile and dependable drug discovery assistant.

Authors

He Cao

2 papers

Zijing Liu

1 papers

Xingyu Lu

1 papers

TL;DR

Abstract

Authors

References100 items

SMILES-Prompting: A Novel Approach to LLM Jailbreak Attacks in Chemical Synthesis

A bioactivity foundation model using pairwise meta-learning

PRESTO: Progressive Pretraining Enhances Synthetic Chemistry Outcomes

AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting

MoleculeQA: A Dataset to Evaluate Factual Accuracy in Molecular Comprehension

CogVLM: Visual Expert for Pretrained Language Models

GraphGPT: Graph Instruction Tuning for Large Language Models

MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter

Graph Foundation Models: Concepts, Opportunities and Challenges

Improved Baselines with Visual Instruction Tuning

Geometry-enhanced pretraining on interatomic potentials

BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine

GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Text

Llama 2: Open Foundation and Fine-Tuned Chat Models

A survey on multimodal large language models

Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models

Empowering Molecule Discovery for Molecule-Caption Translation With Large Language Models: A ChatGPT Perspective

MolFM: A Multimodal Molecular Foundation Model

LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day

GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction

Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models

DetGPT: Detect What You Need via Reasoning

DrugChat: Towards Enabling ChatGPT-Like Capabilities on Drug Molecule Graphs

MolXPT: Wrapping Molecules with Text for Generative Pre-training

VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks

PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering

PaLM 2 Technical Report

InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages

mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality

PMC-LLaMA: Towards Building Open-source Language Models for Medicine

MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

Visual Instruction Tuning

GPT-4 Technical Report

LLaMA: Open and Efficient Foundation Language Models

BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

Unifying Molecular and Textual Representations via Multi-task Language Modelling

Multi-modal molecule structure–text model for text-based retrieval and editing

Deep learning in drug discovery: an integrative review and future challenges

Galactica: A Large Language Model for Science

PubChem 2023 update

GLM-130B: An Open Bilingual Pre-trained Model

A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language

ChemBERTa-2: Towards Chemical Foundation Models

Flamingo: a Visual Language Model for Few-Shot Learning

Translation between Molecules and Natural Language

Unified Deep Learning Model for Multitask Reaction Predictions with Explanation

Training language models to follow instructions with human feedback

A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals

Retroformer: Pushing the Limits of Interpretable End-to-end Retrosynthesis Transformer

An Empirical Study of Training End-to-End Vision-and-Language Transformers

3D Infomax improves GNNs for Molecular Property Prediction

Pre-training Molecular Graph Representation with 3D Geometry

Finetuned Language Models Are Zero-Shot Learners

Comprehensive Survey of Recent Drug Discovery Using Deep Learning

Chemformer: a pre-trained transformer for computational chemistry

LoRA: Low-Rank Adaptation of Large Language Models

Learning Transferable Visual Models From Natural Language Supervision

Defining and Exploring Chemical Spaces

Graph Contrastive Learning with Augmentations

ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction

Measuring Massive Multitask Language Understanding

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

Self-Supervised Graph Transformer on Large-Scale Molecular Data

Language Models are Few-Shot Learners

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

SMILES-BERT: Large Scale Unsupervised Pre-Training for Molecular Property Prediction

PUG-View: programmatic access to chemical annotations integrated in PubChem

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Self-referencing embedded strings (SELFIES): A 100% robust molecular string representation

Strategies for Pre-training Graph Neural Networks

SciBERT: A Pretrained Language Model for Scientific Text

BioBERT: a pre-trained biomedical language representation model for biomedical text mining

Molecular Transformer: A Model for Uncertainty-Calibrated Chemical Reaction Prediction

How Powerful are Graph Neural Networks?

Recent applications of deep learning and machine intelligence on in silico drug discovery: methods, tools and databases