GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Text (2023-08-14T00:00:00.000000Z)

TL;DR

GIT-Mol is introduced, a multi-modal large language model that integrates the Graph, Image, and Text information and proposes GIT-Former, a novel architecture that is capable of aligning all modalities into a unified latent space.

Authors

Peng Liu

5 papers

Yiming Ren

1 papers

Zhixiang Ren

1 papers

TL;DR

Authors

References63 items

Geometry-enhanced pretraining on interatomic potentials

DrugGPT: A GPT-based Strategy for Designing Potential Ligands Targeting Specific Proteins

Empowering Molecule Discovery for Molecule-Caption Translation With Large Language Models: A ChatGPT Perspective

X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages

MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

Visual Instruction Tuning

LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention

One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale

Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models

PaLM-E: An Embodied Multimodal Language Model

Language Is Not All You Need: Aligning Perception with Language Models

LLaMA: Open and Efficient Foundation Language Models

BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

Multi-modal molecule structure–text model for text-based retrieval and editing

Multi-modal chemical information reconstruction from images and texts for exploring the near-drug space

Foundation Transformers

BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining

A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language

Diffusion Models: A Comprehensive Survey of Methods and Applications

Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

SwinOCSR: end-to-end optical chemical structure recognition using a Swin Transformer

GraphMAE: Self-Supervised Masked Graph Autoencoders

A Generalist Agent

Flamingo: a Visual Language Model for Few-Shot Learning

Translation between Molecules and Natural Language

Hierarchical Text-Conditional Image Generation with CLIP Latents

Generative models for molecular discovery: Recent advances and challenges

A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals

Chain of Thought Prompting Elicits Reasoning in Large Language Models

VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts

Pre-training Molecular Graph Representation with 3D Geometry

From Show to Tell: A Survey on Deep Learning-Based Image Captioning

LoRA: Low-Rank Adaptation of Large Language Models

DECIMER 1.0: deep learning for chemical image recognition using transformers

The Power of Scale for Parameter-Efficient Prompt Tuning

Learning Transferable Visual Models From Natural Language Supervision

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

GPT-3: Its Nature, Scope, Limits, and Consequences

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

An open source chemical structure curation pipeline using RDKit

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

SciBERT: A Pretrained Language Model for Scientific Text

PubChem 2019 update: improved access to chemical data

Attention is All you Need

MoleculeNet: a benchmark for molecular machine learning

Counting on natural products for drug design.

Deep Residual Learning for Image Recognition

ChEBI in 2016: Improved services and an expanding collection of metabolites

Optical Structure Recognition Software To Recover Chemical Information: OSRA, An Open Source Solution

SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules

Uni-Mol: A Universal 3D Molecular Representation Learning Framework

HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face

Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Text2Mol: Cross-Modal Molecule Retrieval with Natural Language Queries

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

MOLVEC: Open source library for chemical structure recognition

Software Environment: • Operating System: Ubuntu 18.04.6 LTS • Python Version: 3

Embedding Visualization

• Parallel acceleration with DeepSpeed's ZeRO stage 2

Fine-tuning Settings • Molecule Translation Task: The task type determines the hyperparameters, including input and output modalities and batch size

Tools and Libraries: • The Accelerate library from Hugging Face

Field of Study

Journal Information

Name

Page

Volume