Translation between Molecules and Natural Language (2022-04-25T00:00:00.000000Z)

TL;DR

The results show that MolT5-based models are able to generate outputs, both molecules and captions, which in many cases are high quality, and consider several metrics, including a new cross-modal embedding-based metric, to evaluate the tasks of molecule captioning and text-based molecule generation.

Abstract

We present MolT5 - a self-supervised learning framework for pretraining models on a vast amount of unlabeled natural language text and molecule strings. MolT5 allows for new, useful, and challenging analogs of traditional vision-language tasks, such as molecule captioning and text-based de novo molecule generation (altogether: translation between molecules and language), which we explore for the first time. Since MolT5 pretrains models on single-modal data, it helps overcome the chemistry domain shortcoming of data scarcity. Furthermore, we consider several metrics, including a new cross-modal embedding-based metric, to evaluate the tasks of molecule captioning and text-based molecule generation. Our results show that MolT5-based models are able to generate outputs, both molecules and captions, which in many cases are high quality.

Authors

Carl N. Edwards

4 papers

Heng Ji

2 papers

T. Lai

1 papers

TL;DR

Abstract

Authors

References90 items

Hierarchical Text-Conditional Image Generation with CLIP Latents

Scaling Up Models and Data with t5x and seqio

Unified Deep Learning Model for Multitask Reaction Predictions with Explanation

A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals

CLIP-Event: Connecting Text and Images with Event Structures

Scaling Up Vision-Language Pretraining for Image Captioning

MolGPT: Molecular Generation Using a Transformer-Decoder Model

Chemical-Reaction-Aware Molecule Representation Learning

IMG2SMI: Translating Molecular Structure Images to Simplified Molecular-input Line-entry System

Fine-Grained Chemical Entity Typing with Multimodal Knowledge Representation

Chemformer: a pre-trained transformer for computational chemistry

Automated Chemical Reaction Extraction from Scientific Literature

Utilizing graph machine learning within drug discovery and development

End-to-End Attention-based Image Captioning

Extraction of organic chemistry grammar from unsupervised learning of chemical reactions

Learning Transferable Visual Models From Natural Language Supervision

Zero-Shot Text-to-Image Generation

Transformers in Vision: A Survey

Deep learning and generative methods in cheminformatics and chemical biology: navigating small molecule space intelligently

Molecular representation learning with language models and domain-relevant auxiliary tasks

Inferring experimental procedures from text-based representations of chemical reactions

ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction

Mapping the space of chemical reactions using attention-based neural networks

Cross-media Structured Common Space for Multimedia Event Extraction

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

X-Linear Attention Networks for Image Captioning

Scaling Laws for Neural Language Models

Multilingual Denoising Pre-training for Neural Machine Translation

Automated extraction of chemical synthesis actions from experimental procedures

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

HuggingFace's Transformers: State-of-the-art Natural Language Processing

UNITER: UNiversal Image-TExt Representation Learning

UNITER: Learning UNiversal Image-TExt Representations

VL-BERT: Pre-training of Generic Visual-Linguistic Representations

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Image Captioning: Transforming Objects into Words

How Multilingual is Multilingual BERT?

Self-referencing embedded strings (SELFIES): A 100% robust molecular string representation

A Survey on Biomedical Image Captioning

BERTScore: Evaluating Text Generation with BERT

SciBERT: A Pretrained Language Model for Scientific Text

Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models

A Comprehensive Survey of Deep Learning for Image Captioning

Recent applications of deep learning and machine intelligence on in silico drug discovery: methods, tools and databases

Entity-aware Image Caption Generation

Fréchet ChemNet Distance: A Metric for Generative Models for Molecules in Drug Discovery

Mol2vec: Unsupervised Machine Learning Approach with Chemical Intuition

AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks

On the Automatic Generation of Medical Imaging Reports

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Attention is All you Need

Molecular de-novo design through deep reinforcement learning

StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks

Improved Deep Metric Learning with Multi-class N-pair Loss Objective

Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models

Event Specific Multimodal Pattern Mining for Knowledge Base Construction

Generative Adversarial Text to Image Synthesis

ZINC 15 – Ligand Discovery for Everyone

Get Your Atoms in Order - An Open-Source Implementation of a Novel and Robust Molecular Canonicalization Algorithm

Convolutional Networks on Graphs for Learning Molecular Fingerprints

Preparing a collection of radiology examinations for distribution and retrieval

EventNet: A Large Scale Structured Concept Library for Complex Event Detection in Video

Microsoft COCO Captions: Data Collection and Evaluation Server

Show and tell: A neural image caption generator

On the Properties of Neural Machine Translation: Encoder–Decoder Approaches

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Distributed Representations of Words and Phrases and their Compositionality

Efficient Estimation of Word Representations in Vector Space

Extended-Connectivity Fingerprints

Levenshtein Distance: Information theory, Computer science, String (computer science), String metric, Damerau?Levenshtein distance, Spell checker, Hamming distance

METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments

ROUGE: A Package for Automatic Evaluation of Summaries

Automatic image captioning

Reoptimization of MDL Keys for Use in Drug Discovery

Bleu: a Method for Automatic Evaluation of Machine Translation

SMILES. 2. Algorithm for generation of unique SMILES notation