Dual-view Molecule Pre-training (2021-06-17T00:00:00.000000Z)

TL;DR

This work proposes to leverage both the representations and design a new pre-training algorithm, dual-view molecule pre- training (briefly, DMP), that can effectively combine the strengths of both types of molecule representations.

Abstract

Inspired by its success in natural language processing and computer vision, pre-training has attracted substantial attention in cheminformatics and bioinformatics, especially for molecule based tasks. A molecule can be represented by either a graph (where atoms are connected by bonds) or a SMILES sequence (where depth-first-search is applied to the molecular graph with specific rules). Existing works on molecule pre-training use either graph representations only or SMILES representations only. In this work, we propose to leverage both the representations and design a new pre-training algorithm, dual-view molecule pre-training (briefly, DMP), that can effectively combine the strengths of both types of molecule representations. The model of DMP consists of two branches: a Transformer branch that takes the SMILES sequence of a molecule as input, and a GNN branch that takes a molecular graph as input. The training of DMP contains three tasks: (1) predicting masked tokens in a SMILES sequence by the Transformer branch, (2) predicting masked atoms in a molecular graph by the GNN branch, and (3) maximizing the consistency between the two high-level representations output by the Transformer and GNN branches separately. After pre-training, we can use either the Transformer branch (this one is recommended according to empirical results), the GNN branch, or both for downstream tasks. DMP is tested on nine molecular property prediction tasks and achieves state-of-the-art performances on seven of them. Furthermore, we test DMP on three retrosynthesis tasks and achieve state-of-the-art results on them.

Authors

Tao Qin

14 papers

Tie-Yan Liu

17 papers

Wen-gang Zhou

18 papers

TL;DR

Abstract

Authors

References63 items

The Bottleneck

Transformer Neural Network for Structure Constrained Molecular Optimization

IOT: Instance-wise Layer Reordering for Transformer Structures

Learn molecular representations from large-scale unlabeled molecules for drug discovery

Molecular representation learning with language models and domain-relevant auxiliary tasks

PubChem in 2021: new data content and improved web interfaces

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction

Transfer learning enables the molecular transformer to predict regio- and stereoselective reactions on carbohydrates

Molecular representations in AI-driven drug discovery: a review and practical guide

Using Domain-Specific Fingerprints Generated Through Neural Networks to Enhance Ligand-Based Virtual Screening

Self-Supervised Graph Transformer on Large-Scale Molecular Data

DeeperGCN: All You Need to Train Deeper GCNs

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

RetroXpert: Decompose Retrosynthesis Prediction like a Chemist

Contrastive Multi-View Representation Learning on Graphs

Multi-View Graph Neural Networks for Molecular Property Prediction

A Graph to Graphs Framework for Retrosynthesis Prediction

State-of-the-art augmented NLP transformer models for direct and single-step retrosynthesis

Molecule Attention Transformer

Incorporating BERT into Neural Machine Translation

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

A Simple Framework for Contrastive Learning of Visual Representations

Retrosynthesis Prediction with Conditional Graph Logic Network

Momentum Contrast for Unsupervised Visual Representation Learning

SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery

Depth-Adaptive Transformer

SMILES-BERT: Large Scale Unsupervised Pre-Training for Molecular Property Prediction

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Molecular Property Prediction: A Multilevel Quantum Interactions Modeling Perspective

Evaluating Protein Transfer Learning with TAPE

Strategies for Pre-training Graph Neural Networks

Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

Analyzing Learned Molecular Representations for Property Prediction

Molecular Transformer: A Model for Uncertainty-Calibrated Chemical Reaction Prediction

N-Gram Graph: Simple Unsupervised Representation for Graphs, with Applications to Molecules

Speech-Transformer: A No-Recurrence Sequence-to-Sequence Model for Speech Recognition

Image Transformer

Computer-Assisted Retrosynthesis Based on Molecular Similarity

Attention is All you Need

Retrosynthetic Reaction Prediction Using Neural Sequence-to-Sequence Models

MoleculeNet: a benchmark for molecular machine learning

Semi-Supervised Classification with Graph Convolutional Networks

Communication: Understanding molecular representations in machine learning: The role of uniqueness and target similarity.

Computational Exploration of Molecular Scaffolds in Medicinal Chemistry.

Deep Residual Learning for Image Recognition

Convolutional Networks on Graphs for Learning Molecular Fingerprints

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Adam: A Method for Stochastic Optimization

Efficient Estimation of Word Representations in Vector Space

Extended-Connectivity Fingerprints

Dimensionality Reduction by Learning an Invariant Mapping

Reoptimization of MDL Keys for Use in Drug Discovery

Support-Vector Networks

SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules

A Cluster Separation Measure

MolCLR: Molecular Contrastive Learning of Representations via Graph Neural Networks

MoLGNN: Self-Supervised Motif Learning Graph Neural Network for Drug Discovery

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Visualizing Data using t-SNE

ZINC - A Free Database of Commercially Available Compounds for Virtual Screening

Random decision forests

Fast-speech: Fast, robust and controllable text to speech

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names