Molecular representation learning with language models and domain-relevant auxiliary tasks (2020-11-26T00:00:00.000000Z)

TL;DR

The Transformer architecture is applied, specifically BERT, to learn flexible and high quality molecular representations for drug discovery problems, and molecular representations learnt by the model `MolBert' improve upon the current state of the art on the benchmark datasets.

Abstract

We apply a Transformer architecture, specifically BERT, to learn flexible and high quality molecular representations for drug discovery problems. We study the impact of using different combinations of self-supervised tasks for pre-training, and present our results for the established Virtual Screening and QSAR benchmarks. We show that: i) The selection of appropriate self-supervised task(s) for pre-training has a significant impact on performance in subsequent downstream tasks such as Virtual Screening. ii) Using auxiliary tasks with more domain relevance for Chemistry, such as learning to predict calculated molecular properties, increases the fidelity of our learnt representations. iii) Finally, we show that molecular representations learnt by our model `MolBert' improve upon the current state of the art on the benchmark datasets.

Authors

Marwin H. S. Segler

7 papers

Marco Fiscato

2 papers

Benedek Fabian

1 papers

TL;DR

Abstract

Authors

References43 items

Molecular representations in AI-driven drug discovery: a review and practical guide

Molecule Attention Transformer

Predicting Binding from Screening Assays with Transformer Network Embeddings

Deep Protein-Ligand Binding Prediction Using Unsupervised Learned Representations

Unified rational protein engineering with sequence-based deep representation learning

Inductive transfer learning for molecular activity prediction: Next-Gen QSAR Models with MolPMoFiT

SMILES-BERT: Large Scale Unsupervised Pre-Training for Molecular Property Prediction

Strategies for Pre-training Graph Neural Networks

Applications of machine learning in drug discovery and development

DeeplyTough: Learning Structural Comparison of Protein Binding Sites

Analyzing Learned Molecular Representations for Property Prediction

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

GuacaMol: Benchmarking Models for De Novo Molecular Design

Molecular Transformer: A Model for Uncertainty-Calibrated Chemical Reaction Prediction

Learning continuous and data-driven molecular descriptors by translating equivalent chemical representations

Improving Chemical Autoencoder Latent Space and Molecular De Novo Generation Diversity with Heteroencoders

Generating Focused Molecule Libraries for Drug Discovery with Recurrent Neural Networks

Using Rule-Based Labels for Weak Supervised Learning: A ChemNet for Transferable Chemical Property Prediction

Retrosynthetic Reaction Prediction Using Neural Sequence-to-Sequence Models

MoleculeNet: a benchmark for molecular machine learning

The ChEMBL database in 2017

Learning to SMILE(S)

Get Your Atoms in Order - An Open-Source Implementation of a Novel and Robust Molecular Canonicalization Algorithm

Adam: A Method for Stochastic Optimization

Heterogeneous Classifier Fusion for Ligand-Based Virtual Screening: Or, How Decision Making by Committee Can Be a Good Thing

Open-source platform to benchmark fingerprints for ligand-based virtual screening

Scikit-learn: Machine Learning in Python

Extended-Connectivity Fingerprints

Evaluating Virtual Screening Methods: Good and Bad Metrics for the "Early Recognition" Problem

SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules

On Canonical Numbering of Atoms in a Molecule and Graph Isomorphism

molecule benchmarks

ChemBench: The molecule benchmarks and MolMapNet datasets

RDKit: Open-Source Cheminformatics Software

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Visualizing Data using t-SNE

A reduction of a graph to a canonical form and an algebra arising during this reduction

DRUGLIKENESS: Subset of descriptors commonly used to assess druglikeness

SIMPLE: Small set of commonly used descriptors

FRAGMENT: Count and fragment based descriptor subset

GRAPH: Graph descriptor subset (following the grouping found in the GraphDescriptors module in RDKit)

: Electrotopological state (e-state) and VSA/e-state descriptor subset

SURFACE: MOE-based surface descriptor subset

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names