Molecule Attention Transformer

Published in

arXiv.org(2020)

External Links:

Generate Graph

TL;DR

This work proposes Molecule Attention Transformer, a key innovation is to augment the attention mechanism in Transformer using inter-atomic distances and the molecular graph structure, and shows that attention weights learned are interpretable from the chemical point of view.

Abstract

Designing a single neural network architecture that performs competitively across a range of molecule property prediction tasks remains largely an open challenge, and its solution may unlock a widespread use of deep learning in the drug discovery industry. To move towards this goal, we propose Molecule Attention Transformer (MAT). Our key innovation is to augment the attention mechanism in Transformer using inter-atomic distances and the molecular graph structure. Experiments show that MAT performs competitively on a diverse set of molecular prediction tasks. Most importantly, with a simple self-supervised pretraining, MAT requires tuning of only a few hyperparameter values to achieve state-of-the-art performance on downstream tasks. Finally, we show that attention weights learned by MAT are interpretable from the chemical point of view.

Authors

Stanislaw Jastrzebski

5 papers

J. Tabor

2 papers

Lukasz Maziarka

1 papers

References55 items

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Do Attention Heads in BERT Track Syntactic Dependencies?

SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery

Building Attention and Edge Convolution Neural Networks for Bioactivity and Physical-Chemical Property Prediction

SMILES-BERT: Large Scale Unsupervised Pre-Training for Molecular Property Prediction

Molecule Attention Transformer

Published in

arXiv.org(2020)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Stanislaw Jastrzebski

5 papers

J. Tabor

2 papers

Lukasz Maziarka

1 papers

References55 items

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Do Attention Heads in BERT Track Syntactic Dependencies?

SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery

Building Attention and Edge Convolution Neural Networks for Bioactivity and Physical-Chemical Property Prediction

SMILES-BERT: Large Scale Unsupervised Pre-Training for Molecular Property Prediction

Tomasz Danel

1 papers

Slawomir Mucha

1 papers

Krzysztof Rataj

1 papers

Self-Attention Based Molecule Representation for Predicting Drug-Target Interaction

Gaussian Transformer: A Lightweight Approach for Natural Language Inference

Alchemy: A Quantum Chemistry Dataset for Benchmarking AI Models

What Does BERT Look at? An Analysis of BERT’s Attention

Pre-training Graph Neural Networks

A Transformer Model for Retrosynthesis

A survey on Big Data and Machine Learning for Chemistry

Attention Augmented Convolutional Networks

Analyzing Learned Molecular Representations for Property Prediction

Graph Warp Module: an Auxiliary Module for Boosting the Power of Graph Neural Networks

Advances of machine learning in molecular modeling and simulation

Enhanced Deep‐Learning Prediction of Molecular Properties via Augmentation of Bond Topology

PubChem 2019 update: improved access to chemical data

Large-scale comparison of machine learning methods for drug target prediction on ChEMBL

Relational inductive biases, deep learning, and graph networks

MetStabOn—Online Platform for Metabolic Stability Predictions

PotentialNet for Molecular Property Prediction

Self-Attention with Relative Position Representations

Edge Attention-based Multi-Relational Graph Convolutional Networks

Estimation of clinical trial success rates and related parameters

Adaptive Graph Convolutional Neural Networks

Generating Focused Molecule Libraries for Drug Discovery with Recurrent Neural Networks

Comparison of Deep Learning With Multiple Machine Learning Methods and Metrics Using Diverse Drug Discovery Data Sets.

Neural Natural Language Inference Models Enhanced with External Knowledge

Graph Attention Networks

Learning Graph-Level Representation for Drug Discovery

Convolutional Embedding of Attributed Molecular Graphs for Physical Property Prediction

Chemception: A Deep Neural Network with Minimal Chemistry Knowledge Matches the Performance of Expert-developed QSAR/QSPR Models

Attention is All you Need

Neural Message Passing for Quantum Chemistry

MoleculeNet: a benchmark for molecular machine learning

Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules

Quantum-chemical insights from deep tensor neural networks

Molecular graph convolutions: moving beyond fingerprints

Learning to SMILE(S)

AtomNet: A Deep Convolutional Neural Network for Bioactivity Prediction in Structure-based Drug Discovery

Convolutional Networks on Graphs for Learning Molecular Fingerprints

Adam: A Method for Stochastic Optimization

Neural Machine Translation by Jointly Learning to Align and Translate

ImageNet classification with deep convolutional neural networks

Random Search for Hyper-Parameter Optimization

ChEMBL: a large-scale bioactivity database for drug discovery

Extended-Connectivity Fingerprints

The properties of known drugs. 1. Molecular frameworks.

SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Three-dimensionally embedded graph convolutional network (3DGCN) for molecule interpretation

Rdkit: Open-source cheminformatics software

Experimental and computational approaches to estimate solubility and permeability in drug discovery and development settings.

Field of Study

Computer SciencePhysicsMathematics

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

arXiv.org

Type

URL

https://arxiv.org

Alternate Names

ArXiv