Molecular Geometry Pretraining with SE(3)-Invariant Denoising Distance Matching (2022-06-27T00:00:00.000000Z)

TL;DR

GeoSSL is proposed, a 3D coordinate denoising pretraining framework to model such an energy landscape based on the dynamic nature of 3D molecules, where the continuous motion of a molecule in the 3D Euclidean space forms a smooth potential energy surface.

Abstract

Molecular representation pretraining is critical in various applications for drug and material discovery due to the limited number of labeled molecules, and most existing work focuses on pretraining on 2D molecular graphs. However, the power of pretraining on 3D geometric structures has been less explored. This is owing to the difficulty of finding a sufficient proxy task that can empower the pretraining to effectively extract essential features from the geometric structures. Motivated by the dynamic nature of 3D molecules, where the continuous motion of a molecule in the 3D Euclidean space forms a smooth potential energy surface, we propose GeoSSL, a 3D coordinate denoising pretraining framework to model such an energy landscape. Further by leveraging an SE(3)-invariant score matching method, we propose GeoSSL-DDM in which the coordinate denoising proxy task is effectively boiled down to denoising the pairwise atomic distances in a molecule. Our comprehensive experiments confirm the effectiveness and robustness of our proposed method.

Authors

Shengchao Liu

3 papers

Jian Tang

8 papers

Hongyu Guo

2 papers

TL;DR

Abstract

Authors

References80 items

A text-guided protein design framework

Multi-modal molecule structure–text model for text-based retrieval and editing

e3nn: Euclidean Neural Networks

Self-Supervised Pre-training for Protein Embeddings Using Tertiary Structures

Pre-training via Denoising for Molecular Property Prediction

MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Design

GFlowNet Foundations

Pre-training Molecular Graph Representation with 3D Geometry

Geometric and Physical Quantities improve E(3) Equivariant Message Passing

Molecule3D: A Benchmark for Predicting 3D Geometries from Molecular Graphs

Geometric deep learning on molecular representations

The Bottleneck

Rotation Invariant Graph Neural Networks using Spin Convolutions

Geometry-enhanced molecular representation learning for property prediction

Non-Autoregressive Electron Redistribution Modeling for Reaction Prediction

GemNet: Universal Directional Graph Neural Networks for Molecules

SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training

Learning Gradient Fields for Molecular Conformation Generation

Graph Self-Supervised Learning: A Survey

Self-Supervised Learning of Graph Neural Networks: A Unified Review

E(n) Equivariant Graph Neural Networks

Equivariant message passing for the prediction of tensorial properties and molecular spectra

GraphEBM: Molecular Graph Generation with Energy-Based Models

How to Train Your Energy-Based Models

ATOM3D: Tasks On Molecules in Three Dimensions

Improved Contrastive Divergence Training of Energy Based Models

Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Molecules

Score-Based Generative Modeling through Stochastic Differential Equations

Exploring Simple Siamese Representation Learning

Graph Contrastive Learning with Augmentations

Functional Regularization for Representation Learning: A Unified Theoretical Perspective

Energy-based View of Retrosynthesis

Learning Graph-Convolutional Representations for Point Cloud Denoising

GPT-GNN: Generative Pre-Training of Graph Neural Networks

SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks

MoFlow: An Invertible Flow Model for Generating Molecular Graphs

Improved Techniques for Training Score-Based Generative Models

Self-Supervised Learning: Generative or Contrastive

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

GEOM: Energy-annotated molecular conformations for property prediction and molecular generation

A Hierarchical Graph Network for 3D Object Detection on Point Clouds

Learning To Navigate The Synthetically Accessible Chemical Space Using Reinforcement Learning

A Graph to Graphs Framework for Retrosynthesis Prediction

Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud

GraphAF: a Flow-based Autoregressive Model for Molecular Graph Generation

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization

Generative Modeling by Estimating Gradients of the Data Distribution

Strategies for Pre-training Graph Neural Networks

Analyzing Learned Molecular Representations for Property Prediction

GuacaMol: Benchmarking Models for De Novo Molecular Design

Representation Learning with Contrastive Predictive Coding

N-Gram Graph: Simple Unsupervised Representation for Graphs, with Applications to Molecules

Tensor Field Networks: Rotation- and Translation-Equivariant Neural Networks for 3D Point Clouds

SchNet - A deep learning architecture for molecules and materials.

SchNet: A continuous-filter convolutional neural network for modeling quantum interactions

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

Neural Message Passing for Quantum Chemistry

Machine learning of accurate energy-conserving molecular force fields

SGDR: Stochastic Gradient Descent with Warm Restarts

Protein Binding Pocket Dynamics.

Convolutional Networks on Graphs for Learning Molecular Fingerprints

Quantum chemistry structures and properties of 134 kilo molecules

Markov state models of biomolecular conformational dynamics.

A Connection Between Score Matching and Denoising Autoencoders

Noise-contrastive estimation: A new estimation principle for unnormalized statistical models

Estimation of Non-Normalized Statistical Models by Score Matching

The PDBbind database: methodologies and updates.

Exploring potential energy surfaces for chemical reactions: An overview of some practical methods

Merck molecular force field. I. Basis, form, scope, parameterization, and performance of MMFF94

Simple GNN Regularisation for 3D Molecular Property Prediction and Beyond

Spherical Message Passing for 3D Graph Networks

Self-supervised on Graphs: Contrastive, Generative, or Predictive

Applied chemoinformatics: achievements and future opportunities

RDKit: A software suite for cheminformatics, computational chemistry, and predictive modeling

We run two backbone models: PaiNN in Appendix F.1 and SchNet in Appendix