Energy-Motivated Equivariant Pretraining for 3D Molecular Graphs (2022-07-18T00:00:00.000000Z)

TL;DR

This work proposes to adopt an equivariant energy-based model as the backbone for pretraining, which enjoys the merits of fulfilling the symmetry of 3D space, and develops a node-level pretraining loss for force prediction, where the Riemann-Gaussian distribution is exploited to ensure the loss to be E(3)-invariant, enabling more robustness.

Abstract

Pretraining molecular representation models without labels is fundamental to various applications. Conventional methods mainly process 2D molecular graphs and focus solely on 2D tasks, making their pretrained models incapable of characterizing 3D geometry and thus defective for downstream 3D tasks. In this work, we tackle 3D molecular pretraining in a complete and novel sense. In particular, we first propose to adopt an equivariant energy-based model as the backbone for pretraining, which enjoys the merits of fulfilling the symmetry of 3D space. Then we develop a node-level pretraining loss for force prediction, where we further exploit the Riemann-Gaussian distribution to ensure the loss to be E(3)-invariant, enabling more robustness. Moreover, a graph-level noise scale prediction task is also leveraged to further promote the eventual performance. We evaluate our model pretrained from a large-scale 3D dataset GEOM-QM9 on two challenging 3D benchmarks: MD17 and QM9. Experimental results demonstrate the efficacy of our method against current state-of-the-art pretraining approaches, and verify the validity of our design for each proposed component. Code is available at https://github.com/jiaor17/3D-EMGP.

Authors

Rui Jiao

2 papers

Yu Rong

3 papers

Wenbing Huang

2 papers

TL;DR

Abstract

Authors

References51 items

Unified 2D and 3D Pre-Training of Molecular Representations

Geometrically Equivariant Graph Neural Networks: A Survey

3D Infomax improves GNNs for Molecular Property Prediction

Pre-training Molecular Graph Representation with 3D Geometry

Geometry-enhanced molecular representation learning for property prediction

Graph Contrastive Learning Automated

Do Transformers Really Perform Bad for Graph Representation?

MoCL: Data-driven Molecular Fingerprint via Knowledge-aware Contrastive Learning from Molecular Graph

Learning Gradient Fields for Molecular Conformation Generation

E(n) Equivariant Graph Neural Networks

Spherical Message Passing for 3D Molecular Graphs

LieTransformer: Equivariant self-attention for Lie Groups

Graph Contrastive Learning with Augmentations

Differentiable Manifold Reconstruction for Point Cloud Denoising

GPT-GNN: Generative Pre-Training of Graph Neural Networks

Self-Supervised Graph Transformer on Large-Scale Molecular Data

SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks

GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training

Improved Techniques for Training Score-Based Generative Models

GEOM: Energy-annotated molecular conformations for property prediction and molecular generation

Directional Message Passing for Molecular Graphs

Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data

InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization

OnionNet: a Multiple-Layer Intermolecular-Contact-Based Convolutional Neural Network for Protein–Ligand Binding Affinity Prediction

Cormorant: Covariant Molecular Neural Networks

Strategies for Pre-training Graph Neural Networks

Fast Graph Representation Learning with PyTorch Geometric

How Powerful are Graph Neural Networks?

Deep Graph Infomax

SchNetPack: A Deep Learning Toolbox For Atomistic Systems.

Tensor Field Networks: Rotation- and Translation-Equivariant Neural Networks for 3D Point Clouds

SchNet: A continuous-filter convolutional neural network for modeling quantum interactions

Inductive Representation Learning on Large Graphs

Neural Message Passing for Quantum Chemistry

Variational Graph Auto-Encoders

Machine learning of accurate energy-conserving molecular force fields

Molecular graph convolutions: moving beyond fingerprints

AtomNet: A Deep Convolutional Neural Network for Bioactivity Prediction in Structure-based Drug Discovery

Riemannian Gaussian Distributions on the Space of Symmetric Positive Definite Matrices

Quantum chemistry structures and properties of 134 kilo molecules

A Connection Between Score Matching and Denoising Autoencoders

Uni-Mol: A Universal 3D Molecular Representation Learning Framework

TorchMD-NET: Equivariant Transformers for Neural Network based Molecular Potentials

Predicting Molecular Conformation via Dynamic Graph Score Matching

MoCL: Contrastive Learning on Molecular Graphs with Multi-level Domain Knowledge

Code and Related Assets Our code is available at https://github.com/jiaor17/3DEMGP. The implementation and hyper-parameter settings are partially based on ConfGF (Shi et al. 2021)

2021) and Pytorch Geometric (Fey and Lenssen 2019)

To transfer or not to transfer

Molecular Modeling and Simulation: An Interdisciplinary Guide

Studien uber das Gleichgewicht der lebenden Kraft

on Machine Learning, ICML 2021, 18-24 July 2021

Field of Study

Venue Information

Name

Type

URL

Alternate Names