Training Graph Neural Networks with 1000 Layers (2021-06-14T00:00:00.000000Z)

TL;DR

This work finds that reversible connections in combination with deep network architectures enable the training of overparameterized GNNs that significantly outperform existing methods on multiple datasets.

Abstract

Deep graph neural networks (GNNs) have achieved excellent results on various tasks on increasingly large graph datasets with millions of nodes and edges. However, memory complexity has become a major obstacle when training deep GNNs for practical applications due to the immense number of nodes, edges, and intermediate activations. To improve the scalability of GNNs, prior works propose smart graph sampling or partitioning strategies to train GNNs with a smaller set of nodes or sub-graphs. In this work, we study reversible connections, group convolutions, weight tying, and equilibrium models to advance the memory and parameter efficiency of GNNs. We find that reversible connections in combination with deep network architectures enable the training of overparameterized GNNs that significantly outperform existing methods on multiple datasets. Our models RevGNN-Deep (1001 layers with 80 channels each) and RevGNN-Wide (448 layers with 224 channels each) were both trained on a single commodity GPU and achieve an ROC-AUC of $87.74 \pm 0.13$ and $88.24 \pm 0.15$ on the ogbn-proteins dataset. To the best of our knowledge, RevGNN-Deep is the deepest GNN in the literature by one order of magnitude. Please visit our project website https://www.deepgcns.org/arch/gnn1000 for more information.

Authors

V. Koltun

29 papers

Matthias Müller

4 papers

Bernard Ghanem

28 papers

TL;DR

Abstract

Authors

References79 items

Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depth

Bag of Tricks for Node Classification with Graph Neural Networks

OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs

Masked Label Prediction: Unified Massage Passing Model for Semi-Supervised Classification

GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training

Towards Deeper Graph Neural Networks

DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters

Simple and Deep Graph Convolutional Networks

Scaling Graph Neural Networks with Approximate PageRank

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

DeeperGCN: All You Need to Train Deeper GCNs

Towards Deeper Graph Neural Networks with Differentiable Group Normalization

Bayesian Graph Neural Networks with Adaptive Connection Sampling

Multiscale Deep Equilibrium Models

Language Models are Few-Shot Learners

Open Graph Benchmark: Datasets for Machine Learning on Graphs

SIGN: Scalable Inception Graph Neural Networks

Geometrically Principled Connections in Graph Neural Networks

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Microsoft Academic Graph: When experts are not enough

Reformer: The Efficient Transformer

Improved protein structure prediction using potentials from deep learning

PyTorch: An Imperative Style, High-Performance Deep Learning Library

DeepGCNs: Making GCNs Go as Deep as CNNs

PairNorm: Tackling Oversmoothing in GNNs

Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks.

Deep Equilibrium Models

MemCNN: A Python/PyTorch package for creating memory-efficient invertible neural networks

DropEdge: Towards Deep Graph Convolutional Networks on Node Classification

GraphSAINT: Graph Sampling Based Inductive Learning Method

Graph Neural Networks Exponentially Lose Expressive Power for Node Classification

Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks

Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation

Graph Normalizing Flows

DeepGCNs: Can GCNs Go As Deep As CNNs?

Fast Graph Representation Learning with PyTorch Geometric

Simplifying Graph Convolutional Networks

Reconciling modern machine-learning practice and the classical bias–variance trade-off

Pitfalls of Graph Neural Network Evaluation

How Powerful are Graph Neural Networks?

Predict then Propagate: Graph Neural Networks meet Personalized PageRank

Trellis Networks for Sequence Modeling

Large-Scale Learnable Graph Convolutional Networks

Representation Learning on Graphs with Jumping Knowledge Networks

FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling

Dynamic Graph CNN for Learning on Point Clouds

Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning

Adaptive Graph Convolutional Neural Networks

Graph Attention Networks

Stochastic Training of Graph Convolutional Networks with Variance Reduction

The Reversible Residual Network: Backpropagation Without Storing Activations

Predicting multicellular function through multi-layer tissue networks

Inductive Representation Learning on Large Graphs

CayleyNets: Graph Convolutional Neural Networks With Complex Rational Spectral Filters

Neural Message Passing for Quantum Chemistry

Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs

Aggregated Residual Transformations for Deep Neural Networks

Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling

Semi-Supervised Classification with Graph Convolutional Networks

Densely Connected Convolutional Networks

Using the Output Embedding to Improve Language Models

Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering

Learning Convolutional Neural Networks for Graphs

Revisiting Semi-Supervised Learning with Graph Embeddings

Deep Residual Learning for Image Recognition

Multi-Scale Context Aggregation by Dilated Convolutions

Deep Convolutional Networks on Graph-Structured Data

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Spectral Networks and Locally Connected Networks on Graphs

ImageNet classification with deep convolutional neural networks

A new model for learning in graph domains

An Introduction to Metric Spaces and Fixed Point Theory

A Class of Methods for Solving Nonlinear Simultaneous Equations

Benchmarking Graph Neural Networks

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

GraphSAINT: Graph Sampling Based Inductive Learning Method