How Attentive are Graph Attention Networks?

Published in

International Conference on Learning Representa...(2021)

External Links:

Generate Graph

TL;DR

It is shown that GAT computes a very limited kind of attention: the ranking of the attention scores is unconditioned on the query node, and a simple fix is introduced by modifying the order of operations and proposed GATv2: a dynamic graph attention variant that is strictly more expressive than GAT.

Abstract

Graph Attention Networks (GATs) are one of the most popular GNN architectures and are considered as the state-of-the-art architecture for representation learning with graphs. In GAT, every node attends to its neighbors given its own representation as the query. However, in this paper we show that GAT computes a very limited kind of attention: the ranking of the attention scores is unconditioned on the query node. We formally define this restricted kind of attention as static attention and distinguish it from a strictly more expressive dynamic attention. Because GATs use a static attention mechanism, there are simple graph problems that GAT cannot express: in a controlled problem, we show that static attention hinders GAT from even fitting the training data. To remove this limitation, we introduce a simple fix by modifying the order of operations and propose GATv2: a dynamic graph attention variant that is strictly more expressive than GAT. We perform an extensive evaluation and show that GATv2 outperforms GAT across 11 OGB and other benchmarks while we match their parametric costs. Our code is available at https://github.com/tech-srl/how_attentive_are_gats . GATv2 is available as part of the PyTorch Geometric library, the Deep Graph Library, and the TensorFlow GNN library.

Authors

Uri Alon

7 papers

Shaked Brody

4 papers

Eran Yahav

4 papers

References72 items

How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision

Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges

A Generalization of Transformer Networks to Graphs

Combining Label Propagation and Simple Models Out-performs Graph Neural Networks

Masked Label Prediction: Unified Massage Passing Model for Semi-Supervised Classification

How Attentive are Graph Attention Networks?

Published in

International Conference on Learning Representa...(2021)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Uri Alon

7 papers

Shaked Brody

4 papers

Eran Yahav

4 papers

References72 items

How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision

Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges

A Generalization of Transformer Networks to Graphs

Combining Label Propagation and Simple Models Out-performs Graph Neural Networks

Masked Label Prediction: Unified Massage Passing Model for Semi-Supervised Classification

Entity-Aware Dependency-Based Deep Graph Attention Network for Comparative Preference Classification

Self-Supervised Graph Transformer on Large-Scale Molecular Data

Minimum Width for Universal Approximation

Pointer Graph Networks

On the Bottleneck of Graph Neural Networks and its Practical Implications

Contrastive Multi-View Representation Learning on Graphs

Open Graph Benchmark: Datasets for Machine Learning on Graphs

Adaptive Structural Fingerprints for Graph Attention Networks

Distilling Knowledge From Graph Convolutional Networks

Improving Graph Attention Networks with Large Margin-based Constraints

PairNorm: Tackling Oversmoothing in GNNs

Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks.

Syntax-Aware Aspect Level Sentiment Classification with Graph Attention Networks

DropEdge: Towards Deep Graph Convolutional Networks on Node Classification

GraphSAINT: Graph Sampling Based Inductive Learning Method

Graph Representation Learning via Hard and Channel-Wise Attention Networks

Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks

GNN-FiLM: Graph Neural Networks with Feature-wise Linear Modulation

Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs

DeepGCNs: Can GCNs Go As Deep As CNNs?

Heterogeneous Graph Attention Network

Fast Graph Representation Learning with PyTorch Geometric

Simplifying Graph Convolutional Networks

How Powerful are Graph Neural Networks?

Relational Graph Attention Networks

Attention Models in Graphs

DeepInf: Social Influence Prediction with Deep Learning

On the Practical Computational Power of Finite Precision RNNs for Language Recognition

GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs

Attention-based Graph Neural Network for Semi-supervised Learning

Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning

Learning to Represent Programs with Graphs

Graph Attention Networks

Programmable Agents

VAIN: Attentional Multi-agent Predictive Modeling

Attention is All you Need

Inductive Representation Learning on Large Graphs

A simple neural network module for relational reasoning

Neural Message Passing for Quantum Chemistry

One-Shot Imitation Learning

Interaction Networks for Learning about Objects, Relations and Physics

Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs

Geometric Deep Learning: Going beyond Euclidean data

Semi-Supervised Classification with Graph Convolutional Networks

Deep Residual Learning for Image Recognition

Gated Graph Sequence Neural Networks

Diffusion-Convolutional Neural Networks

Convolutional Networks on Graphs for Learning Molecular Fingerprints

Effective Approaches to Attention-based Neural Machine Translation

Neural Machine Translation by Jointly Learning to Align and Translate

Quantum chemistry structures and properties of 134 kilo molecules

Collective Classification in Network Data

A new model for learning in graph domains

Cryptosporidium Parvum Genome Project

Approximation theory of the MLP model in neural networks

Original Contribution: Multilayer feedforward networks with a nonpolynomial activation function can approximate any function

Approximation capabilities of multilayer feedforward networks

Approximation by superpositions of a sigmoidal function

Multilayer feedforward networks are universal approximators

On the approximate realization of continuous mappings by neural networks

Benchmarking Graph Neural Networks

Bag of Tricks of Semi-Supervised Classification with Graph Neural Networks

runs±std). In all datasets, GATv2 outperforms GAT

A Comprehensive Survey on Graph Neural Networks

Time Complexity GAT

The Graph Neural Network Model

Collective Classi!cation in Network Data

Field of Study

Computer Science

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

International Conference on Learning Representations

Type

conference

URL

https://iclr.cc/

Alternate Names

Int Conf Learn Represent
ICLR