GNOT: A General Neural Operator Transformer for Operator Learning (2023-02-28T00:00:00.000000Z)

TL;DR

This work proposes a general neural operator transformer (GNOT), a scalable and effective transformer-based framework for learning operators that is highly flexible to handle multiple input functions and irregular meshes and introduces a geometric gating mechanism which could be viewed as a soft domain decomposition to solve the multi-scale problems.

Abstract

Learning partial differential equations' (PDEs) solution operators is an essential problem in machine learning. However, there are several challenges for learning operators in practical applications like the irregular mesh, multiple input functions, and complexity of the PDEs' solution. To address these challenges, we propose a general neural operator transformer (GNOT), a scalable and effective transformer-based framework for learning operators. By designing a novel heterogeneous normalized attention layer, our model is highly flexible to handle multiple input functions and irregular meshes. Besides, we introduce a geometric gating mechanism which could be viewed as a soft domain decomposition to solve the multi-scale problems. The large model capacity of the transformer architecture grants our model the possibility to scale to large datasets and practical problems. We conduct extensive experiments on multiple challenging datasets from different domains and achieve a remarkable improvement compared with alternative methods. Our code and data are publicly available at \url{https://github.com/thu-ml/GNOT}.

Authors

Zhongkai Hao

2 papers

Chengyang Ying

1 papers

Zhengyi Wang

2 papers

TL;DR

Abstract

Authors

References37 items

NUNO: A General Framework for Learning Parametric PDEs with Non-Uniform Data

Augmented Physics-Informed Neural Networks (APINNs): A gating network-based soft domain decomposition methodology

Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications

Fourier Neural Operator with Learned Deformations for PDEs on General Geometries

Transformer for Partial Differential Equations' Operator Learning

Variable-Input Deep Operator Networks

MIONet: Learning multiple-input operators via tensor product

Factorized Fourier Neural Operators

Improved Architectures and Training Algorithms for Deep Operator Networks

Multiwavelet-based Operator Learning for Differential Equations

U-FNO - an enhanced Fourier neural operator based-deep learning model for multiphase flow

Choose a Transformer: Fourier or Galerkin

Learning the solution operator of parametric partial differential equations with physics-informed DeepONets

Random Feature Attention

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Fourier Neural Operator for Parametric Partial Differential Equations

Rethinking Attention with Performers

Efficient Transformers: A Survey

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

Extended Physics-informed Neural Networks (XPINNs): A Generalized Space-Time Domain Decomposition based Deep Learning Framework for Nonlinear Partial Differential Equations

Longformer: The Long-Document Transformer

Scaling Laws for Neural Language Models

Reformer: The Efficient Transformer

Generating Long Sequences with Sparse Transformers

CCNet: Criss-Cross Attention for Semantic Segmentation

Super-convergence: very fast training of neural networks using large learning rates

Decoupled Weight Decay Regularization

U-Net: Convolutional Networks for Biomedical Image Segmentation

Principles of Multiscale Modeling

Adaptive Mixtures of Local Experts

HT-Net: Hierarchical Transformer based Operator Learning Model for Multiscale PDEs

Towards Large-Scale Learned Solvers for Parametric PDEs with Model-Parallel Fourier Neural Operators

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators

A Survey of Unstructured Mesh Generation Technology

Introduction to partial differential equations with applications

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names