Scalable Transformer for PDE Surrogate Modeling (2023-05-27T00:00:00.000000Z)

TL;DR

Factorized Transformer(FactFormer), which is based on an axial factorized kernel integral, is proposed, which is able to simulate 2D Kolmogorov flow on a 256 by 256 grid and 3D smoke buoyancy on a 64 by64 by 64 grid with good accuracy and efficiency.

Abstract

Transformer has shown state-of-the-art performance on various applications and has recently emerged as a promising tool for surrogate modeling of partial differential equations (PDEs). Despite the introduction of linear-complexity variant, applying attention to a large number of grid points can result in instability and is still expensive to compute. In this work, we propose Factorized Transformer(FactFormer), which is based on an axial factorized kernel integral. Concretely, we introduce a learnable projection operator that decomposes the input function into multiple sub-functions with one-dimensional domain. These sub-functions are then evaluated and used to compute the instance-based kernel with an axial factorized scheme. We showcase that the proposed model is able to simulate 2D Kolmogorov flow on a 256 by 256 grid and 3D smoke buoyancy on a 64 by 64 by 64 grid with good accuracy and efficiency. In addition, we find out that with the factorization scheme, the attention matrices enjoy a more compact spectrum than full softmax-free attention matrices.

Authors

A. Farimani

2 papers

Zijie Li

1 papers

Dule Shu

1 papers

TL;DR

Abstract

Authors

References123 items

Multi-Grid Tensorized Fourier Neural Operator for High-Resolution PDEs

ViTO: Vision Transformer-Operator

GNOT: A General Neural Operator Transformer for Operator Learning

Eagle: Large-Scale Learning of Turbulent Fluid Dynamics with Mesh Transformers

Continuous Spatiotemporal Transformers

ClimaX: A foundation model for weather and climate

GraphCast: Learning skillful medium-range global weather forecasting

Reliable extrapolation of deep neural operators informed by physics or sparse observations

A Physics-informed Diffusion Model for High-fidelity Flow Field Reconstruction

Mitigating spectral bias for the multiscale operator learning

Guaranteed Conservation of Momentum for Learning Particle-based Fluid Dynamics

Towards Multi-spatiotemporal-scale Generalized PDE Modeling

Transformer Meets Boundary Value Inverse Problems

Clifford Neural Layers for PDE Modeling

Earthformer: Exploring Space-Time Transformers for Earth System Forecasting

Neural Green's function for Laplacian systems

Learning the Solution Operator of Boundary Value Problems using Graph Neural Networks

Transformer for Partial Differential Equations' Operator Learning

Learning Green's functions associated with time-dependent partial differential equations

Respecting causality is all you need for training physics-informed neural networks

FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators

Lie Point Symmetry Data Augmentation for Neural PDE Solvers

MIONet: Learning multiple-input operators via tensor product

How to Understand Masked Autoencoders

Message Passing Neural PDE Solvers

Graph neural network-accelerated Lagrangian fluid simulation

Predicting Physics in Mesh-reduced Space with Temporal Attention

Learned Coarse Models for Efficient Turbulence Simulation

High-Resolution Image Synthesis with Latent Diffusion Models

Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers

A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data

Physics-Informed Neural Operator for Learning Partial Differential Equations

Multiwavelet-based Operator Learning for Differential Equations

U-FNO - an enhanced Fourier neural operator based-deep learning model for multiphase flow

Neural Operator: Learning Maps Between Function Spaces

Highly accurate protein structure prediction with AlphaFold

On universal approximation and error bounds for Fourier Neural Operators

Deep Learning for Reduced Order Modelling and Efficient Temporal Evolution of Fluid Simulations

Eigen Analysis of Self-Attention and its Reconstruction from Partial Computation

Transformers are Deep Infinite-Dimensional Non-Mercer Binary Kernel Machines

Choose a Transformer: Fourier or Galerkin

Physics-informed neural networks (PINNs) for fluid mechanics: a review

RoFormer: Enhanced Transformer with Rotary Position Embedding

Learning the solution operator of parametric partial differential equations with physics-informed DeepONets

Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth

Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention

Machine learning–accelerated computational fluid dynamics

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Fourier Neural Operator for Parametric Partial Differential Equations

Learning Mesh-Based Simulation with Graph Networks

Transformers for Modeling Physical Systems

Rethinking Attention with Performers

Using Machine Learning to Augment Coarse-Grid Computational Fluid Dynamics Simulations

Big Bird: Transformers for Longer Sequences

Deep spatial transformers for autoregressive data-driven forecasting of geophysical turbulence

Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solvers

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains

Multipole Graph Neural Operator for Parametric Partial Differential Equations

Linformer: Self-Attention with Linear Complexity

Extended Physics-informed Neural Networks (XPINNs): A Generalized Space-Time Domain Decomposition based Deep Learning Framework for Nonlinear Partial Differential Equations

Language Models are Few-Shot Learners

End-to-End Object Detection with Transformers

Model Reduction and Neural Networks for Parametric PDEs

MESHFREEFLOWNET: A Physics-Constrained Deep Continuous Space-Time Super-Resolution Framework

Lagrangian Fluid Simulation with Continuous Convolutions

Longformer: The Long-Document Transformer

Neural Operator: Graph Kernel Network for Partial Differential Equations

Learning to Simulate Complex Physics with Graph Networks

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

WeatherBench: A Benchmark Data Set for Data‐Driven Weather Forecasting

Reformer: The Efficient Transformer

Towards Physics-informed Deep Learning for Turbulent Flow Prediction

Axial Attention in Multidimensional Transformers

Transformer Dissection: An Unified Understanding for Transformer’s Attention via the Lens of Kernel

DeepXDE: A Deep Learning Library for Solving Differential Equations