Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations (2024-07-09T00:00:00.000000Z)

TL;DR

It is concluded that ML-for-PDE-solving research is overoptimistic: weak baselines lead to overly positive results, while reporting biases lead to under-reporting of negative results.

Abstract

One of the most promising applications of machine learning in computational physics is to accelerate the solution of partial differential equations (PDEs). The key objective of machine-learning-based PDE solvers is to output a sufficiently accurate solution faster than standard numerical methods, which are used as a baseline comparison. We first perform a systematic review of the ML-for-PDE-solving literature. Out of all of the articles that report using ML to solve a fluid-related PDE and claim to outperform a standard numerical method, we determine that 79% (60/76) make a comparison with a weak baseline. Second, we find evidence that reporting biases are widespread, especially outcome reporting and publication biases. We conclude that ML-for-PDE-solving research is overoptimistic: weak baselines lead to overly positive results, while reporting biases lead to under-reporting of negative results. To a large extent, these issues seem to be caused by factors similar to those of past reproducibility crises: researcher degrees of freedom and a bias towards positive results. We call for bottom-up cultural changes to minimize biased reporting as well as top-down structural reforms to reduce perverse incentives for doing so. A systematic review of machine learning approaches to solve partial differential equations related to fluid dynamics highlights concerns about reproducibility and indicates that studies in this area have reached overly optimistic conclusions.

Authors

Nick McGreivy

1 papers

Ammar Hakim

1 papers

TL;DR

Abstract

Authors

References190 items

Promising directions of machine learning for partial differential equations

Deep neural operators as accurate surrogates for shape optimization

REFORMS: Reporting Standards for Machine Learning Based Science

PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solvers

Cost vs Accuracy: Dns of Turbulent Flow Over a Sphere Using Structured Immersed-Boundary, Unstructured Finite-Volume, and Spectral-Element Methods

Leakage and the reproducibility crisis in machine-learning-based science

Predictive Limitations of Physics-Informed Neural Networks in Vortex Shedding

E(3) Equivariant Graph Neural Networks for Particle-Based Fluid Mechanics

Invariant preservation in machine learned PDE solvers via error correction

Fourier Neural Operator for Plasma Modelling

Can physics-informed neural networks beat the finite element method?

Deep neural operators can serve as accurate surrogates for shape optimization: A case study for airfoils

Evolve Smoothly, Fit Consistently: Learning Smooth Latent Dynamics For Advection-Dominated Systems

Benchmarking of Machine Learning Ocean Subgrid Parameterizations in an Idealized Model

A Poisson’s Equation Solver Based on Neural Network Precondtioned CG Method

Physics-informed neural networks for data-driven simulation: Advantages, limitations, and opportunities

DS-GPS : A Deep Statistical Graph Poisson Solver (for faster CFD simulations)

Meta-PDE: Learning to Solve PDEs Quickly Without a Mesh

Neural operator prediction of linear instability waves in high-speed boundary layers

Immersed boundary-physics informed machine learning approach for fluid–solid coupling

A composable machine-learning approach for steady-state simulations on high-resolution grids

Linear attention coupled Fourier neural operator for simulation of three-dimensional turbulence

Towards Multi-spatiotemporal-scale Generalized PDE Modeling

Rapid Seismic Waveform Modeling and Inversion With Neural Operators

Deep Learning Surrogate for the Temporal Propagation and Scattering of Acoustic Waves

A machine learning based solver for pressure Poisson equations

Accelerating high order discontinuous Galerkin solvers using neural networks: 3D compressible Navier-Stokes equations

Fourier Neural Operator with Learned Deformations for PDEs on General Geometries

Learning to correct spectral methods for simulating turbulent flows

Neural Green's function for Laplacian systems

LordNet: An efficient neural network for learning to solve parametric partial differential equations without simulated data

Learning to Accelerate Partial Differential Equations via Latent Global Evolution

Residual-based physics-informed transfer learning: A hybrid method for accelerating long-term CFD simulations via deep learning

Local Randomized Neural Networks with Discontinuous Galerkin Methods for Partial Differential Equations

Learning to Solve PDE-constrained Inverse Problems with Graph Networks

Experience report of physics-informed neural networks in fluid simulations: pitfalls and frustration

Solving inverse problems in physics by optimizing a discrete loss: Fast and accurate learning without neural networks

REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research

Leveraging Stochastic Predictions of Bayesian Neural Networks for Fluid Simulations

Numerical Computation of Partial Differential Equations by Hidden-Layer Concatenated Extreme Learning Machine

Sources of Irreproducibility in Machine Learning: A Review

Multigrid-augmented deep learning preconditioners for the Helmholtz equation

Simulating Liquids with Graph Networks

Catalogue of bias: selective outcome reporting bias

Learned turbulence modelling with differentiable fluid solvers: physics-based loss functions and optimisation horizons

Deep Neural Networks to Correct Sub-Precision Errors in CFD

Message Passing Neural PDE Solvers

Graph neural network-accelerated Lagrangian fluid simulation

Deep Petrov-Galerkin Method for Solving Partial Differential Equations

Predicting Physics in Mesh-reduced Space with Temporal Attention

Scientific Machine Learning Through Physics–Informed Neural Networks: Where we are and What’s Next

A Critical Evaluation of Physics-informed Neural Networks for the Solution of Elliptic Partial Differential Equations

Accelerating high order discontinuous Galerkin solvers using neural networks: 1D Burgers’ equation

Learned Coarse Models for Efficient Turbulence Simulation

Machine Learning for Computational Science and Engineering - a brief introduction and some critical questions

Data-dredging bias

Machine learning in the search for new fundamental physics

WENO3-NN: A maximum-order three-point data-driven weighted essentially non-oscillatory scheme

Navigating the pitfalls of applying machine learning in genomics

Physics-Informed Neural Operator for Learning Partial Differential Equations

On Computing the Hyperparameter of Extreme Learning Machines: Algorithm and Application to Computational PDEs, and Comparison with Classical and High-Order Finite Elements

Fast PDE-constrained optimization via self-supervised operator learning

Machine learning accelerated particle-in-cell plasma simulations

A composable autoencoder-based iterative algorithm for accelerating numerical simulations

Attention-enhanced neural network models for turbulence simulation

Enhancing computational fluid dynamics with machine learning

Improved Architectures and Training Algorithms for Deep Operator Networks

Performance and accuracy assessments of an incompressible fluid solver coupled with a deep convolutional neural network

Physics-based Deep Learning

Semi-Implicit Neural Solver for Time-dependent Partial Differential Equations

Characterizing possible failure modes in physics-informed neural networks

Neural Operator: Learning Maps Between Function Spaces

Machine Learning in Drug Discovery: A Review

Highly accurate protein structure prediction with AlphaFold

Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth by Stuart Ritchie

Acceleration of Boltzmann Collision Integral Calculation Using Machine Learning