Integration of Neural Network-Based Symbolic Regression in Deep Learning for Scientific Discovery (2019-12-10T00:00:00.000000Z)

TL;DR

This article uses a neural network-based architecture for symbolic regression called the equation learner (EQL) network and integrates it with other deep learning architectures such that the whole system can be trained end-to-end through backpropagation.

Abstract

Symbolic regression is a powerful technique to discover analytic equations that describe data, which can lead to explainable models and the ability to predict unseen data. In contrast, neural networks have achieved amazing levels of accuracy on image recognition and natural language processing tasks, but they are often seen as black-box models that are difficult to interpret and typically extrapolate poorly. In this article, we use a neural network-based architecture for symbolic regression called the equation learner (EQL) network and integrate it with other deep learning architectures such that the whole system can be trained end-to-end through backpropagation. To demonstrate the power of such systems, we study their performance on several substantially different tasks. First, we show that the neural network can perform symbolic regression and learn the form of several functions. Next, we present an MNIST arithmetic task where a convolutional network extracts the digits. Finally, we demonstrate the prediction of dynamical systems where an unknown parameter is extracted through an encoder. We find that the EQL-based architecture can extrapolate quite well outside of the training data set compared with a standard neural network-based architecture, paving the way for deep learning to be applied in scientific exploration and discovery.

Authors

M. Soljačić

2 papers

Samuel Kim

2 papers

Peter Y. Lu

1 papers

TL;DR

Abstract

Authors

References29 items

Visual Physics: Discovering Physical Laws from Videos

Extracting Interpretable Physical Parameters from Spatiotemporal Systems using Unsupervised Learning

The State of Sparsity in Deep Neural Networks

PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network

Gradient Descent Finds Global Minima of Deep Neural Networks

Neural Arithmetic Logic Units

Unsupervised Learning of Latent Physical Properties Using Perception-Prediction Networks

Learning Equations for Extrapolation and Control

Data-Driven Identification of Parametric Partial Differential Equations

Learning Sparse Neural Networks through L0 Regularization

PDE-Net: Learning PDEs from Data

To prune, or not to prune: exploring the efficacy of pruning for model compression

Variational Dropout Sparsifies Deep Neural Networks

Training Sparse Neural Networks

Extrapolation and learning equations

Data-driven discovery of partial differential equations

Discovering governing equations from data by sparse identification of nonlinear dynamical systems

The Loss Surfaces of Multilayer Networks

Convergence of online gradient method for feedforward neural networks with smoothing L1/2 regularization penalty

Batch gradient method with smoothing L1/2 regularization for training of feedforward neural networks

Representative of L1/2 Regularization among Lq (0 < q ≤ 1) Regularizations: an Experimental Study Based on Phase Diagram

L1/2 regularization

Distilling Free-Form Natural Laws from Experimental Data

Sparse Approximate Solutions to Linear Systems

Genetic programming as a means for programming computers by natural selection

Learning partial differential equations via data discovery and sparse optimization

software available from

“Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude,”

“TensorFlow: Large-scale machine learning on heterogeneous systems,”

Field of Study

Journal Information

Name

Page

Volume

Venue Information

Name

Type

URL

Alternate Names