Semantic Probabilistic Layers for Neuro-Symbolic Learning (2022-06-01T00:00:00.000000Z)

TL;DR

This work designs a predictive layer for structured-output prediction that can be plugged into any neural network guaranteeing its predictions are consistent with a set of predefined symbolic constraints, and empirically demonstrates that SPLs outperform these competitors in terms of accuracy on challenging SOP tasks.

Abstract

We design a predictive layer for structured-output prediction (SOP) that can be plugged into any neural network guaranteeing its predictions are consistent with a set of predefined symbolic constraints. Our Semantic Probabilistic Layer (SPL) can model intricate correlations, and hard constraints, over a structured output space all while being amenable to end-to-end learning via maximum likelihood. SPLs combine exact probabilistic inference with logical reasoning in a clean and modular way, learning complex distributions and restricting their support to solutions of the constraint. As such, they can faithfully, and efficiently, model complex SOP tasks beyond the reach of alternative neuro-symbolic approaches. We empirically demonstrate that SPLs outperform these competitors in terms of accuracy on challenging SOP tasks including hierarchical multi-label classification, pathfinding and preference learning, while retaining perfect constraint satisfaction.

Authors

Stefano Teso

2 papers

Guy Van den Broeck

3 papers

Kai-Wei Chang

20 papers

TL;DR

Abstract

Authors

References100 items

PYLON: A PyTorch Framework for Learning with Constraints

Deep Learning with Logical Constraints

Efficient and Reliable Probabilistic Interactive Learning with Structured Outputs

Neuro-Symbolic Entropy Regularization

GreaseLM: Graph REASoning Enhanced Language Models for Question Answering

Lossless Compression with Probabilistic Circuits

Solving Marginal MAP Exactly by Probabilistic Circuit Transformations

MultiplexNet: Towards Fully Satisfied Logical Constraints in Neural Networks

Conditional sum-product networks: Modular probabilistic circuits via gate functions

Strudel: A fast and accurate learner of structured-decomposable probabilistic circuits

On the Opportunities and Risks of Foundation Models

A review of some techniques for inclusion of domain-knowledge into deep neural networks

Tractable Regularization of Probabilistic Circuits

Implicit MLE: Backpropagating Through Discrete Exponential Family Distributions

Neuro-symbolic artificial intelligence

Multi-Label Classification Neural Networks with Hard Logical Constraints

Probabilistic Circuits for Variational Inference in Discrete Graphical Models

Coherent Hierarchical Multi-Label Classification Networks

Efficient Generation of Structured Objects with Constrained Adversarial Networks

Joints in Random Forests

Counterexample-Guided Learning of Monotonic Neural Networks

Handling Missing Data in Decision Trees: A Probabilistic Approach

Efficient Second-Order TreeCRF for Neural Dependency Parsing

Einsum Networks: Fast and Scalable Learning of Tractable Probabilistic Circuits

Learning Weighted Model Integration Distributions

Injecting Numerical Reasoning Skills into Language Models

Differentiation of Blackbox Combinatorial Solvers

PyTorch: An Imperative Style, High-Performance Deep Learning Library

On Tractable Computation of Expected Predictions

Learning and Inference for Structured Prediction: A Unifying Perspective

Random Sum-Product Networks: A Simple and Effective Approach to Probabilistic Deep Learning

DL2: Training and Querying Neural Networks with Logic

Conditional Sum-Product Networks: Imposing Structure on Deep Probabilistic Architectures

On a Convex Logic Fragment for Learning and Reasoning

MolGAN: An implicit generative model for small molecular graphs

DeepProbLog: Neural Probabilistic Logic Programming

Learning Constraints From Examples

Explainable Prediction of Medical Codes from Clinical Text

A Semantic Loss Function for Deep Learning with Symbolic Knowledge

Semantic-based regularization for learning and inference

Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models

Visualizing and understanding Sum-Product Networks

Concrete Problems in AI Safety

On the Latent Variable Interpretation in Sum-Product Networks

Deep Residual Learning for Image Recognition

Implementing Efficient All Solutions SAT Solvers

The importance of the label hierarchy in hierarchical multi-label classification

Simplifying, Regularizing and Strengthening Sum-Product Network Structure Learning

ProbLog2: Probabilistic Logic Programming

A survey on multi‐output regression

A Top-Down Compiler for Sentential Decision Diagrams

Neural CRF Parsing

CRF Learning with CNN Features for Image Segmentation

The importance of the label hierarchy in hierarchical multi-label classification

Cutset Networks: A Simple, Tractable, and Scalable Approach for Improving the Accuracy of Chow-Liu Trees

Improved multi-level protein–protein interaction prediction with semantic-based regularization

Dynamic Minimization of Sentential Decision Diagrams

A large-scale evaluation of computational protein function prediction

On label dependence and loss minimization in multi-label classification

Sum-product networks: A new deep architecture

Learning Latent Tree Graphical Models

Probabilistic Graphical Models: Principles and Techniques - Adaptive Computation and Machine Learning

New Compilation Languages Based on Structured Decomposability

Multi-Label Classification: An Overview

Support vector machine learning for interdependent and structured output spaces

A Knowledge Compilation Map

Ordered binary decision diagrams

Approximating discrete probability distributions with dependence trees

RELATED WORKS In this section, we position SPLs against state-of-the-art In-depth surveys on this topic can be found in [Dash et

q(Yi; θi). The neuro-symbolic entropy (NESYENT) [Ahmed et al., 2022

A Compositional Atlas of Tractable Circuit Operations for Probabilistic Inference

Scallop: From Probabilistic Deductive Databases to Scalable Differentiable Reasoning

Note that O(|q||c|) is a loose upperbound and the size of r is in practice smaller [Vergari et al., 2021

Probabilistic Circuits: A Unifying Framework for Tractable Probabilistic Models∗

Fast and Accurate Neural CRF Constituency Parsing

guarantee that the predictions will be consistent at test time. Common losses include translating logical constraints into a differentiable fuzzy logic [Diligenti et