Semantic Probabilistic Layers for Neuro-Symbolic Learning

Published in

Neural Information Processing Systems(2022)

External Links:

Generate Graph DownloadPDF

TL;DR

This work designs a predictive layer for structured-output prediction that can be plugged into any neural network guaranteeing its predictions are consistent with a set of predefined symbolic constraints, and empirically demonstrates that SPLs outperform these competitors in terms of accuracy on challenging SOP tasks.

Abstract

We design a predictive layer for structured-output prediction (SOP) that can be plugged into any neural network guaranteeing its predictions are consistent with a set of predefined symbolic constraints. Our Semantic Probabilistic Layer (SPL) can model intricate correlations, and hard constraints, over a structured output space all while being amenable to end-to-end learning via maximum likelihood. SPLs combine exact probabilistic inference with logical reasoning in a clean and modular way, learning complex distributions and restricting their support to solutions of the constraint. As such, they can faithfully, and efficiently, model complex SOP tasks beyond the reach of alternative neuro-symbolic approaches. We empirically demonstrate that SPLs outperform these competitors in terms of accuracy on challenging SOP tasks including hierarchical multi-label classification, pathfinding and preference learning, while retaining perfect constraint satisfaction.

Authors

Stefano Teso

2 papers

Guy Van den Broeck

3 papers

Kai-Wei Chang

20 papers

References100 items

PYLON: A PyTorch Framework for Learning with Constraints

Deep Learning with Logical Constraints

Efficient and Reliable Probabilistic Interactive Learning with Structured Outputs

Neuro-Symbolic Entropy Regularization

GreaseLM: Graph REASoning Enhanced Language Models for Question Answering

Semantic Probabilistic Layers for Neuro-Symbolic Learning

Published in

Neural Information Processing Systems(2022)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Stefano Teso

2 papers

Guy Van den Broeck

3 papers

Kai-Wei Chang

20 papers

References100 items

PYLON: A PyTorch Framework for Learning with Constraints

Deep Learning with Logical Constraints

Efficient and Reliable Probabilistic Interactive Learning with Structured Outputs

Neuro-Symbolic Entropy Regularization

GreaseLM: Graph REASoning Enhanced Language Models for Question Answering

Kareem Ahmed

1 papers

Antonio Vergari

1 papers

Lossless Compression with Probabilistic Circuits

Solving Marginal MAP Exactly by Probabilistic Circuit Transformations

MultiplexNet: Towards Fully Satisfied Logical Constraints in Neural Networks

Conditional sum-product networks: Modular probabilistic circuits via gate functions

Strudel: A fast and accurate learner of structured-decomposable probabilistic circuits

On the Opportunities and Risks of Foundation Models

A review of some techniques for inclusion of domain-knowledge into deep neural networks

Tractable Regularization of Probabilistic Circuits

Implicit MLE: Backpropagating Through Discrete Exponential Family Distributions

Neuro-symbolic artificial intelligence

Multi-Label Classification Neural Networks with Hard Logical Constraints

Probabilistic Circuits for Variational Inference in Discrete Graphical Models

Coherent Hierarchical Multi-Label Classification Networks

Efficient Generation of Structured Objects with Constrained Adversarial Networks

Joints in Random Forests

Counterexample-Guided Learning of Monotonic Neural Networks

Handling Missing Data in Decision Trees: A Probabilistic Approach

Efficient Second-Order TreeCRF for Neural Dependency Parsing

Einsum Networks: Fast and Scalable Learning of Tractable Probabilistic Circuits

Learning Weighted Model Integration Distributions

Injecting Numerical Reasoning Skills into Language Models

Differentiation of Blackbox Combinatorial Solvers

PyTorch: An Imperative Style, High-Performance Deep Learning Library

On Tractable Computation of Expected Predictions

Learning and Inference for Structured Prediction: A Unifying Perspective

Random Sum-Product Networks: A Simple and Effective Approach to Probabilistic Deep Learning

DL2: Training and Querying Neural Networks with Logic

Conditional Sum-Product Networks: Imposing Structure on Deep Probabilistic Architectures

On a Convex Logic Fragment for Learning and Reasoning

MolGAN: An implicit generative model for small molecular graphs

DeepProbLog: Neural Probabilistic Logic Programming

Learning Constraints From Examples

Explainable Prediction of Medical Codes from Clinical Text

A Semantic Loss Function for Deep Learning with Symbolic Knowledge

Semantic-based regularization for learning and inference

Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models

Visualizing and understanding Sum-Product Networks

Concrete Problems in AI Safety

On the Latent Variable Interpretation in Sum-Product Networks

Deep Residual Learning for Image Recognition

Implementing Efficient All Solutions SAT Solvers

The importance of the label hierarchy in hierarchical multi-label classification

Simplifying, Regularizing and Strengthening Sum-Product Network Structure Learning

ProbLog2: Probabilistic Logic Programming

A survey on multi‐output regression

A Top-Down Compiler for Sentential Decision Diagrams

Neural CRF Parsing

CRF Learning with CNN Features for Image Segmentation

The importance of the label hierarchy in hierarchical multi-label classification

Cutset Networks: A Simple, Tractable, and Scalable Approach for Improving the Accuracy of Chow-Liu Trees

Improved multi-level protein–protein interaction prediction with semantic-based regularization

Dynamic Minimization of Sentential Decision Diagrams

A large-scale evaluation of computational protein function prediction

On label dependence and loss minimization in multi-label classification

Sum-product networks: A new deep architecture

Learning Latent Tree Graphical Models

Probabilistic Graphical Models: Principles and Techniques - Adaptive Computation and Machine Learning

New Compilation Languages Based on Structured Decomposability

Multi-Label Classification: An Overview

Support vector machine learning for interdependent and structured output spaces

A Knowledge Compilation Map

Ordered binary decision diagrams

Approximating discrete probability distributions with dependence trees

RELATED WORKS In this section, we position SPLs against state-of-the-art In-depth surveys on this topic can be found in [Dash et

q(Yi; θi). The neuro-symbolic entropy (NESYENT) [Ahmed et al., 2022

A Compositional Atlas of Tractable Circuit Operations for Probabilistic Inference

Scallop: From Probabilistic Deductive Databases to Scalable Differentiable Reasoning

Note that O(|q||c|) is a loose upperbound and the size of r is in practice smaller [Vergari et al., 2021

Probabilistic Circuits: A Unifying Framework for Tractable Probabilistic Models∗

Fast and Accurate Neural CRF Constituency Parsing

guarantee that the predictions will be consistent at test time. Common losses include translating logical constraints into a differentiable fuzzy logic [Diligenti et

2018] in employing a 5-layer

On Theoretical Properties of Sum-Product Networks

We can further guarantee that the circuit’s output are normalized probabilities by enforcing the parameters ω of each sum unit to form a convex combination

PREFLIB: A Library for Preferences

Bridging logic and kernel machines

A Literature Survey on Algorithms for Multi-label Learning

Products of Experts

A Tutorial on Energy-Based Learning

Binary Decision Diagrams

An introduction to hidden Markov models

Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence SDD: A New Canonical Representation of Propositional Knowledge Bases

International Journal of Approximate Reasoning Multi-dimensional Classification with Bayesian Networks

gating function is less striking instead, with a two-layer gating function achieving highest performances on 9 datasets

Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content

Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation

Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

D3. Consistent : It should always output predictions that are consistent with the prespeciﬁed symbolic knowledge, i.e., for all x and y , if ( x , y ) = K then p ( y x ) = 0

b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

D4. General : It should support rich logical constraints over the labels expressed in some formal language

D5. Modular : It should be applicable to any off-the-shelf (and possibly pretrained) neural network in a modular fashion, enabling end-to-end learning and rapid prototyping

100

If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots

Field of Study

Computer Science

Venue Information

Name

Neural Information Processing Systems

Type

conference

URL

http://neurips.cc/

Alternate Names

Neural Inf Process Syst
NeurIPS
NIPS