Binding Language Models in Symbolic Languages (2022-10-06T00:00:00.000000Z)

TL;DR

Binder is a training-free neural-symbolic framework that maps the task input to a program, which allows binding a unified API of language model (LM) functionalities to a programming language (e.g., SQL, Python) to extend its grammar coverage and thus tackle more diverse questions.

Abstract

Though end-to-end neural approaches have recently been dominating NLP tasks in both performance and ease-of-use, they lack interpretability and robustness. We propose Binder, a training-free neural-symbolic framework that maps the task input to a program, which (1) allows binding a unified API of language model (LM) functionalities to a programming language (e.g., SQL, Python) to extend its grammar coverage and thus tackle more diverse questions, (2) adopts an LM as both the program parser and the underlying model called by the API during execution, and (3) requires only a few in-context exemplar annotations. Specifically, we employ GPT-3 Codex as the LM. In the parsing stage, with only a few in-context exemplars, Codex is able to identify the part of the task input that cannot be answerable by the original programming language, correctly generate API calls to prompt Codex to solve the unanswerable part, and identify where to place the API calls while being compatible with the original grammar. In the execution stage, Codex can perform versatile functionalities (e.g., commonsense QA, information extraction) given proper prompts in the API calls. Binder achieves state-of-the-art results on WikiTableQuestions and TabFact datasets, with explicit output programs that benefit human debugging. Note that previous best systems are all finetuned on tens of thousands of task-specific samples, while Binder only uses dozens of annotations as in-context exemplars without any training. Our code is available at https://github.com/HKUNLP/Binder .

Authors

Caiming Xiong

15 papers

Luke S. Zettlemoyer

3 papers

Dragomir R. Radev

26 papers

TL;DR

Abstract

Authors

References67 items

Scaling Instruction-Finetuned Language Models

Selective Annotation Makes Language Models Better Few-Shot Learners

CodeT: Code Generation with Generated Tests

OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering

TaCube: Pre-computing Data Cubes for Answering Numerical-Reasoning Questions over Tabular Data

Large Language Models are Zero-Shot Reasoners

Neural-Symbolic Models for Logical Queries on Knowledge Graphs

Natural Language to Code Translation with Execution

PaLM: Scaling Language Modeling with Pathways

Training Compute-Optimal Large Language Models

In-Context Learning for Few-Shot Dialogue State Tracking

Evaluating the Text-to-SQL Capabilities of Large Language Models

Chain of Thought Prompting Elicits Reasoning in Large Language Models

UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models

PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models

TAPEX: Table Pre-training via Learning a Neural SQL Executor

Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills

Evaluating Large Language Models Trained on Code

Database reasoning over text

MultiModalQA: Complex Question Answering over Text, Tables and Images

Learn to Combine Linguistic and Symbolic Information for Table-based Fact Verification

Benchmarking Meaning Representations in Neural Semantic Parsing

Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?

On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries

Program Enhanced Fact Verification with Verbalization and Graph Attention Network

Understanding tables with intermediate pre-training

GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing

Language Models are Few-Shot Learners

TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data

Neural Symbolic Reader: Scalable Integration of Distributed and Symbolic Representations for Reading Comprehension

TaPas: Weakly Supervised Table Parsing via Pre-training

LogicalFactChecker: Leveraging Logical Operations for Fact Checking with Graph Module Network

Break It Down: A Question Understanding Benchmark

Neural Module Networks for Reasoning over Text

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Learning Semantic Parsers from Denotations with Latent Structured Alignments and Abstract Programs

TabFact: A Large-scale Dataset for Table-based Fact Verification

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Iterative Search for Weakly Supervised Semantic Parsing

Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts

Learning to Generalize from Sparse and Underspecified Rewards

Neural Modular Control for Embodied Question Answering

Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task

Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing

Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning

Program Synthesis

Macro Grammars and Holistic Triggering for Efficient Semantic Parsing

Learning to Reason: End-to-End Module Networks for Visual Question Answering

Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision

Neural Programmer: Inducing Latent Programs with Gradient Descent

Neural Module Networks

Compositional Semantic Parsing on Semi-Structured Tables

Semantic Parsing on Freebase from Question-Answer Pairs

Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars

Learning to Parse Database Queries Using Inductive Logic Programming

Learning to Generate Programs for Table Fact Verification via Structure-Aware Semantic Parsing

Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

Weakly Supervised Semantic Parsing by Learning from Mistakes

Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Due to page limits, we didn't list the complete previous systems in main results of WIKITQ and TABFACT. In this section, we present comparisons between more previous methods and ours

Generate the B INDER programs with Codex given the in-context examples and the inference query. Codex hyper-parameters can be adjusted based on the task

Before parsing the text into B INDER programs

False negative : the annotation of the gold answer is wrong; or the prediction is actually right while the executor misjudges it as wrong due to limited ability

Bind models with the neural module functionalities in B INDER program

A syntactic neural model for general-purpose code generation

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names