TAPEX: Table Pre-training via Learning a Neural SQL Executor (2021-07-16T00:00:00.000000Z)

TL;DR

TAPEX is proposed to show that table pre-training can be achieved by learning a neural SQL executor over a synthetic corpus, which is obtained by automatically synthesizing executable SQL queries and their execution outputs.

Abstract

Recent progress in language model pre-training has achieved a great success via leveraging large-scale unstructured textual data. However, it is still a challenge to apply pre-training on structured tabular data due to the absence of large-scale high-quality tabular data. In this paper, we propose TAPEX to show that table pre-training can be achieved by learning a neural SQL executor over a synthetic corpus, which is obtained by automatically synthesizing executable SQL queries and their execution outputs. TAPEX addresses the data scarcity challenge via guiding the language model to mimic a SQL executor on the diverse, large-scale and high-quality synthetic corpus. We evaluate TAPEX on four benchmark datasets. Experimental results demonstrate that TAPEX outperforms previous table pre-training approaches by a large margin and achieves new state-of-the-art results on all of them. This includes the improvements on the weakly-supervised WikiSQL denotation accuracy to 89.5% (+2.3%), the WikiTableQuestions denotation accuracy to 57.5% (+4.8%), the SQA denotation accuracy to 74.5% (+3.5%), and the TabFact accuracy to 84.2% (+3.2%). To our knowledge, this is the first work to exploit table pre-training via synthetic executable programs and to achieve new state-of-the-art results on various downstream tasks. Our code can be found at https://github.com/microsoft/Table-Pretraining.

Authors

Qian Liu

4 papers

Jian-Guang Lou

6 papers

Zeqi Lin

2 papers

TL;DR

Abstract

Authors

References50 items

Awakening Latent Grounding from Pretrained Language Models for Semantic Parsing

Logic-level Evidence Retrieval and Graph-based Verification Network for Table-based Fact Verification

Learning to Synthesize Data for Semantic Parsing

UniK-QA: Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answering

Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training

Learn to Combine Linguistic and Symbolic Information for Table-based Fact Verification

Table Fact Verification with Structure-Aware Transformer

Structure-Grounded Pretraining for Text-to-SQL

TUTA: Tree-based Transformers for Generally Structured Table Pre-training

On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries

Program Enhanced Fact Verification with Verbalization and Graph Attention Network

Understanding tables with intermediate pre-training

GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing

Grounded Adaptation for Zero-shot Executable Semantic Parsing

Language Models are Few-Shot Learners

TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data

TaPas: Weakly Supervised Table Parsing via Pre-training

LogicalFactChecker: Leveraging Logical Operations for Fact Checking with Graph Module Network

UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training

How Far are We from Effective Context Modeling ? An Exploratory Study on Semantic Parsing in Context

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

A Split-and-Recombine Approach for Follow-up Query Analysis

A Discrete Hard EM Approach for Weakly Supervised Question Answering

Learning Semantic Parsers from Denotations with Latent Structured Alignments and Abstract Programs

TabFact: A Large-scale Dataset for Table-based Fact Verification

Answering Conversational Questions on Structured Data without Logical Forms

Using Database Rule for Weak Supervised Text-to-SQL Generation

Learning Deep Transformer Models for Machine Translation

Iterative Search for Weakly Supervised Semantic Parsing

fairseq: A Fast, Extensible Toolkit for Sequence Modeling

Learning to Generalize from Sparse and Underspecified Rewards

Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task

Knowledge-Aware Conversational Semantic Parsing Over Web Tables

Robust Text-to-SQL Generation with Execution-Guided Decoding

Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing

Weakly Supervised Semantic Parsing with Abstract Examples

Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning

Macro Grammars and Holistic Triggering for Efficient Semantic Parsing

Attention is All you Need

Search-based Neural Structured Learning for Sequential Question Answering

Learning a Natural Language Interface with Neural Programmer

Neural Programmer: Inducing Latent Programs with Gradient Descent

Compositional Semantic Parsing on Semi-Structured Tables

SCoRe: Pre-Training for Context Representation in Conversational Semantic Parsing

Weakly Supervised Semantic Parsing by Learning from Mistakes

Unified Open-Domain Question Answering with Structured and Unstructured Knowledge

question as the “sentence

TURL: table understanding through representation learning

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names