LayoutLM: Pre-training of Text and Layout for Document Image Understanding (2019-12-31T00:00:00.000000Z)

TL;DR

The LayoutLM is proposed to jointly model interactions between text and layout information across scanned document images, which is beneficial for a great number of real-world document image understanding tasks such as information extraction from scanned documents.

Abstract

Pre-training techniques have been verified successfully in a variety of NLP tasks in recent years. Despite the widespread use of pre-training models for NLP applications, they almost exclusively focus on text-level manipulation, while neglecting layout and style information that is vital for document image understanding. In this paper, we propose the LayoutLM to jointly model interactions between text and layout information across scanned document images, which is beneficial for a great number of real-world document image understanding tasks such as information extraction from scanned documents. Furthermore, we also leverage image features to incorporate words' visual information into LayoutLM. To the best of our knowledge, this is the first time that text and layout are jointly learned in a single framework for document-level pre-training. It achieves new state-of-the-art results in several downstream tasks, including form understanding (from 70.72 to 79.27), receipt understanding (from 94.02 to 95.24) and document image classification (from 93.07 to 94.42). The code and pre-trained LayoutLM models are publicly available at https://aka.ms/layoutlm.

Authors

Ming Zhou

14 papers

Shaohan Huang

5 papers

Yiheng Xu

7 papers

TL;DR

Abstract

Authors

References30 items

Modular Multimodal Architecture for Document Classification

Visual Detection with Context for Document Layout Analysis

PubLayNet: Largest Dataset Ever for Document Layout Analysis

Deterministic Routing between Layout Abstractions for Multi-Scale Classification of Visually Rich Documents

RoBERTa: A Robustly Optimized BERT Pretraining Approach

FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents

Graph Convolution for Multimodal Information Extraction from Visually Rich Documents

Chargrid: Towards Understanding 2D Documents

Document Image Classification with Intra-Domain Transfer Learning and Stacked Generalization of Deep Convolutional Neural Networks

DeepDeSRT: Deep Learning for Detection and Structure Recognition of Tables in Document Images

Fast CNN-Based Document Layout Analysis

Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Networks

Cutting the Error by Half: Investigation of Very Deep CNN and Advanced Training Strategies for Document Image Classification

Mask R-CNN

A Table Detection Method for PDF Documents Based on Convolutional Neural Networks

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Deep Residual Learning for Image Recognition

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Evaluation of deep convolutional nets for document image classification and retrieval

Evaluation of SVM, MLP and GMM Classifiers for Layout Analysis of Historical Documents

Fast Document Segmentation Using Contour and X-Y Cut Technique

Building a test collection for complex document information processing

Learning nongenerative grammatical models for document analysis

A Fast Algorithm for Bottom-Up Document Layout Analysis

Document page decomposition by the bounding-box project

The Document Spectrum for Page Layout Analysis

A fast and efficient method for extracting text paragraphs and graphics from unconstrained documents

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Artificial neural networks for document analysis and recognition

Field of Study

Journal Information

Name

Venue Information

Name

Type

URL

Alternate Names