LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding (2020-12-29T00:00:00.000000Z)

TL;DR

LayoutLMv2 architecture with new pre-training tasks to model the interaction among text, layout, and image in a single multi-modal framework and achieves new state-of-the-art results on a wide variety of downstream visually-rich document understanding tasks.

Abstract

Pre-training of text and layout has proved effective in a variety of visually-rich document understanding tasks due to its effective model architecture and the advantage of large-scale unlabeled scanned/digital-born documents. We propose LayoutLMv2 architecture with new pre-training tasks to model the interaction among text, layout, and image in a single multi-modal framework. Specifically, with a two-stream multi-modal Transformer encoder, LayoutLMv2 uses not only the existing masked visual-language modeling task but also the new text-image alignment and text-image matching tasks, which make it better capture the cross-modality interaction in the pre-training stage. Meanwhile, it also integrates a spatial-aware self-attention mechanism into the Transformer architecture so that the model can fully understand the relative positional relationship among different text blocks. Experiment results show that LayoutLMv2 outperforms LayoutLM by a large margin and achieves new state-of-the-art results on a wide variety of downstream visually-rich document understanding tasks, including FUNSD (0.7895 to 0.8420), CORD (0.9493 to 0.9601), SROIE (0.9524 to 0.9781), Kleister-NDA (0.8340 to 0.8520), RVL-CDIP (0.9443 to 0.9564), and DocVQA (0.7295 to 0.8672).

Authors

Lidong Zhou

2 papers

Cha Zhang

4 papers

Wanxiang Che

19 papers

TL;DR

Abstract

Authors

References51 items

Transformers: State-of-the-Art Natural Language Processing

DocVQA: A Dataset for VQA on Document Images

Representation Learning for Information Extraction from Form-like Documents

TRIE: End-to-End Text Reading and Information Extraction for Document Understanding

Robust Layout-aware IE for Visually Rich Documents with Pre-trained Language Models

ZeroShotCeres: Zero-Shot Relation Extraction from Semi-Structured Webpages

Spatial Dependency Parsing for Semi-Structured Document Information Extraction

PICK: Processing Key Information Extraction from Documents using Improved Graph Learning-Convolutional Networks

Kleister: A novel task for Information Extraction involving Long Documents with Complex Layout

UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training

LayoutLM: Pre-training of Text and Layout for Document Image Understanding

Modular Multimodal Architecture for Document Classification

Visual Detection with Context for Document Layout Analysis

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

UNITER: UNiversal Image-TExt Representation Learning

CORD: A Consolidated Receipt Dataset for Post-OCR Parsing

ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction

VL-BERT: Pre-training of Generic Visual-Linguistic Representations

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

PubLayNet: Largest Dataset Ever for Document Layout Analysis

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Deterministic Routing between Layout Abstractions for Multi-Scale Classification of Visually Rich Documents

FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents

VideoBERT: A Joint Model for Video and Language Representation Learning

Graph Convolution for Multimodal Information Extraction from Visually Rich Documents

Chargrid: Towards Understanding 2D Documents

Self-Attention with Relative Position Representations

Document Image Classification with Intra-Domain Transfer Learning and Stacked Generalization of Deep Convolutional Neural Networks

Decoupled Weight Decay Regularization

Fast CNN-Based Document Layout Analysis

Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Networks

Cutting the Error by Half: Investigation of Very Deep CNN and Advanced Training Strategies for Document Image Classification

Mask R-CNN

Aggregated Residual Transformations for Deep Neural Networks

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

SQuAD: 100,000+ Questions for Machine Comprehension of Text

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

Evaluation of deep convolutional nets for document image classification and retrieval

Adam: A Method for Stochastic Optimization

Evaluation of SVM, MLP and GMM Classifiers for Layout Analysis of Historical Documents

Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics

Building a test collection for complex document information processing

Learning nongenerative grammatical models for document analysis

BROS: A PRE-TRAINED LANGUAGE MODEL

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Feature networks for object detection

Artificial neural networks for document analysis and recognition

classiﬁcation of visually rich documents

System Demonstrations

split into 149 for training and 50 for testing. ofﬁcial OCR annotation is directly used with layout information

categories and the task aims to label each word to the right ﬁeld. The evaluation metric is entity-level F1

Field of Study

Venue Information

Name

Type

URL

Alternate Names