Multimodal deep networks for text and image-based document classification (2019-07-15T00:00:00.000000Z)

TL;DR

A multimodal neural network able to learn from word embeddings, computed on text extracted by OCR, and from the image is designed that boosts pure image accuracy by 3% on Tobacco3482 and RVL-CDIP augmented by the new QS-OCR text dataset, even without clean text information.

Abstract

Classification of document images is a critical step for archival of old manuscripts, online subscription and administrative procedures. Computer vision and deep learning have been suggested as a first solution to classify documents based on their visual appearance. However, achieving the fine-grained classification that is required in real-world setting cannot be achieved by visual analysis alone. Often, the relevant information is in the actual text content of the document. We design a multimodal neural network that is able to learn from word embeddings, computed on text extracted by OCR, and from the image. We show that this approach boosts pure image accuracy by 3% on Tobacco3482 and RVL-CDIP augmented by our new QS-OCR text dataset (this https URL), even without clean text information.

Authors

N. Audebert

2 papers

Catherine Herold

1 papers

K. Slimani

1 papers

TL;DR

Abstract

Authors

References40 items

Magnitude: A Fast, Efficient Universal Vector Embedding Utility Package

dhSegment: A Generic Deep-Learning Approach for Document Segmentation

Deep Contextualized Word Representations

Document Image Classification with Intra-Domain Transfer Learning and Stacked Generalization of Deep Convolutional Neural Networks

MobileNetV2: Inverted Residuals and Linear Bottlenecks

Analysis of Convolutional Neural Networks for Document Image Classification

Mimicking Word Embeddings using Subword RNNs

Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Networks

A Simple but Tough-to-Beat Baseline for Sentence Embeddings

Cutting the Error by Half: Investigation of Very Deep CNN and Advanced Training Strategies for Document Image Classification

Identity Documents Classification as an Image Classification Problem

Xception: Deep Learning with Depthwise Separable Convolutions

Embedded Textual Content for Document Image Classification with Convolutional Neural Networks

Enriching Word Vectors with Subword Information

Bag of Tricks for Efficient Text Classification

Hierarchical Attention Networks for Document Classification

Deep Residual Learning for Image Recognition

Multimodal deep learning for robust RGB-D object recognition

Evaluation of deep convolutional nets for document image classification and retrieval

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

GloVe: Global Vectors for Word Representation

Convolutional Neural Networks for Sentence Classification

Structural similarity for document image classification and retrieval

Improving Classification of an Industrial Document Image Database by Combining Visual and Textual Features

CNN Features Off-the-Shelf: An Astounding Baseline for Recognition

Efficient Estimation of Word Representations in Vector Space

Statistical topic models for multi-label document classification

Tesseract: an open-source optical character recognition engine

A survey of document image classification: problem statement, classifier architecture and performance evaluation

One-Class SVMs for Document Classification

Classification of binary document images into textual or nontextual data blocks using network models

Segmentation and classification for mixed text/image documents using neural network

Document Analysis System

Automatic Document Classification

spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing

Usability Engineering

Classification of binary document images into textual or nontextual data blocks using neural network models

Gradient-based learning applied to document recognition

Chapter 4 – The Usability Engineering Lifecycle

A Threshold Selection Method from Gray-Level Histograms

Field of Study

Journal Information

Name

Volume