Natural Language Processing (Almost) from Scratch (2011-02-01T00:00:00.000000Z)

TL;DR

A unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including part-of-speech tagging, chunking, named entity recognition, and semantic role labeling is proposed.

Abstract

We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including part-of-speech tagging, chunking, named entity recognition, and semantic role labeling. This versatility is achieved by trying to avoid task-specific engineering and therefore disregarding a lot of prior knowledge. Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data. This work is then used as a basis for building a freely available tagging system with good performance and minimal computational requirements.

Authors

R. Collobert

6 papers

K. Kavukcuoglu

26 papers

L. Bottou

7 papers

TL;DR

Abstract

Authors

References101 items

Deep Learning for Efficient Discriminative Parsing

Word Representations: A Simple and General Method for Semi-Supervised Learning

Distributional Representations for Handling Sparsity in Supervised Sequence-Labeling

Phrase Clustering for Discriminative Learning

Curriculum learning

Design Challenges and Misconceptions in Named Entity Recognition

Modeling Latent-Dynamic in Shallow Parsing: A Latent Conditional Model with Improved Inference

Structure compilation: trading structure for features

Deep learning via semi-supervised embedding

Semi-Supervised Sequential Labeling and Segmentation Using Giga-Word Scale Unlabeled Data

Simple Semi-supervised Dependency Parsing

Three new graphical models for statistical language modelling

Guided Learning for Bidirectional Sequence Classification

A discriminative language model with pseudo-negative samples

Transductive learning for statistical machine translation

Using Corpus Statistics on Entities to Improve Semi-supervised Relation Extraction from the Web

Learning to Rank with Nonsmooth Cost Functions

Greedy Layer-Wise Training of Deep Networks

Ranking the Best Instances

Semi-Supervised Learning (Adaptive Computation and Machine Learning)

A Fast Learning Algorithm for Deep Belief Nets

Effective Self-Training for Parsing

A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data

Comparing and Combining Finite-State and Context-Free Parsers

Flexible Text Segmentation with Structured Multilabel Classification

Composition of Conditional Random Fields for Transfer Learning

The Necessity of Syntactic Parsing for Semantic Role Labeling

Semantic Role Chunking Combining Complementary Syntactic Views

Generalized Inference with Multiple Semantic Role Labeling Systems

Joint Parsing and Semantic Role Labeling

A Joint Model for Semantic Role Labeling

Semantic Role Labelling with Tree Conditional Random Fields

Contrastive Estimation: Training Log-Linear Models on Unlabeled Data

Voting Between Multiple Data Representations for Text Chunking

The Proposition Bank: An Annotated Corpus of Semantic Roles

RCV1: A New Benchmark Collection for Text Categorization Research

Dynamic conditional random fields: factorized probabilistic models for labeling and segmenting sequence data

SVMTool: A general POS Tagger Generator Based on Support Vector Machines

Head-Driven Statistical Models for Natural Language Parsing

Named Entity Recognition through Classifier Combination

Named Entity Recognition with a Maximum Entropy Approach

Early results for Named Entity Recognition with Conditional Random Fields, Feature Induction and Web-Enhanced Lexicons

Shallow Parsing with Conditional Random Fields

Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network

A Neural Probabilistic Language Model

The Necessity of Parsing for Predicate Argument Recognition

Connectionist language modeling for large vocabulary continuous speech recognition

Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

Chunking with Support Vector Machines

Natural Language Grammar Induction Using a Constituent-Context Model

Automatic Labeling of Semantic Roles

Use of Support Vector Learning for Chunk Identification

A Maximum-Entropy-Inspired Parser

A Novel Use of Statistical Parsing to Extract Information from Text

Transductive Inference for Text Classification using Support Vector Machines

Learning to Order Things

An algorithm for suffix stripping

Multitask Learning

Global training of document processing systems using graph transformer networks

The entropy of English using PPM-based models

Distributional Part-of-Speech Tagging

Class-Based n-gram Models of Natural Language

An Estimate of an Upper Bound for the Entropy of English

Symbolic-Neural Systems and the Use of Hints for Developing Complex Systems

Adaptive Algorithms and Stochastic Approximations

A Framework for the Cooperation of Learning Algorithms

Phoneme recognition using time-delay neural networks

A tutorial on hidden Markov models and selected applications in speech recognition

Learning Quickly When Irrelevant Attributes Abound: A New Linear-Threshold Algorithm

Learning sets of filters using back-propagation

A Grammar of English on Mathematical Principles

Parsing

A convergent gambling estimate of the entropy of English

Continuous speech recognition by statistical methods

Three models for the description of language

Semantic Roles