Bridging CNNs, RNNs, and Weighted Finite-State Machines (2018-05-15T00:00:00.000000Z)

TL;DR

SoPa combines neural representation learning with weighted finite-state automata (WFSAs) to learn a soft version of traditional surface patterns, and it is shown that SoPa is an extension of a one-layer CNN, and that such CNNs are equivalent to a restricted version of SoPa, and accordingly, to arestricted form of WFSA.

Abstract

Recurrent and convolutional neural networks comprise two distinct families of models that have proven to be useful for encoding natural language utterances. In this paper we present SoPa, a new model that aims to bridge these two approaches. SoPa combines neural representation learning with weighted finite-state automata (WFSAs) to learn a soft version of traditional surface patterns. We show that SoPa is an extension of a one-layer CNN, and that such CNNs are equivalent to a restricted version of SoPa, and accordingly, to a restricted form of WFSA. Empirically, on three text classification tasks, SoPa is comparable or better than both a BiLSTM (RNN) baseline and a CNN baseline, and is particularly useful in small data settings.

Authors

Noah A. Smith

34 papers

Roy Schwartz

7 papers

Sam Thomson

2 papers

TL;DR

Abstract

Authors

References65 items

Recurrent Neural Networks as Weighted Language Recognizers

Recurrent Additive Networks

Convolutional Sequence to Sequence Learning

The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Task

Understanding Neural Networks through Representation Erasure

Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling

Inside-Outside and Forward-Backward Algorithms Are Just Backprop (tutorial paper)

Rationalizing Neural Predictions

Symmetric Patterns and Coordinations: Fast and Enhanced Representations of Verbs and Adjectives

Weighting Finite-State Transductions With Neural Context

A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories

Improving Hypernymy Detection with an Integrated Path-based and Distributional Method

Representation of Linguistic Form and Function in Recurrent Neural Networks

“Why Should I Trust You?”: Explaining the Predictions of Any Classifier

A Primer on Neural Network Models for Natural Language Processing

Evaluation of Word Vector Representations by Subspace Alignment

Character-Aware Neural Language Models

Molding CNNs for text: non-linear, non-consecutive convolutions

Modeling Word Forms Using Latent Underlying Morphs and Phonology

Multichannel Variable-Size Convolution for Sentence Classification

How Well Do Distributional Models Capture Different Types of Semantic Knowledge?

Symmetric Pattern Based Word Embeddings for Improved Word Similarity Prediction

Deep Unordered Composition Rivals Syntactic Methods for Text Classification

Understanding Neural Networks Through Deep Visualization

Bayesian Optimization of Text Representations

Adam: A Method for Stochastic Optimization

GloVe: Global Vectors for Word Representation

Neural Machine Translation by Jointly Learning to Align and Translate

Convolutional Neural Networks for Sentence Classification

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Linguistic Structured Sparsity in Text Categorization

Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps

Visualizing and Understanding Convolutional Networks

Hidden factors and hidden topics: understanding rating dimensions with review text

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

WFST Enabled Solutions to ASR Problems: Beyond HMM Decoding

Scikit-learn: Machine Learning in Python

Multi-Level Structured Models for Document-Level Sentiment Classification

Enhanced Sentiment Learning Using Twitter Hashtags and Smileys

ICWSM - A Great Catchy Name: Semi-Supervised Recognition of Sarcastic Sentences in Online Product Reviews

Unsupervised Discovery of Generic Relationships Using Pattern Clusters and its Evaluation by Automatically Generated SAT Analogy Questions

Juicer: A Weighted Finite-State Transducer Speech Decoder

Unsupervised named-entity extraction from the Web: An experimental study

Identifying Synonyms among Distributionally Similar Words

Parameter Estimation for Probabilistic Finite-State Transducers

Hidden Markov model interpretations of neural networks

Semiring Parsing

The Kleene-Schützenberger Theorem for Formal Power Series in Partially Commuting Variables

Long Short-Term Memory

Finite-State Transducers in Language and Speech Processing

Automatic Acquisition of Hyponyms from Large Text Corpora

On the computational power of neural nets

Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks

Finding Structure in Time

Finite State Automata and Simple Recurrent Networks

Error bounds for convolutional codes and an asymptotically optimum decoding algorithm

Statistical Inference for Probabilistic Functions of Finite State Markov Chains

On the Definition of a Family of Automata

Modeling Skip-Grams for Event Detection with Convolutional Neural Networks

2015) with a batch size of 150. We use 300-dimensional GloVe 840B embeddings (Pennington et al., 2014) normalized to unit length

A non-parametric model for the discovery of inflectional paradigms from plain text using graphical models over strings

Rational and Recognisable Power Series

The MIT finite-state transducer toolkit for speech and language processing

Weighted finite-state transducers in speech recognition

Gradient-based learning applied to document recognition

Field of Study

Venue Information

Name

Type

URL

Alternate Names