Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints (2017-06-01T00:00:00.000000Z)

TL;DR

The evaluation shows that the Attract-Repel method can make use of existing cross-lingual lexicons to construct high-quality vector spaces for a plethora of different languages, facilitating semantic transfer from high- to lower-resource ones.

Abstract

We present Attract-Repel, an algorithm for improving the semantic quality of word vectors by injecting constraints extracted from lexical resources. Attract-Repel facilitates the use of constraints from mono- and cross-lingual resources, yielding semantically specialised cross-lingual vector spaces. Our evaluation shows that the method can make use of existing cross-lingual lexicons to construct high-quality vector spaces for a plethora of different languages, facilitating semantic transfer from high- to lower-resource ones. The effectiveness of our approach is demonstrated with state-of-the-art results on semantic similarity datasets in six languages. We next show that Attract-Repel-specialised vectors boost performance in the downstream task of dialogue state tracking (DST) across multiple languages. Finally, we show that cross-lingual vector spaces produced by our algorithm facilitate the training of multilingual DST models, which brings further performance improvements.

Authors

A. Korhonen

17 papers

N. Mrksic

7 papers

Diarmuid Ó Séaghdha

4 papers

TL;DR

Abstract

Authors

References89 items

Intent detection using semantically enriched word embeddings

HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment

SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity

Adjusting Word Embeddings with Semantic Intensity Orders

Measuring Semantic Similarity of Words Using Concept Networks

Charagram: Embedding Words and Sentences via Character n-grams

Learning Crosslingual Word Embeddings without Bilingual Corpora

Neural Belief Tracker: Data-Driven Dialogue State Tracking

Integrating Distributional Lexical Contrast into Word Embeddings for Antonym-Synonym Distinction

A Network-based End-to-End Trainable Task-oriented Dialogue System

Cross-lingual Models of Word Embeddings: An Empirical Comparison

Counter-fitting Word Vectors to Linguistic Constraints

Many Languages, One Parser

A Dual Embedding Space Model for Document Ranking

A Primer on Neural Network Models for Natural Language Processing

Bilingual Distributed Word Representations from Document-Aligned Comparable Data

Reasoning about Entailment with Neural Attention

Encoding Prior Knowledge with Eigenword Embeddings

Any-language frame-semantic parsing

Trans-gram, Fast Cross-lingual Word-embeddings

Specializing Word Embeddings for Similarity or Relatedness

Monolingual and Cross-Lingual Information Retrieval Models Based on (Bilingual) Word Embeddings

Separated by an Un-common Language: Towards Judgment Language Informed Vector Space Modeling

Learning Semantic Word Embeddings based on Ordinal Knowledge Constraints

Cross-lingual Dependency Parsing Based on Distributed Representations

Hubness and Pollution: Delving into Cross-Space Mapping for Zero-Shot Learning

AutoExtend: Extending Word Embeddings to Embeddings for Synsets and Lexemes

Symmetric Pattern Based Word Embeddings for Improved Word Similarity Prediction

PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification

Multi-domain Dialog State Tracking using Recurrent Neural Networks

Non-distributional Word Vector Representations

From Paraphrase Database to Compositional Paraphrase Model and Back

Bilingual Word Representations with Monolingual Quality in Mind

A Hybrid Distributional and Knowledge-based Model of Lexical Semantics

Ontologically Grounded Multi-sense Representation Learning for Semantic Vector Space Models

Texts in, meaning out: neural language models in semantic similarity task for Russian

Inverted indexing for cross-lingual NLP

Multiview LSA: Representation Learning via Generalized CCA

Leveraging Monolingual Data for Crosslingual Compositional Word Representations

Improving zero-shot learning by mitigating the hubness problem

RC-NET: A General Framework for Incorporating Knowledge into Word Representations

BilBOWA: Fast Bilingual Distributed Representations without Word Alignments

GloVe: Global Vectors for Word Representation

Revisiting Embedding Features for Simple Semi-supervised Learning

A Neural Network for Factoid Question Answering over Paragraphs

Knowledge-Powered Deep Learning for Word Embedding

SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation

Fast and Robust Neural Network Joint Models for Statistical Machine Translation

Tailoring Continuous Word Representations for Dependency Parsing

Dependency-Based Word Embeddings

Improving Lexical Embeddings with Semantic Knowledge

The Second Dialog State Tracking Challenge

Representing Multilingual Data as Linked Data: the Case of BabelNet 2.0

The Multilingual Paraphrase Database

Multilingual Models for Compositional Distributed Semantics

Improving Vector Space Word Representations Using Multilingual Correlation

Robust dialog state tracking using delexicalised recurrent neural networks and unsupervised adaptation

An Autoencoder Approach to Learning Bilingual Word Representations

Multilingual Distributed Representations without Word Alignment

Distributed Representations of Words and Phrases and their Compositionality

Bilingual Word Embeddings for Phrase-Based Machine Translation

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Exploiting Similarities among Languages for Machine Translation

Parsing with Compositional Vector Grammars

PPDB: The Paraphrase Database

POMDP-Based Statistical Spoken Dialog Systems: A Review

BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network

Inducing Crosslingual Distributed Representations of Words

Polarity Inducing Latent Semantic Analysis

Representation Learning: A Review and New Perspectives

Natural Language Processing (Almost) from Scratch

Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

Word Representations: A Simple and General Method for Semi-Supervised Learning

Understanding the difficulty of training deep feedforward neural networks

Efficient Unsupervised Discovery of Word Categories Using Symmetric Patterns and High Frequency Words

The Berkeley FrameNet Project