Contract Discovery: Dataset and a Few-shot Semantic Retrieval Challenge with Competitive Baselines (2020-10-08T00:00:00.000000Z)

TL;DR

It is shown that state-of-the-art pretrained encoders fail to provide satisfactory results on the task proposed, and Language Model-based solutions perform better, especially when unsupervised fine-tuning is applied.

Abstract

We propose a new shared task of semantic retrieval from legal texts, in which a so-called contract discovery is to be performed – where legal clauses are extracted from documents, given a few examples of similar clauses from other legal acts. The task differs substantially from conventional NLI and shared tasks on legal information extraction (e.g., one has to identify text span instead of a single document, page, or paragraph). The specification of the proposed task is followed by an evaluation of multiple solutions within the unified framework proposed for this branch of methods. It is shown that state-of-the-art pretrained encoders fail to provide satisfactory results on the task proposed. In contrast, Language Model-based solutions perform better, especially when unsupervised fine-tuning is applied. Besides the ablation studies, we addressed questions regarding detection accuracy for relevant text fragments depending on the number of examples available. In addition to the dataset and reference results, LMs specialized in the legal domain were made publicly available.

Authors

Łukasz Borchmann

4 papers

Dawid Wisniewski

2 papers

Andrzej Gretkowski

1 papers

TL;DR

Abstract

Authors

References53 items

SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles

Efficient Sentence Embedding using Discrete Cosine Transform

Few-shot Text Classification with Distributional Signatures

MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

GEval: Tool for Debugging NLP Datasets and Models

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Search on speech from spoken queries: the Multi-domain International ALBAYZIN 2018 Query-by-Example Spoken Term Detection Evaluation

Combining Similarity and Transformer Methods for Case Law Entailment

Generalizing from a Few Examples

Vector of Locally-Aggregated Word Embeddings (VLAWE): A Novel Document-level Representation

CBOW Is Not All You Need: Combining CBOW with the Compositional Matrix Space Model

No Training Required: Exploring Random Encoders for Sentence Classification

Few-shot classification in named entity recognition task

End-to-End Retrieval in Continuous Space

Zero-training Sentence Embedding via Orthogonal Basis

Learning Universal Sentence Representations with Mean-Max Attention Autoencoder

Extracting Fairness Policies from Legal Documents

Convolutional Neural Network for Universal Sentence Embeddings

Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms

Universal Sentence Encoder

Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning

Deep Contextualized Word Representations

Attention is All you Need

Overview of COLIEE 2017

Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

A Simple but Tough-to-Beat Baseline for Sentence Embeddings

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

A large annotated corpus for learning natural language inference

From Word Embeddings To Document Distances

Skip-Thought Vectors

ActivityNet: A large-scale video benchmark for human activity understanding

GloVe: Global Vectors for Word Representation

Efficient Estimation of Word Representations in Vector Space

Factors affecting relevance judgment: a report from TREC Legal track

Evaluation of information retrieval for E-discovery

An Evaluation Framework for Plagiarism Detection

Information Retrieval: Implementing and Evaluating Search Engines

Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions

Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions

Concept and Context in Legal Information Retrieval

Independent component analysis: algorithms and applications

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil

Improving Language Understanding by Generative Pre-Training

THUMOS challenge: Action recognition with a large number of classes

A Machine Learning Approach to Identifying Sections in Legal Briefs

TREC 2006 Legal Track Overview

National Archives, Records Administration, and Office Of General

Assignment Problems

Assignment Problems

public shareholders generally

Field of Study

Venue Information

Name

Type

URL