Text Segmentation by Cross Segment Attention
TL
TL;DR
This work proposes three transformer-based architectures and provides comprehensive comparisons with previously proposed approaches on three standard datasets, and establishes a new state-of-the-art, reducing in particular the error rates by a large margin in all cases.
Abstract
Document and discourse segmentation are two fundamental NLP tasks pertaining to breaking up text into constituents, which are commonly used to help downstream tasks such as information retrieval or text summarization. In this work, we propose three transformer-based architectures and provide comprehensive comparisons with previously proposed approaches on three standard datasets. We establish a new state-of-the-art, reducing in particular the error rates by a large margin in all cases. We further analyze model sizes and find that we can build models with many fewer parameters while keeping good performance, thus facilitating real-world applications.
References39 items
1
Semantic Label Smoothing for Sequence to Sequence Problems
2
Does label smoothing mitigate label noise?
3
SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization
4
BeamSeg: A Joint Model for Multi-Document Segmentation and Topic Identification
5