Language Modelling

Language Modeling is the task of predicting the next word or character in a document. This technique can be used to train language models that can further be applied to a wide range of natural language tasks like text generation, text classification, and question answering. The common types of language modeling techniques involve: N-gram Language Models Neural Langauge Models A model's language modeling capability is measured using cross-entropy and perplexity. Some datasets to evaluate language modeling are WikiText-103, One Billion Word, Text8, C4, among others. One of the most recent popular benchmarks to evaluate language modeling capabilities is called SuperGLUE. Some popular and notable state-of-the-art language models, include: GPT-3 Megatron-LM BERT Check below for all state-of-the-art models. Here are some additional readings to go deeper on the task: Language Modeling - Lena Voita ( Image credit: Exploring the Limits of Language Modeling )