TaxoLLaMA: WordNet-based Model for Solving Multiple Lexical Sematic Tasks (2024-03-14T00:00:00.000000Z)

TL;DR

TaxoLLaMA, the everything-in-one model, lightweight due to 4-bit quantization and LoRA, demonstrates very strong zero-shot performance on Lexical Entailment and Taxonomy Construction with no fine-tuning and explores its hidden multilingual and domain adaptation capabilities with a little tuning or few-shot learning.

Abstract

In this paper, we explore the capabilities of LLMs in capturing lexical-semantic knowledge from WordNet on the example of the LLaMA-2-7b model and test it on multiple lexical semantic tasks. As the outcome of our experiments, we present TaxoLLaMA, the everything-in-one model, lightweight due to 4-bit quantization and LoRA. It achieves 11 SotA results, 4 top-2 results out of 16 tasks for the Taxonomy Enrichment, Hypernym Discovery, Taxonomy Construction, and Lexical Entailment tasks. Moreover, it demonstrates very strong zero-shot performance on Lexical Entailment and Taxonomy Construction with no fine-tuning. We also explore its hidden multilingual and domain adaptation capabilities with a little tuning or few-shot learning. All datasets, code, and model are available online at https://github.com/VityaVitalich/TaxoLLaMA

Authors

Viktor Moskvoretskii

2 papers

Irina Nikishina

2 papers

Alexander Panchenko

2 papers

TL;DR

Abstract

Authors

References47 items

Extreme Compression of Large Language Models via Additive Quantization

What Do Llamas Really Think? Revealing Preference Biases in Language Model Representations

Unlearn What You Want to Forget: Efficient Unlearning for LLMs

Llama 2: Open Foundation and Fine-Tuned Chat Models

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

QLoRA: Efficient Finetuning of Quantized LLMs

A Survey on In-context Learning

Event knowledge in large language models: the gap between the impossible and the unlikely

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

Smoothing Entailment Graphs with Language Models

Distilling Hypernymy Relations from Language Models: On the Effectiveness of Zero-Shot Taxonomy Induction

TaxoEnrich: Self-Supervised Taxonomy Completion via Structure-Semantic Representations

LoRA: Low-Rank Adaptation of Large Language Models

Enhancing Taxonomy Completion with Concept Generation via Fusing Relational Representations

Taxonomy Completion via Triplet Matching Network

Taxonomy Construction of Unseen Domains via Graph-based Cross-Domain Knowledge Transfer

Understanding LSTM - a tutorial into Long Short-Term Memory Recurrent Neural Networks

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

A Latent Variable Model for Learning Distributional Relation Vectors

Probabilistic Models of Relational Implication

The Effectiveness of Simple Hybrid Systems for Hypernym Discovery

Every Child Should Have Parents: A Taxonomy Refinement Algorithm Based on Hyperbolic Term Embeddings

Learning Typed Entailment Graphs with Global Soft Constraints

pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference

SeVeN: Augmenting Word Embeddings with Unsupervised Relation Vectors

Unsupervised Learning of Distributional Relation Vectors

Graph Convolutional Neural Networks for Web-Scale Recommender Systems

CRIM at SemEval-2018 Task 9: A Hybrid Approach to Hypernym Discovery

300-sparsans at SemEval-2018 Task 9: Hypernymy as interaction of sparse attributes

SemEval-2018 Task 9: Hypernym Discovery

Specialising Word Vectors for Lexical Entailment

HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment

SemEval-2016 Task 13: Taxonomy Extraction Evaluation (TExEval-2)

SemEval-2016 Task 14: Semantic Taxonomy Enrichment

SemEval-2015 Task 17: Taxonomy Extraction Evaluation (TExEval)

Are Large Language Models Good at Lexical Semantics? A Case of Taxonomy Learning

Predicting Terms in IS-A Relations with Pre-trained Transformers

AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

No clues good clues: out of context Lexical Relation Classification

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: EMNLP 2023 - Industry Track, Singapore, December 6-10, 2023

SMASH: Improving SMAll Language Models' Few-SHot Ability with Prompt-Based Distillation

Hypernym Discovery via a Recurrent Mapping Model

Open-Domain Contextual Link Prediction and its Complementarity with Entailment Graphs

The Graph Neural Network Model

WordNet: An Electronic Lexical Database

llms replace knowledge graphs?

2023. Head-to-tail: How knowl-edgeable are large language models (llm)? aka will

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names