Skip to content

Read-only version: For security reasons, the Research Information System (FIS) has been temporarily replaced by this read-only version. All public content and full texts remain available; login, editing, submission and statistics are not available. Questions: fis@uni-bamberg.de

Home › Publications › Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference

Publication

Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference

Kadiķis, Emīls; Srivastav, Vaibhav; Klinger, Roman (2024): Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference, in: Bamberg: Otto-Friedrich-Universität, S. 6031–6037.

Details

Faculty/Chair
Faculty/Professorship
Author
Kadiķis, Emīls
Srivastav, Vaibhav
Publisher Information
Otto-Friedrich-Universität : Bamberg
Year of publication
2024
Source/Other editions
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies / Marine Carpuat, Marie-Catherine de Marneffe, Ivan Vladimir Meza Ruiz (Hg.). - Seattle : Association for Computational Linguistics, 2022, S. 6031–6037
Year of first publication
2022
Language
English
Licence
URN
Abstract

The task of natural language inference (NLI), to decide if a hypothesis entails or contradicts a premise, received considerable attention in recent years. All competitive systems build on top of contextualized representations and make use of transformer architectures for learning an NLI model. When somebody is faced with a particular NLI task, they need to select the best model that is available. This is a time-consuming and resource-intense endeavour. To solve this practical problem, we propose a simple method for predicting the performance without actually fine-tuning the model. We do this by testing how well the pre-trained models perform on the aNLI task when just comparing sentence embeddings with cosine similarity to what kind of performance is achieved when training a classifier on top of these embeddings. We show that the accuracy of the cosine similarity approach correlates strongly with the accuracy of the classification approach with a Pearson correlation coefficient of 0.65. Since the similarity is orders of magnitude faster to compute on a given dataset (less than a minute vs. hours), our method can lead to significant time savings in the process of model selection.

GND Keywords
Computerlinguistik
Automatische Sprachanalyse
Inferenz <Künstliche Intelligenz>
Keywords
Abductive Natural Language Inference
DDC Classification
004 Computer science
RVK Classification
ST 306
Peer Reviewed
yes
International Distribution
yes
Open Access Journal
yes
Type
Conferenceobject
URI
Activation date
2024-08-19
Project(s)

Permalink: https://fis.uni-bamberg.de/handle/uniba/96472 · Data as JSON · Cite: BibTeX · RIS

Data as of: 2026-11-16

Restricted read-only access: login, editing and submission are temporarily disabled.