Skip to content

Read-only version: For security reasons, the Research Information System (FIS) has been temporarily replaced by this read-only version. All public content and full texts remain available; login, editing, submission and statistics are not available. Questions: fis@uni-bamberg.de

Home › Publications › Language Models Are Poor Learners of Directional Inference

Publication

Language Models Are Poor Learners of Directional Inference

Li, Tianyi; Hosseini, Mohammad Javad; Weber, Sabine; u. a. (2025): Language Models Are Poor Learners of Directional Inference, in: Bamberg: Otto-Friedrich-Universität, S. 903–921.

Details

Faculty/Chair
Faculty/Professorship
Author
Li, Tianyi
Hosseini, Mohammad Javad
Steedman, Mark
Publisher Information
Otto-Friedrich-Universität : Bamberg
Year of publication
2025
Source/Other editions
Yoav Goldberg, Zornitsa Kozareva, und Yue Zhang (Hrsg.), Findings of the Association for Computational Linguistics: EMNLP 2022, Association for Computational Linguistics, 2022, S. 903–921
Year of first publication
2022
Language
English
Licence
URN
Abstract

We examine LMs’ competence of directional predicate entailments by supervised fine-tuning with prompts. Our analysis shows that contrary to their apparent success on standard NLI, LMs show limited ability to learn such directional inference; moreover, existing datasets fail to test directionality, and/or are infested by artefacts that can be learnt as proxy for entailments, yielding over-optimistic results. In response, we present BoOQA (Boolean Open QA), a robust multi-lingual evaluation benchmark for directional predicate entailments, extrinsic to existing training sets. On BoOQA, we establish baselines and show evidence of existing LM-prompting models being incompetent directional entailment learners, in contrast to entailment graphs, however limited by sparsity.

Keywords
Language Models
Peer Reviewed
yes
International Distribution
yes
Open Access Journal
yes
Type
Conferenceobject
URI
Activation date
2025-11-10

Permalink: https://fis.uni-bamberg.de/handle/uniba/110866 · Data as JSON · Cite: BibTeX · RIS

Data as of: 2026-11-16

Restricted read-only access: login, editing and submission are temporarily disabled.