Skip to content

Read-only version: For security reasons, the Research Information System (FIS) has been temporarily replaced by this read-only version. All public content and full texts remain available; login, editing, submission and statistics are not available. Questions: fis@uni-bamberg.de

Home › Publications › Language Models Are Poor Learners of Directional Inference

Publication

Language Models Are Poor Learners of Directional Inference

Li, Tianyi; Hosseini, Mohammad Javad; Weber, Sabine; u. a. (2022): Language Models Are Poor Learners of Directional Inference, in: Yoav Goldberg, Zornitsa Kozareva, Yue Zhang, u. a. (Hrsg.), Findings of the Association for Computational Linguistics: EMNLP 2022, Association for Computational Linguistics, S. 903–921, doi: 10.18653/v1/2022.findings-emnlp.64.

Details

Author
Li, Tianyi
Hosseini, Mohammad Javad
Steedman, Mark
Title of the compilation
Findings of the Association for Computational Linguistics: EMNLP 2022
Editors
Goldberg, Yoav
Kozareva, Zornitsa
Zhang, Yue
Conference
EMNLP 2022, Dezember 2022, Abu Dhabi
Publisher Information
Association for Computational Linguistics
Year of publication
2022
Language
English
DOI
Abstract

We examine LMs’ competence of directional predicate entailments by supervised fine-tuning with prompts. Our analysis shows that contrary to their apparent success on standard NLI, LMs show limited ability to learn such directional inference; moreover, existing datasets fail to test directionality, and/or are infested by artefacts that can be learnt as proxy for entailments, yielding over-optimistic results. In response, we present BoOQA (Boolean Open QA), a robust multi-lingual evaluation benchmark for directional predicate entailments, extrinsic to existing training sets. On BoOQA, we establish baselines and show evidence of existing LM-prompting models being incompetent directional entailment learners, in contrast to entailment graphs, however limited by sparsity.

Keywords
Language Models
Peer Reviewed
yes
International Distribution
yes
Open Access Journal
yes
Type
Conferenceobject
URI
Activation date
2024-08-12

Permalink: https://fis.uni-bamberg.de/handle/uniba/97222 · Data as JSON · Cite: BibTeX · RIS

Data as of: 2026-11-16

Restricted read-only access: login, editing and submission are temporarily disabled.