Skip to content

Read-only version: For security reasons, the Research Information System (FIS) has been temporarily replaced by this read-only version. All public content and full texts remain available; login, editing, submission and statistics are not available. Questions: fis@uni-bamberg.de

Home › Publications › Which Demographics do LLMs Default to During Annotation?

Publication

Which Demographics do LLMs Default to During Annotation?

Schäfer, Johannes; Combs, Aidan; Bagdon, Christopher Doyle; u. a. (2025): Which Demographics do LLMs Default to During Annotation?, in: Wanxiang Che, Joyce Nabende, Ekaterina Shutova, u. a. (Hrsg.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, S. 17331–17348, doi: 10.18653/v1/2025.acl-long.848.

Details

Faculty/Chair
Faculty/Professorship
Author
Schäfer, Johannes; Combs, Aidan; Bagdon, Christopher Doyle; Li, Jiahui; ORCID iDProbol, Nadine; Greschner, Lynn; ORCID iDPapay, Sean; Menchaca Resendiz, Yarik; Velutharambath, Aswathy; Wuehrl, Amelie; ORCID iDWeber, Sabine; ORCID iDKlinger, Roman
Title of the compilation
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics
Volume Number/Title
1: Long Papers
Editors
Che, Wanxiang
Nabende, Joyce
Shutova, Ekaterina
Pilehvar, Mohammad Taher
Conference
63rd Annual Meeting of the Association for Computational Linguistics
Publisher Information
Association for Computational Linguistics
Year of publication
2025
ISBN
979-8-89176-251-0
Language
English
DOI
Abstract

Demographics and cultural background of annotators influence the labels they assign in text annotation – for instance, an elderly woman might find it offensive to read a message addressed to a “bro”, but a male teenager might find it appropriate. It is therefore important to acknowledge label variations to not underrepresent members of a society. Two research directions developed out of this observation in the context of using large language models (LLM) for data annotations, namely (1) studying biases and inherent knowledge of LLMs and (2) injecting diversity in the output by manipulating the prompt with demographic information. We combine these two strands of research and ask the question to which demographics an LLM resorts when no demographics is given. To answer this question, we evaluate which attributes of human annotators LLMs inherently mimic. Furthermore, we compare non-demographic conditioned prompts and placebo-conditioned prompts (e.g., “you are an annotator who lives in house number 5”) to demographics-conditioned prompts (“You are a 45 year old man and an expert on politeness annotation. How do you rate {instance}”). We study these questions for politeness and offensiveness annotations on the POPQUORN data set, a corpus created in a controlled manner to investigate human label variations based on demographics which has not been used for LLM-based analyses so far. We observe notable influences related to gender, race, and age in demographic prompting, which contrasts with previous studies that found no such effects

Keywords
-
Peer Reviewed
yes
International Distribution
yes
Open Access Journal
yes
Type
Conferenceobject
URI
Activation date
2025-08-07
Project(s)

Permalink: https://fis.uni-bamberg.de/handle/uniba/109541 · Data as JSON · Cite: BibTeX · RIS

Data as of: 2026-11-16

Restricted read-only access: login, editing and submission are temporarily disabled.