TY - CPAPER TI - Which Demographics do LLMs Default to During Annotation? AU - Schäfer, Johannes AU - Combs, Aidan AU - Bagdon, Christopher Doyle AU - Li, Jiahui AU - Probol, Nadine AU - Greschner, Lynn AU - Papay, Sean AU - Menchaca Resendiz, Yarik AU - Velutharambath, Aswathy AU - Wuehrl, Amelie AU - Weber, Sabine AU - Klinger, Roman PY - 2026 SP - 17331–17348 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/114413 LA - eng AB - Demographics and cultural background of annotators influence the labels they assign in text annotation – for instance, an elderly woman might find it offensive to read a message addressed to a “bro”, but a male teenager might find it appropriate. It is therefore important to acknowledge label variations to not underrepresent members of a society. Two research directions developed out of this observation in the context of using large language models (LLM) for data annotations, namely (1) studying biases and inherent knowledge of LLMs and (2) injecting diversity in the output by manipulating the prompt with demographic information. We combine these two strands of research and ask the question to which demographics an LLM resorts when no demographics is given. To answer this question, we evaluate which attributes of human annotators LLMs inherently mimic. Furthermore, we compare non-demographic conditioned prompts and placebo-conditioned prompts (e.g., “you are an annotator who lives in house number 5”) to demographics-conditioned prompts (“You are a 45 year old man and an expert on politeness annotation. How do you rate {instance}”). We study these questions for politeness and offensiveness annotations on the POPQUORN data set, a corpus created in a controlled manner to investigate human label variations based on demographics which has not been used for LLM-based analyses so far. We observe notable influences related to gender, race, and age in demographic prompting, which contrasts with previous studies that found no such effects KW - - ER - TY - CPAPER TI - Donate or Create? : Comparing Data Collection Strategies for Emotion-labeled Multimodal Social Media Posts AU - Bagdon, Christopher Doyle AU - Combs, Aidan AU - Silberer, Carina AU - Klinger, Roman PY - 2026 SP - 17307–17330 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/114410 LA - eng AB - Accurate modeling of subjective phenomena such as emotion expression requires data annotated with authors’ intentions. Commonly such data is collected by asking study participants to donate and label genuine content produced in the real world, or create content fitting particular labels during the study. Asking participants to create content is often simpler to implement and presents fewer risks to participant privacy than data donation. However, it is unclear if and how study-created content may differ from genuine content, and how differences may impact models. We collect study-created and genuine multimodal social media posts labeled for emotion and compare them on several dimensions, including model performance. We find that compared to genuine posts, study-created posts are longer, rely more on their text and less on their images for emotion expression, and focus more on emotion-prototypical events. The samples of participants willing to donate versus create posts are demographically different. Study-created data is valuable to train models that generalize well to genuine data, but realistic effectiveness estimates require genuine data. KW - - ER - TY - CPAPER TI - Which Demographics do LLMs Default to During Annotation? AU - Schäfer, Johannes AU - Combs, Aidan AU - Bagdon, Christopher Doyle AU - Li, Jiahui AU - Probol, Nadine AU - Greschner, Lynn AU - Papay, Sean AU - Menchaca Resendiz, Yarik AU - Velutharambath, Aswathy AU - Wuehrl, Amelie AU - Weber, Sabine AU - Klinger, Roman PY - 2025 T2 - Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics SP - 17331–17348 PB - Association for Computational Linguistics SN - 979-8-89176-251-0 DO - 10.18653/v1/2025.acl-long.848 UR - https://fis.uni-bamberg.de/handle/uniba/109541 LA - eng AB - Demographics and cultural background of annotators influence the labels they assign in text annotation – for instance, an elderly woman might find it offensive to read a message addressed to a “bro”, but a male teenager might find it appropriate. It is therefore important to acknowledge label variations to not underrepresent members of a society. Two research directions developed out of this observation in the context of using large language models (LLM) for data annotations, namely (1) studying biases and inherent knowledge of LLMs and (2) injecting diversity in the output by manipulating the prompt with demographic information. We combine these two strands of research and ask the question to which demographics an LLM resorts when no demographics is given. To answer this question, we evaluate which attributes of human annotators LLMs inherently mimic. Furthermore, we compare non-demographic conditioned prompts and placebo-conditioned prompts (e.g., “you are an annotator who lives in house number 5”) to demographics-conditioned prompts (“You are a 45 year old man and an expert on politeness annotation. How do you rate {instance}”). We study these questions for politeness and offensiveness annotations on the POPQUORN data set, a corpus created in a controlled manner to investigate human label variations based on demographics which has not been used for LLM-based analyses so far. We observe notable influences related to gender, race, and age in demographic prompting, which contrasts with previous studies that found no such effects KW - - ER - TY - CPAPER TI - Donate or Create? : Comparing Data Collection Strategies for Emotion-labeled Multimodal Social Media Posts AU - Bagdon, Christopher Doyle AU - Combs, Aidan AU - Silberer, Carina AU - Klinger, Roman PY - 2025 T2 - Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics SP - 17307–17330 PB - Association for Computational Linguistics SN - 979-8-89176-251-0 DO - 10.18653/v1/2025.acl-long.847 UR - https://fis.uni-bamberg.de/handle/uniba/109544 LA - eng AB - Accurate modeling of subjective phenomena such as emotion expression requires data annotated with authors’ intentions. Commonly such data is collected by asking study participants to donate and label genuine content produced in the real world, or create content fitting particular labels during the study. Asking participants to create content is often simpler to implement and presents fewer risks to participant privacy than data donation. However, it is unclear if and how study-created content may differ from genuine content, and how differences may impact models. We collect study-created and genuine multimodal social media posts labeled for emotion and compare them on several dimensions, including model performance. We find that compared to genuine posts, study-created posts are longer, rely more on their text and less on their images for emotion expression, and focus more on emotion-prototypical events. The samples of participants willing to donate versus create posts are demographically different. Study-created data is valuable to train models that generalize well to genuine data, but realistic effectiveness estimates require genuine data. KW - - ER - TY - CPAPER TI - “You are an expert annotator” : Automatic Best–Worst-Scaling Annotations for Emotion Intensity Modeling AU - Bagdon, Christopher Doyle AU - Karmalkar, Prathamesh AU - Gurulingappa, Harsha AU - Klinger, Roman PY - 2024 T2 - Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies SP - 7917–7929 PB - Association for Computational Linguistics CY - Mexico City, Mexico UR - https://fis.uni-bamberg.de/handle/uniba/95820 LA - eng ER -