TY - CPAPER TI - A Formal Model of Lexical Negation in Discrete Communication AU - Golecki, Mikolaj AU - Bernard, Timothée PY - 2026 SP - 135-147 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/117790 LA - eng AB - Natural languages distinguish between objects satisfying a predicate and those satisfying its complement, often using a simple lexical negation. In emergent communication, however, a system may separate positive and negative meanings without developing a single negation marker: polarity may be tied to the thing being negated, distributed across multiple symbols, or reflected only in accidental correlations. We propose an information-theoretic account of negation applicable to discrete communication systems. We first study these metrics on toy languages, showing how they can be used to detect various patterns and how these are indeed related to negation. We then apply them to languages emerging in a signalling game with setcomplement relations, under pressures known to favour compositionality. The results suggest that these pressures can produce high-scoring polarity-sensitive features, but not necessarily a compositional encoding of negation. More generally, we highlight both the usefulness and the limits of targeted semantic diagnostics for analysing structure in emergent languages. KW - _ ER - TY - CPAPER TI - Actor Identification in Discourse : A Challenge for LLMs? AU - Barić, Ana AU - Padó, Sebastian AU - Papay, Sean PY - 2026 SP - 64–70 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/115586 LA - eng AB - The identification of political actors who put forward claims in public debate is a crucial step in the construction of discourse networks, which are helpful to analyze societal debates. Actor identification is, however, rather challenging: Often, the locally mentioned speaker of a claim is only a pronoun (“He proposed that [claim]”), so recovering the canonical actor name requires discourse understanding. We compare a traditional pipeline of dedicated NLP components (similar to those applied to the related task of coreference) with a LLM, which appears a good match for this generation task. Evaluating on a corpus of German actors in newspaper reports, we find surprisingly that the LLM performs worse. Further analysis reveals that the LLM is very good at identifying the right reference, but struggles to generate the correct canonical form. This points to an underlying issue in LLMs with controlling generated output. Indeed, a hybrid model combining the LLM with a classifier to normalize its output substantially outperforms both initial models. KW - Actor Identification ER - TY - CPAPER TI - Automatic Analysis of Political Debates and Manifestos : Successes and Challenges AU - Ceron, Tanise AU - Barić, Ana AU - Blessing, André AU - Haunss, Sebastian AU - Kuhn, Jonas AU - Lapesa, Gabriella AU - Padó, Sebastian AU - Papay, Sean AU - Zauchner, Patricia F. PY - 2026 SP - 71–88 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/115585 LA - eng AB - The opinions of political actors (e.g., politicians, parties, organizations) expressed through claims are the core elements of political debates and decision-making. Political actors communicate through different channels: parties publish manifestos for major elections, while individual actors make statements on a day-to-day basis as reflected in the media. These two channels offer different approaches for analysis: Manifestos, on the one hand, are useful to characterize the parties’ positions at a global ideological level over time. In contrast, individual statements can be collected to analyze debates in particular policy domains on a fine-grained level, in terms of individual actors and claims. In this article, we summarize a series of studies we have carried out. We apply NLP-driven (semi-)automatic analyses on these two channels and compare their potentials and challenges. The fine-grained analysis yields rich insights into the communication but comes at the cost of three challenges: (a) a substantial hunger for manual annotation, introducing practical hurdles for analysis both within and across languages; (b) difficulties in claim classification arising from the uneven frequency distribution over the theory-based annotation schemas; (c) the need to map actor mentions onto canonical versions. Manifesto-based analysis avoids these challenges to a substantial extent when a more coarse-grained analysis of party positions is sufficient. We highlight the benefits and challenges of both approaches, and conclude by outlining perspectives for addressing the challenges in future research. KW - Claim identification KW - discourse network analysis KW - party positioning KW - argument mining ER - TY - CPAPER TI - Medical Summarization in Practice : Design, Deployment, and Analysis of a Clinical Summarization System for a German Hospital AU - Rauf, Moiz AU - Papay, Sean PY - 2026 SP - 455-466 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/115584 LA - eng AB - Through the course of hospital treatment, a large number of electronic health records (EHRs) are created for a patient, detailingaspects of care history such as lab results, physician notes, and treatments administered.At the conclusion of treatment, this collection of EHRs must be summarized into a discharge summary,describing the course or care clearly and cohesively.In this paper, we present the design and development of a clinical summarization system integrated into a live German hospital workflowto help with the generation of discharge summaries.We first describe the system, its components, and its context of use within a hospital,before performing a number of experiments to gain insights into how best to use and evaluate our system.We investigate summarization performance across multiple input encoding strategies, compare expert judgments against automatic evaluation of summaries,and analyze the consistency of model summaries across multiple text generations.This work not only acts as a case study to demonstrate the feasibility of LLM integration into healthcare infrastructure,but also provides actionable insights into the use and evaluation of such systems. KW - Clinical Summarization System ER - TY - CPAPER TI - Medical Summarization in Practice : Design, Deployment, and Analysis of a Clinical Summarization System for a German Hospital AU - Rauf, Moiz AU - Papay, Sean PY - 2026 T2 - Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics SP - 455–466 PB - Association for Computational Linguistics SN - 979-8-89176-384-5 DO - 10.18653/v1/2026.eacl-industry.34 UR - https://fis.uni-bamberg.de/handle/uniba/115295 LA - eng AB - Through the course of hospital treatment, a large number of electronic health records (EHRs) are created for a patient, detailingaspects of care history such as lab results, physician notes, and treatments administered.At the conclusion of treatment, this collection of EHRs must be summarized into a discharge summary,describing the course or care clearly and cohesively.In this paper, we present the design and development of a clinical summarization system integrated into a live German hospital workflowto help with the generation of discharge summaries.We first describe the system, its components, and its context of use within a hospital,before performing a number of experiments to gain insights into how best to use and evaluate our system.We investigate summarization performance across multiple input encoding strategies, compare expert judgments against automatic evaluation of summaries,and analyze the consistency of model summaries across multiple text generations.This work not only acts as a case study to demonstrate the feasibility of LLM integration into healthcare infrastructure,but also provides actionable insights into the use and evaluation of such systems. KW - Clinical Summarization System ER - TY - CPAPER TI - Strategies for political-statement segmentation and labelling in unstructured text AU - Nikolaev, Dmitry AU - Papay, Sean PY - 2026 T2 - Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities SP - 437–451 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/115245 LA - eng AB - Analysis of parliamentary speeches and political-party manifestos has become an integral area of computational study of political texts. While speeches have been overwhelmingly analysed using unsupervised methods, a large corpus of manifestos with by-statement political-stance labels has been created by the participants of the MARPOR project. It has been recently shown that these labels can be predicted by a neural model; however, the current approach relies on provided statement boundaries, limiting out-of-domain applicability. In this work, we propose and test a range of unified split-and-label frameworks—based on linear-chain CRFs, fine-tuned text-to-text models, and the combination of in-context learning with constrained decoding—that can be used to jointly segment and classify statements from raw textual data. We show that our approaches achieve competitive accuracy when applied to raw text of political manifestos, and then demonstrate the research potential of our method by applying it to the records of the UK House of Commons and tracing the political trajectories of four major parties in the last three decades. KW - manifesto KW - corpus KW - annotation ER - TY - CPAPER TI - Fearful Falcons and Angry Llamas : Emotion Category Annotations of Arguments by Humans and LLMs AU - Greschner, Lynn AU - Klinger, Roman PY - 2026 T2 - Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities SP - 628–646 PB - Otto-Friedrich-Universität CY - Bamberg SN - 979-8-89176-234-3 UR - https://fis.uni-bamberg.de/handle/uniba/115224 LA - eng AB - Arguments evoke emotions, influencing the effect of the argument itself. Not only the emotional intensity but also the category influences the argument’s effects, for instance, the willingness to adapt stances. While binary emotionality has been studied in argumentative texts, there is no work on discrete emotion categories (e.g., ‘anger’) in such data. To fill this gap, we crowdsource subjective annotations of emotion categories in a German argument corpus and evaluate automatic LLM-based labeling methods. Specifically, we compare three prompting strategies (zero-shot, one-shot, chain-of-thought) on three large instruction-tuned language models (Falcon-7b-instruct, Llama-3.1-8B-instruct, GPT-4o-mini). We further vary the definition of the output space to be binary (is there emotionality in the argument?), closed-domain (which emotion from a given label set is in the argument?), or open-domain (which emotion is in the argument?). We find that emotion categories enhance the prediction of emotionality in arguments, emphasizing the need for discrete emotion annotations in arguments. Across all prompt settings and models, automatic predictions show a high recall but low precision for predicting anger and fear, indicating a strong bias toward negative emotions. KW - argument KW - emotion KW - corpus KW - large language model ER - TY - JOUR TI - External Knowledge Integration in Large Language Models: A Survey on Methods, Challenges, and Future Directions AU - Yadav, Itisha AU - Sirko Schindler AU - Diana Peters AU - Klinger, Roman PY - 2026 JO - Semantic web : interoperability, usability, applicability VL - 17 IS - 4 SP - 1-27 PB - Sage CY - London SN - 1570-0844 DO - 10.1177/22104968261453132 UR - https://fis.uni-bamberg.de/handle/uniba/115017 LA - eng ER - TY - CPAPER TI - Trust Me, I Can Convince You: The Contextualized Argument Appraisal Framework and the ContArgA Corpus AU - Greschner, Lynn AU - Weber, Sabine AU - Klinger, Roman PY - 2026 T2 - Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026) SP - 8327–8346 PB - European Languages Resources Association (ELRA) CY - Paris SN - 978-2-493814-49-4 DO - 10.63317/484rpnvebop5 UR - https://fis.uni-bamberg.de/handle/uniba/115018 LA - eng ER - TY - CPAPER TI - PARL: Prompt-based Agents for Reinforcement Learning AU - Menchaca Resendiz, Yarik AU - Klinger, Roman PY - 2026 T2 - Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026) SP - 6166–6184 PB - European Languages Resources Association (ELRA) CY - Paris SN - 978-2-493814-49-4 DO - 10.63317/3z4zqifrngk9 UR - https://fis.uni-bamberg.de/handle/uniba/115019 LA - eng ER - TY - CPAPER TI - Categorical Emotions or Appraisals: Which Emotion Model Explains Argument Convincingness Better? AU - Greschner, Lynn AU - Bauer, Meike AU - Weber, Sabine AU - Klinger, Roman PY - 2026 T2 - Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026) SP - 8190–8203 PB - European Languages Resources Association (ELRA) CY - Paris SN - 978-2-493814-49-4 DO - 10.63317/3vrvrgvtnvhn UR - https://fis.uni-bamberg.de/handle/uniba/115020 LA - eng ER - TY - CPAPER TI - Entity-Level Sentiment Analysis with Sentence Relevance Detection AU - Rønningstad, Egil AU - Klinger, Roman AU - Øvrelid, Lilja AU - Velldal, Erik PY - 2026 T2 - Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026) SP - 8040–8055 PB - European Languages Resources Association (ELRA) CY - Paris SN - 978-2-493814-49-4 DO - 10.63317/35ideqx4jk89 UR - https://fis.uni-bamberg.de/handle/uniba/115021 LA - eng ER - TY - CPAPER TI - Disambiguation of Emotion Annotations by Contextualizing Events in Plausible Narratives AU - Schaefer, Johannes AU - Klinger, Roman PY - 2026 T2 - Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026) SP - 9635–9656 PB - European Languages Resources Association (ELRA) CY - Paris SN - 978-2-493814-49-4 DO - 10.63317/2agma6tpnh8h UR - https://fis.uni-bamberg.de/handle/uniba/115022 LA - eng ER - TY - CPAPER TI - Less Is More?: The Role of Demographic Author Information in Emotion Classification of Ambiguous Text AU - Weber, Sabine AU - Greschner, Lynn AU - Klinger, Roman PY - 2026 T2 - Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026) SP - 8147–8161 PB - European Languages Resources Association (ELRA) CY - Paris SN - 978-2-493814-49-4 DO - 10.63317/2cw8tpo82h55 UR - https://fis.uni-bamberg.de/handle/uniba/115023 LA - eng ER - TY - CPAPER TI - The Proceedings for the 15th Workshop on Computational Approaches to Subjectivity, Sentiment Social Media Analysis (WASSA 2026) ED - Barnes, Jeremy ED - Barriere, Valentin ED - Clercq, Orphee De ED - Klinger, Roman ED - Nouri, Célia ED - Nozza, Debora ED - Singh, Pranaydeep PY - 2026 PB - Association for Computational Linguistics SN - 979-8-89176-378-4 UR - https://fis.uni-bamberg.de/handle/uniba/114412 LA - eng ER - TY - CPAPER TI - Says Who? : Argument Convincingness and Reader Stance Are Correlated with Perceived Author Personality AU - Weber, Sabine AU - Greschner, Lynn AU - Klinger, Roman PY - 2026 T2 - The Proceedings for the 15th Workshop on Computational Approaches to Subjectivity, Sentiment Social Media Analysis (WASSA 2026) SP - 265–277 PB - Association for Computational Linguistics SN - 979-8-89176-378-4 UR - https://fis.uni-bamberg.de/handle/uniba/114414 LA - eng ER - TY - CPAPER TI - Emotionally Charged, Logically Blurred : AI-driven Emotional Framing Impairs Human Fallacy Detection AU - Chen, Yanran AU - Greschner, Lynn AU - Klinger, Roman AU - Klenk, Michael AU - Eger, Steffen PY - 2026 T2 - Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics SP - 6709–6732 PB - Association for Computational Linguistics SN - 979-8-89176-380-7 UR - https://fis.uni-bamberg.de/handle/uniba/114415 LA - eng ER - TY - CPAPER TI - Appraisal Trajectories in Narratives Reveal Distinct Patterns of Emotion Evocation AU - Schäfer, Johannes AU - Wagner, Janne AU - Klinger, Roman PY - 2026 T2 - The Proceedings for the 15th Workshop on Computational Approaches to Subjectivity, Sentiment Social Media Analysis (WASSA 2026) SP - 73–82 PB - Association for Computational Linguistics SN - 979-8-89176-378-4 UR - https://fis.uni-bamberg.de/handle/uniba/114416 LA - eng ER - TY - JOUR TI - What if Deception Cannot be Detected? : A Cross-Linguistic Study on the Limits of Deception Detection from Text AU - Velutharambath, Aswathy AU - Sassenberg, Kai AU - Klinger, Roman PY - 2026 JO - Computational Linguistics IS - Online First SP - 1-71 PB - MIT Press CY - Cambridge, MA SN - 1530-9312 DO - 10.1162/COLI.a.614 UR - https://fis.uni-bamberg.de/handle/uniba/114417 LA - eng ER - TY - CPAPER TI - Which Demographics do LLMs Default to During Annotation? AU - Schäfer, Johannes AU - Combs, Aidan AU - Bagdon, Christopher Doyle AU - Li, Jiahui AU - Probol, Nadine AU - Greschner, Lynn AU - Papay, Sean AU - Menchaca Resendiz, Yarik AU - Velutharambath, Aswathy AU - Wuehrl, Amelie AU - Weber, Sabine AU - Klinger, Roman PY - 2026 SP - 17331–17348 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/114413 LA - eng AB - Demographics and cultural background of annotators influence the labels they assign in text annotation – for instance, an elderly woman might find it offensive to read a message addressed to a “bro”, but a male teenager might find it appropriate. It is therefore important to acknowledge label variations to not underrepresent members of a society. Two research directions developed out of this observation in the context of using large language models (LLM) for data annotations, namely (1) studying biases and inherent knowledge of LLMs and (2) injecting diversity in the output by manipulating the prompt with demographic information. We combine these two strands of research and ask the question to which demographics an LLM resorts when no demographics is given. To answer this question, we evaluate which attributes of human annotators LLMs inherently mimic. Furthermore, we compare non-demographic conditioned prompts and placebo-conditioned prompts (e.g., “you are an annotator who lives in house number 5”) to demographics-conditioned prompts (“You are a 45 year old man and an expert on politeness annotation. How do you rate {instance}”). We study these questions for politeness and offensiveness annotations on the POPQUORN data set, a corpus created in a controlled manner to investigate human label variations based on demographics which has not been used for LLM-based analyses so far. We observe notable influences related to gender, race, and age in demographic prompting, which contrasts with previous studies that found no such effects KW - - ER - TY - CPAPER TI - Donate or Create? : Comparing Data Collection Strategies for Emotion-labeled Multimodal Social Media Posts AU - Bagdon, Christopher Doyle AU - Combs, Aidan AU - Silberer, Carina AU - Klinger, Roman PY - 2026 SP - 17307–17330 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/114410 LA - eng AB - Accurate modeling of subjective phenomena such as emotion expression requires data annotated with authors’ intentions. Commonly such data is collected by asking study participants to donate and label genuine content produced in the real world, or create content fitting particular labels during the study. Asking participants to create content is often simpler to implement and presents fewer risks to participant privacy than data donation. However, it is unclear if and how study-created content may differ from genuine content, and how differences may impact models. We collect study-created and genuine multimodal social media posts labeled for emotion and compare them on several dimensions, including model performance. We find that compared to genuine posts, study-created posts are longer, rely more on their text and less on their images for emotion expression, and focus more on emotion-prototypical events. The samples of participants willing to donate versus create posts are demographically different. Study-created data is valuable to train models that generalize well to genuine data, but realistic effectiveness estimates require genuine data. KW - - ER - TY - CPAPER TI - QoLAS : A Reddit Corpus of Health-Related Quality of Life Aspects of Mental Disorders AU - Greschner, Lynn AU - Wührl, Amelie AU - Klinger, Roman PY - 2026 SP - 201–216 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/114409 LA - eng AB - ‘Quality of Life’ (QoL) refers to a person’s subjective perception of various aspects of their life. For medical practitioners, it is one of the most important concepts for treatment decisions. Therefore, it is essential to understand in which aspects a medical condition affects a patient’s subjective perception of their life. With this paper, we focus on the under-resourced domain of mental health-related QoL, and contribute the first corpus to study and model this concept: We (1) annotate 240 Reddit posts with a set of 11 QoL aspects (such as ‘independence’, ‘mood’, or ‘relationships’) and their sentiment polarity. Based on this novel corpus, we (2) evaluate a pipeline to detect QoL mentions and classify them into aspects using open domain aspect-based sentiment analysis. We find that users frequently discuss healthrelated QoL in their posts, focusing primarily on the aspects ‘relationships’ and ‘selfimage’. Our method reliably predicts such mentions and their sentiment, however, detecting finegrained individual aspects remains challenging. An analysis of a large corpus of automatically labeled data reveals that social media content contains novel aspects pertinent to patients that are not covered by existing QoL taxonomies KW - - ER - TY - CPAPER TI - Regular-pattern-sensitive CRFs for Distant Label Interactions AU - Papay, Sean AU - Klinger, Roman AU - Padó, Sebastian PY - 2026 SP - 26–35 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/114408 LA - eng AB - While LLMs have grown popular in sequence labeling, linear-chain conditional random fields (CRFs) remain a popular alternative with the ability to directly model interactions between labels. However, the Markov assumption limits them to interactions between adjacent labels. Weighted finite state transducers (FSTs), in contrast, can model distant label–label interactions, but exact label inference is intractable in general. In this work, we present regular-pattern-sensitive CRFs (RPCRFs), a method of enriching standard linear-chain CRFs with the ability to learn long-distance label interactions through userspecified patterns. This approach allows users to write regular-expression label patterns concisely specifying which types of interactions the model should take into account, allowing the model to learn from data whether and in which contexts these patterns occur. The result can be interpreted alternatively as a CRF augmented with additional, non-local potentials, or as a finite-state transducer whose structure is defined by a set of easily-interpretable patterns. Critically, exact training and inference are tractable for many pattern sets. We detail how an RPCRF can be automatically constructed from a set of user-specified patterns, and demonstrate the model’s effectiveness on a sequence of three synthetic sequence modeling datasets. KW - - ER - TY - CPAPER TI - iPrOp : Interactive Prompt Optimization for Large Language Models with a Human in the Loop AU - Li, Jiahui AU - Klinger, Roman PY - 2026 SP - 276–285 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/114406 LA - eng AB - Prompt engineering has made significant contributions to the era of large language models, yet its effectiveness depends on the skills of a prompt author. This paper introduces iPrOp, a novel interactive prompt optimization approach, to bridge manual prompt engineering and automatic prompt optimization while offering users the flexibility to assess evolving prompts. We aim to provide users with task-specific guidance to enhance human engagement in the optimization process, which is structured through prompt variations, informative instances, predictions generated by large language models along with their corresponding explanations, and relevant performance metrics. This approach empowers users to choose and further refine the prompts based on their individual preferences and needs. It can not only assist non-technical domain experts in generating optimal prompts tailored to their specific tasks or domains, but also enable to study the intrinsic parameters that influence the performance of prompt optimization. The evaluation shows that our approach has the capability to generate improved prompts, leading to enhanced task performance. KW - - ER - TY - CPAPER TI - Supporting Plain Language Summarization of Psychological Meta-Analyses with Large Language Models AU - Menchaca Resendiz, Yarik AU - Kerwer, Martin AU - Chasiotis, Anita AU - Bodemer, Marlene AU - Sassenberg, Kai AU - Klinger, Roman PY - 2026 SP - 25–35 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/113163 LA - eng AB - Communicating complex scientific findings to non-experts remains a major challenge in fields like psychology, where research is often presented in highly technical language. One effective way to improve accessibility, for non-experts, is through plain language summaries, which summarize key insights into simple and understandable terms. However, the limited number of institutions that produce lay summaries typically relies on psychology experts to create them manually – an approach that ensures high quality but requires significant expertise, time, and effort. In this paper, we introduce the KLARpsy App, a system designed to support psychology experts in creating plain language summaries of psychological meta-analyses using Large Language Models (LLM). Our system generates initial draft summaries based on a 37-criterion guideline developed to ensure clarity for non-experts. All summaries produced through the system are manually validated and edited by KLARpsy authors to ensure factual correctness and readability. We demonstrate how the system integrates LLM-generated content into an expert-in-the-loop workflow. The automatic evaluation showed a mean semantic-similarity score of 0.73 against expert-written summaries, and human evaluation on a 5-point Likert scale averaged above 3 (higher is better), indicate that the generated drafts are of high quality. The application and code are open source. KW - Large Language Model ER - TY - CPAPER TI - Are Humans as Brittle as Large Language Models? AU - Li, Jiahui AU - Papay, Sean AU - Klinger, Roman PY - 2026 SP - 2130–2155 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/113162 LA - eng AB - The output of large language models (LLMs) is unstable, due both to non-determinism of the decoding process as well as to prompt brittleness. While the intrinsic non-determinism of LLM generation may mimic existing uncertainty in human annotations through distributional shifts in outputs, it is largely assumed, yet unexplored, that the prompt brittleness effect is unique to LLMs. This raises the question: do human annotators show similar sensitivity to prompt changes? If so, should prompt brittleness in LLMs be considered problematic? One may alternatively hypothesize that prompt brittleness correctly reflects human annotation variances. To fill this research gap, we systematically compare the effects of prompt modifications on LLMs and identical instruction modifications for human annotators, focusing on the question of whether humans are similarly sensitive to prompt perturbations. To study this, we prompt both humans and LLMs for a set of text classification tasks conditioned on prompt variations. Our findings indicate that both humans and LLMs exhibit increased brittleness in response to specific types of prompt modifications, particularly those involving the substitution of alternative label sets or label formats. However, the distribution of human judgments is less affected by typographical errors and reversed label order than that of LLMs. KW - Large Language Model ER - TY - JOUR TI - Politically knowledgeable ideologues believe they are less effective in political conversations AU - Combs, Aidan PY - 2026 JO - Humanities and Social Sciences Communications VL - 13 IS - 1, 170 SP - 1-7 PB - Springer Nature CY - London SN - 2662-9992 DO - 10.1057/s41599-025-06470-8 UR - https://fis.uni-bamberg.de/handle/uniba/113105 LA - eng AB - How accurately can people estimate how influential they are when they talk about politics? Using data from two field experiments in which partisans discussed political issues with anonymous partners via messaging apps, I find that peoples’ perceptions of their influence are not related to their actual influence. Instead, perceptions of influence are shaped by political knowledge and ideology. In both cross-party and all-Democrat conversations, people who know more about politics believe they are less influential. In cross-party but not all-Democrat conversations, those who are more ideologically extreme believe they are less influential. Notably, political knowledge has no effect on the amount of influence respondents actually have. These findings suggest a need to better understand potential differential effects of conversation-based depolarization interventions by level of participant political knowledge. KW - Politics and international relations KW - Sociology ER - TY - CPAPER TI - Prompt-based Personality Profiling : Reinforcement Learning for Relevance Filtering AU - Hofmann, Jan AU - Sindermann, Cornelia AU - Klinger, Roman PY - 2025 T2 - Proceedings of the 1st Workshop for Research on Agent Language Models (REALM 2025) SP - 1-16 CY - Kerrville, TX SN - 9798891762640 DO - 10.18653/v1/2025.realm-1.1 UR - https://fis.uni-bamberg.de/handle/uniba/117544 LA - eng ER - TY - JOUR TI - Can We Build AI That Does Not Harm Queer People? AU - Weber, Sabine PY - 2025 JO - Communications of the ACM VL - 68 IS - 5 SP - 21-23 PB - Association for Computing Machinery (ACM) CY - New York, NY SN - 1557-7317 DO - 10.1145/3720537 UR - https://fis.uni-bamberg.de/handle/uniba/116845 LA - eng ER - TY - CPAPER TI - Supporting Plain Language Summarization of Psychological Meta-Analyses with Large Language Models AU - Menchaca Resendiz, Yarik AU - Kerwer, Martin AU - Chasiotis, Anita AU - Bodemer, Marlene AU - Sassenberg, Kai AU - Klinger, Roman PY - 2025 T2 - Proceedings of The 14th International Joint Conference on Natural Language Processing and The 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics : System Demonstrations SP - 25–35 PB - Association for Computational Linguistics CY - Mumbai, India SN - 979-8-89176-301-2 UR - https://fis.uni-bamberg.de/handle/uniba/112677 LA - eng AB - Communicating complex scientific findings to non-experts remains a major challenge in fields like psychology, where research is often presented in highly technical language. One effective way to improve accessibility, for non-experts, is through plain language summaries, which summarize key insights into simple and understandable terms. However, the limited number of institutions that produce lay summaries typically relies on psychology experts to create them manually – an approach that ensures high quality but requires significant expertise, time, and effort. In this paper, we introduce the KLARpsy App, a system designed to support psychology experts in creating plain language summaries of psychological meta-analyses using Large Language Models (LLM). Our system generates initial draft summaries based on a 37-criterion guideline developed to ensure clarity for non-experts. All summaries produced through the system are manually validated and edited by KLARpsy authors to ensure factual correctness and readability. We demonstrate how the system integrates LLM-generated content into an expert-in-the-loop workflow. The automatic evaluation showed a mean semantic-similarity score of 0.73 against expert-written summaries, and human evaluation on a 5-point Likert scale averaged above 3 (higher is better), indicate that the generated drafts are of high quality. The application and code are open source. KW - Large Language Model ER - TY - CPAPER TI - Are Humans as Brittle as Large Language Models? AU - Li, Jiahui AU - Papay, Sean AU - Klinger, Roman PY - 2025 T2 - Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics SP - 2130–2155 PB - The Asian Federation of Natural Language Processing and The Association for Computational Linguistics CY - Mumbai, India SN - 979-8-89176-298-5 UR - https://fis.uni-bamberg.de/handle/uniba/112676 LA - eng AB - The output of large language models (LLMs) is unstable, due both to non-determinism of the decoding process as well as to prompt brittleness. While the intrinsic non-determinism of LLM generation may mimic existing uncertainty in human annotations through distributional shifts in outputs, it is largely assumed, yet unexplored, that the prompt brittleness effect is unique to LLMs. This raises the question: do human annotators show similar sensitivity to prompt changes? If so, should prompt brittleness in LLMs be considered problematic? One may alternatively hypothesize that prompt brittleness correctly reflects human annotation variances. To fill this research gap, we systematically compare the effects of prompt modifications on LLMs and identical instruction modifications for human annotators, focusing on the question of whether humans are similarly sensitive to prompt perturbations. To study this, we prompt both humans and LLMs for a set of text classification tasks conditioned on prompt variations. Our findings indicate that both humans and LLMs exhibit increased brittleness in response to specific types of prompt modifications, particularly those involving the substitution of alternative label sets or label formats. However, the distribution of human judgments is less affected by typographical errors and reversed label order than that of LLMs. KW - Large Language Model ER - TY - JOUR TI - Bias Mitigation for Capturing Potentially Illegal Hate Speech AU - Schäfer, Johannes PY - 2025 SP - 41-51 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/112210 LA - eng AB - Hate speech is a persistent issue in social media. Researchers have analyzed and developed detection methods for hate speech on the basis of example data, even though the phenomenon is only rather vaguely defined. This paper provides an approach to identify hate speech in terms of German laws, which are used as a basis for annotation guidelines applied to real world data. We annotate six labels in a corpus of 1,385 German short text messages: four subcategories of illegal hate speech, offensive language and a neutral class. We consider hate speech expressions as illegal if the linguistic content could be interpreted in a given context possibly violating a specific law. This interpretation and a check by lawyers would be the next step which is not yet included in our annotation. In this paper, I report on strategies to avoid certain biases in data for illegal hate speech. These strategies may serve as a model for building a larger dataset. In experiments, I investigate the capability of a Transformer-based neural network model to learn our classification. The results show that this multiclass classification is still difficult to learn, probably due to the small size of the dataset. I suggest that it is crucial to be aware of data biases and to apply bias mitigation techniques when training hate speech detection systems on such data. Data and scripts of the experiments are made publicly available. KW - Hate speech detection KW - Hate speech data KW - Illegal hate speech KW - Identity term bias ER - TY - CHAP TI - Hate Speech behandeln : Diagnosewerkzeuge aus der Computerlinguistik AU - Schäfer, Johannes PY - 2025 SP - 89-109 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/112140 LA - deu AB - Hate Speech stellt in sozialen Medien ein offenkundiges Problem dar, welches in der Forschung in den letzten Jahren zunehmend interdisziplinär untersucht wird. In diesem Artikel beleuchte ich das Phänomen aus computerlinguistischer Sicht und beschreibe näherungsweise erprobte Herangehensweisen für Lösungsansätze. Durch eine Diskussion von Anwendungsmöglichkeiten ordne ich das Problem in die computerlinguistische Forschung ein, wobei sich hier die automatische Erkennung von Hate Speech in Kurznachrichten vordergründig als Hauptaufgabe herausgestellt hat. Ich zeige jedoch, dass damit nur ein Teil des Problems untersucht wird. Beispielsweise der Kontext von Nachrichten bleibt in den meisten Forschungen zur Hate-Speech-Erkennung außen vor, was eine Diskrepanz zum realen Vorkommen des Phänomens darstellt. Auch zeige ich, dass es weitere, wenig untersuchte Forschungsrichtungen zum Phänomen Hate Speech gibt, wie zum Beispiel die Analyse von Teiläußerungen. Schließlich schlage ich eine Einteilung von Lösungsansätzen zur Erkennung von Hate Speech in drei Kategorien vor, zu denen ich jeweils einen einleitenden Überblick gebe: Lexikonbasierte Systeme, erklärbare maschinelle Lernsysteme und neuronale Netzwerke. KW - Hate Speech ER - TY - CPAPER TI - How Entangled is Factuality and Deception in German? AU - Velutharambath, Aswathy AU - Wuehrl, Amelie AU - Klinger, Roman PY - 2025 SP - 9538–9554 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/111944 LA - eng AB - The statement “The earth is flat” is factually inaccurate, but if someone truly believes and argues in its favor, it is not deceptive. Research on deception detection and fact checking often conflates factual accuracy with the truthfulness of statements. This assumption makes it difficult to (a) study subtle distinctions and interactions between the two and (b) gauge their effects on downstream tasks. The belief-based deception framework disentangles these properties by defining texts as deceptive when there is a mismatch between what people say and what they truly believe. In this study, we assess if presumed patterns of deception generalize to German language texts. We test the effectiveness of computational models in detecting deception using an established corpus of belief-based argumentation. Finally, we gauge the impact of deception on the downstream task of fact checking and explore if this property confounds verification models. Surprisingly, our analysis finds no correlation with established cues of deception. Previous work claimed that computational models can outperform humans in deception detection accuracy, however, our experiments show that both traditional and state-of-the-art models struggle with the task, performing no better than random guessing. For fact checking, we find that natural language inference-based verification performs worse on non-factual and deceptive content, while prompting large language models for the same task is less sensitive to these properties. KW - Factuality KW - Deception ER - TY - CPAPER TI - Understanding Fine-grained Distortions in Reports of Scientific Findings AU - Wührl, Amelie AU - Wright, Dustin AU - Klinger, Roman AU - Augenstein, Isabelle PY - 2025 SP - 6175–6191 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/111088 LA - eng AB - Distorted science communication harms individuals and society as it can lead to unhealthy behavior change and decrease trust in scientific institutions. Given the rapidly increasing volume of science communication in recent years, a fine-grained understanding of how findings from scientific publications are reported to the general public, and methods to detect distortions from the original work automatically, are crucial. Prior work focused on individual aspects of distortions or worked with unpaired data. In this work, we make three foundational contributions towards addressing this problem: (1) annotating 1,600 instances of scientific findings from academic papers paired with corresponding findings as reported in news articles and tweets wrt. four characteristics: causality, certainty, generality and sensationalism; (2) establishing baselines for automatically detecting these characteristics; and (3) analyzing the prevalence of changes in these characteristics in both human-annotated and large-scale unlabeled data. Our results show that scientific findings frequently undergo subtle distortions when reported. Tweets distort findings more often than science news reports. Detecting fine-grained distortions automatically poses a challenging task. In our experiments, fine-tuned task-specific models consistently outperform few-shot LLM prompting. KW - Fine-grained Distortions ER - TY - CPAPER TI - Entity-Level Sentiment : More than the Sum of Its Parts AU - Rønningstad, Egil AU - Klinger, Roman AU - Øvrelid, Lilja AU - Velldal, Erik PY - 2025 SP - 84–96 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/111087 LA - eng AB - In sentiment analysis of longer texts, there may be a variety of topics discussed, of entities mentioned, and of sentiments expressed regarding each entity. We find a lack of studies exploring how such texts express their sentiment towards each entity of interest, and how these sentiments can be modelled. In order to better understand how sentiment regarding persons and organizations (each entity in our scope) is expressed in longer texts, we have collected a dataset of expert annotations where the overall sentiment regarding each entity is identified, together with the sentence-level sentiment for these entities separately. We show that the reader’s perceived sentiment regarding an entity often differs from an arithmetic aggregation of sentiments at the sentence level. Only 70% of the positive and 55% of the negative entities receive a correct overall sentiment label when we aggregate the (human-annotated) sentiment labels for the sentences where the entity is mentioned. Our dataset reveals the complexity of entity-specific sentiment in longer texts, and allows for more precise modelling and evaluation of such sentiment expressions. KW - Entity-Level Sentiment ER - TY - CPAPER TI - Hierarchical Adversarial Correction to Mitigate Identity Term Bias in Toxicity Detection AU - Schäfer, Johannes AU - Heid, Ulrich AU - Klinger, Roman PY - 2025 SP - 35–51 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/111086 LA - eng AB - Corpora that are the fundament for toxicity detection contain such expressions typically directed against a target individual or group, e.g., people of a specific gender or ethnicity. Prior work has shown that the target identity mention can constitute a confounding variable. As an example, a model might learn that Christians are always mentioned in the context of hate speech. This misguided focus can lead to a limited generalization to newly emerging targets that are not found in the training data. In this paper, we hypothesize and subsequently show that this issue can be mitigated by considering targets on different levels of specificity. We distinguish levels of (1) the existence of a target, (2) a class (e.g., that the target is a religious group), or (3) a specific target group (e.g., Christians or Muslims). We define a target label hierarchy based on these three levels and then exploit this hierarchy in an adversarial correction for the lowest level (i.e. (3)) while maintaining some basic target features. This approach does not lower the toxicity detection performance but increases the generalization to targets not being available at training time. KW - Hierarchical Adversarial Correction ER - TY - CPAPER TI - IMS_medicALY at #SMM4H 2024 : Detecting Impacts of Outdoor Spaces on Social Anxiety with Data Augmented Ensembling AU - Wührl, Amelie AU - Greschner, Lynn AU - Menchaca Resendiz, Yarik AU - Klinger, Roman PY - 2025 SP - 83–87 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/111077 LA - eng AB - Many individuals affected by Social Anxiety Disorder turn to social media platforms to share their experiences and seek advice. This includes discussing the potential benefits of engaging with outdoor environments. As part of #SMM4H 2024, Shared Task 3 focuses on classifying the effects of outdoor spaces on social anxiety symptoms in Reddit posts. In our contribution to the task, we explore the effectiveness of domain-specific models (trained on social media data – SocBERT) against general domain models (trained on diverse datasets – BERT, RoBERTa, GPT-3.5) in predicting the sentiment related to outdoor spaces. Further, we assess the benefits of augmenting sparse human-labeled data with synthetic training instances and evaluate the complementary strengths of domain-specific and general classifiers using an ensemble model. Our results show that (1) fine-tuning small, domain-specific models generally outperforms large general language models in most cases. Only one large language model (GPT-4) exhibits performance comparable to the fine-tuned models (52% F1). Further, we find that (2) synthetic data does improve the performance of fine-tuned models in some cases, and (3) models do not appear to complement each other in our ensemble setup. KW - Data Augmented Ensembling ER - TY - JOUR TI - The effect of occupational status on health : Putting the social in socioeconomic status AU - Combs, Aidan AU - Freeland, Robert E. AU - Hudak, Katelin M. Alfaro AU - Mumford, Elizabeth A. PY - 2025 SP - 1-10 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/110867 LA - eng AB - High status occupations support positive health outcomes through providing access to both material and psychosocial resources. However, common measures of occupational status such as occupational prestige scores fail to capture cultural esteem that certain occupations can provide because they are primarily associated with the material dimensions of status, like income. Drawing on Weberian conceptions of status and a body of social psychological research on the measurement of cultural meaning, we argue that measuring people's ratings of their occupations on three dimensions—evaluation (good/bad), potency (powerful/weak), and activity (active/inactive)—provides an occupational status indicator that more fully captures psychosocial resources like esteem that are associated with health than more commonly used occupational prestige scores. Using a nationally representative longitudinal health and wellbeing survey of 940 American law enforcement officers collected between 2020 and 2022, we evaluate the predictive ability of evaluation, potency, and activity (EPA) ratings across thirteen measures of health and wellbeing. We find that EPA ratings were significant and positive predictors of eleven of thirteen outcomes with stronger effects for mental health outcomes compared to physical health outcomes. EPA ratings were more predictive than more commonly used occupational prestige scores. We conclude that EPA ratings are better predictors of health outcomes than occupational prestige scores and so may allow health researchers to better understand the relationship between occupational status and health. KW - Socioeconomic status KW - Social psychology KW - Occupations KW - Affect control theory KW - United States KW - Law enforcement ER - TY - CPAPER TI - Language Models Are Poor Learners of Directional Inference AU - Li, Tianyi AU - Hosseini, Mohammad Javad AU - Weber, Sabine AU - Steedman, Mark PY - 2025 SP - 903–921 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/110866 LA - eng AB - We examine LMs’ competence of directional predicate entailments by supervised fine-tuning with prompts. Our analysis shows that contrary to their apparent success on standard NLI, LMs show limited ability to learn such directional inference; moreover, existing datasets fail to test directionality, and/or are infested by artefacts that can be learnt as proxy for entailments, yielding over-optimistic results. In response, we present BoOQA (Boolean Open QA), a robust multi-lingual evaluation benchmark for directional predicate entailments, extrinsic to existing training sets. On BoOQA, we establish baselines and show evidence of existing LM-prompting models being incompetent directional entailment learners, in contrast to entailment graphs, however limited by sparsity. KW - Language Models ER - TY - CPAPER TI - Cross-lingual Inference with A Chinese Entailment Graph AU - Li, Tianyi AU - Weber, Sabine AU - Hosseini, Mohammad Javad AU - Guillou, Liane AU - Steedman, Mark PY - 2025 SP - 1214–1233 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/110858 LA - eng AB - Predicate entailment detection is a crucial task for question-answering from text, where previous work has explored unsupervised learning of entailment graphs from typed open relation triples. In this paper, we present the first pipeline for building Chinese entailment graphs, which involves a novel high-recall open relation extraction (ORE) method and the first Chinese fine-grained entity typing dataset under the FIGER type ontology. Through experiments on the Levy-Holt dataset, we verify the strength of our Chinese entailment graph, and reveal the cross-lingual complementarity: on the parallel Levy-Holt dataset, an ensemble of Chinese and English entailment graphs outperforms both monolingual graphs, and raises unsupervised SOTA by 4.7 AUC points. KW - Chinese Entailment Graph ER - TY - CPAPER TI - Fine-grained General Entity Typing in German using GermaNet AU - Weber, Sabine AU - Steedman, Mark PY - 2025 SP - 138–143 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/110857 LA - eng AB - Fine-grained entity typing is important to tasks like relation extraction and knowledge base construction. We find however, that fine-grained entity typing systems perform poorly on general entities (e.g. “ex-president”) as compared to named entities (e.g. “Barack Obama”). This is due to a lack of general entities in existing training data sets. We show that this problem can be mitigated by automatically generating training data from WordNets. We use a German WordNet equivalent, GermaNet, to automatically generate training data for German general entity typing. We use this data to supplement named entity data to train a neural fine-grained entity typing system. This leads to a 10% improvement in accuracy of the prediction of level 1 FIGER types for German general entities, while decreasing named entity type prediction accuracy by only 1%. KW - GermaNet ER - TY - CPAPER TI - Zero-Shot Cross-Lingual Transfer is a Hard Baseline to Beat in German Fine-Grained Entity Typing AU - Weber, Sabine AU - Steedman, Mark PY - 2025 SP - 42-48 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/110856 LA - eng AB - The training of NLP models often requires large amounts of labelled training data, which makes it difficult to expand existing models to new languages. While zero-shot cross-lingual transfer relies on multilingual word embeddings to apply a model trained on one language to another, Yarowski and Ngai (2001) propose the method of annotation projection to generate training data without manual annotation. This method was successfully used for the tasks of named entity recognition and coarse-grained entity typing, but we show that it is outperformed by zero-shot cross-lingual transfer when applied to the similar task of fine-grained entity typing. In our study of fine-grained entity typing with the FIGER type ontology for German, we show that annotation projection amplifies the English model’s tendency to underpredict level 2 labels and is beaten by zero-shot cross-lingual transfer on three novel test sets. KW - Zero-Shot Cross-Lingual Transfer ER - TY - CPAPER TI - Different German and English Coreference Resolution Models for Multi-domain Content Curation Scenarios AU - Srivastava, Ankit AU - Weber, Sabine AU - Bourgonje, Peter AU - Rehm, Georg PY - 2025 SP - 48-61 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/110847 LA - eng AB - Coreference Resolution is the process of identifying all words and phrases in a text that refer to the same entity. It has proven to be a useful intermediary step for a number of natural language processing applications. In this paper, we describe three implementations for performing coreference resolution: rule-based, statistical, and projection-based (from English to German). After a comparative evaluation on benchmark datasets, we conclude with an application of these systems on German and English texts from different scenarios in digital curation such as an archive of personal letters, excerpts from a museum exhibition, and regional news articles. KW - Multi-domain Content Curation Scenarios ER - TY - CPAPER TI - EmoProgress : Cumulated Emotion Progression Analysis in Dreams and Customer Service Dialogues AU - Wemmer, Eileen AU - Labat, Sofie AU - Klinger, Roman PY - 2025 SP - 5660–5677 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/110605 LA - eng AB - Emotion analysis often involves the categorization of isolated textual units, but these are parts of longer discourses, like dialogues or stories. This leads to two different established emotion classification setups: (1) Classification of a longer text into one or multiple emotion categories. (2) Classification of the parts of a longer text (sentences or utterances), either (2a) with or (2b) without consideration of the context. None of these settings, does, however, enable to answer the question which emotion is presumably experienced at a specific moment in time. For instance, a customer’s request of “My computer broke.” would be annotated with anger. This emotion persists in a potential follow-up reply “It is out of warranty.” which would also correspond to the global emotion label. An alternative reply “We will send you a new one.” might, in contrast, lead to relief. Modeling these label relations requires classification of textual parts under consideration of the past, but without access to the future. Consequently, we propose a novel annotation setup for emotion categorization corpora, in which the annotations reflect the emotion up to the annotated sentence. We ensure this by uncovering the textual parts step-by-step to the annotator, asking for a label in each step. This perspective is important to understand the final, global emotion, while having access to the individual sentence’s emotion contributions to this final emotion. In modeling experiments, we use these data to check if the context is indeed required to automatically predict such cumulative emotion progressions. KW - Emotion ER - TY - CPAPER TI - Can Factual Statements Be Deceptive? : The DeFaBel Corpus of Belief-based Deception AU - Velutharambath, Aswathy AU - Wührl, Amelie AU - Klinger, Roman PY - 2025 SP - 2708-2723 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/110604 LA - eng AB - If a person firmly believes in a non-factual statement, such as “The Earth is flat”, and argues in its favor, there is no inherent intention to deceive. As the argumentation stems from genuine belief, it may be unlikely to exhibit the linguistic properties associated with deception or lying. This interplay of factuality, personal belief, and intent to deceive remains an understudied area. Disentangling the influence of these variables in argumentation is crucial to gain a better understanding of the linguistic properties attributed to each of them. To study the relation between deception and factuality, based on belief, we present the DeFaBel corpus, a crowd-sourced resource of belief-based deception. To create this corpus, we devise a study in which participants are instructed to write arguments supporting statements like “eating watermelon seeds can cause indigestion”, regardless of its factual accuracy or their personal beliefs about the statement. In addition to the generation task, we ask them to disclose their belief about the statement. The collected instances are labelled as deceptive if the arguments are in contradiction to the participants’ personal beliefs. Each instance in the corpus is thus annotated (or implicitly labelled) with personal beliefs of the author, factuality of the statement, and the intended deceptiveness. The DeFaBel corpus contains 1031 texts in German, out of which 643 are deceptive and 388 are non-deceptive. It is the first publicly available corpus for studying deception in German. In our analysis, we find that people are more confident in the persuasiveness of their arguments when the statement is aligned with their belief, but surprisingly less confident when they are generating arguments in favor of facts. The DeFaBel corpus can be obtained from https://www.ims.uni-stuttgart.de/data/defabel . KW - Factual Statements ER - TY - CPAPER TI - Dissecting Span Identification Tasks with Performance Prediction AU - Papay, Sean AU - Klinger, Roman AU - Padó, Sebastian PY - 2025 SP - 4881-4895 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108319 LA - eng AB - Span identification (in short, span ID) tasks such as chunking, NER, or code-switching detection, ask models to identify and classify relevant spans in a text. Despite being a staple of NLP, and sharing a common structure, there is little insight on how these tasks’ properties influence their difficulty, and thus little guidance on what model families work well on span ID tasks, and why. We analyze span ID tasks via performance prediction, estimating how well neural architectures do on different tasks. Our contributions are: (a) we identify key properties of span ID tasks that can inform performance prediction; (b) we carry out a large-scale experiment on English data, building a model to predict performance for unseen span ID tasks that can support architecture choices; (c), we investigate the parameters of the meta model, yielding new insights on how model and task properties interact to affect span ID performance. We find, e.g., that span frequency is especially important for LSTMs, and that CRFs help when spans are infrequent and boundaries non-distinctive. KW - Span Identification Tasks ER - TY - JOUR TI - Fast approximate STEM image simulations from a machine learning model AU - Combs, Aidan H. AU - Maldonis, Jason J. AU - Feng, Jie AU - Xu, Zhongnan AU - Voyles, Paul M. AU - Morgan, Dane PY - 2025 SP - 1-10 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/110868 LA - eng AB - Accurate quantum mechanical scanning transmission electron microscopy image simulation methods such as the multislice method require computation times that are too large to use in applications in high-resolution materials imaging that require very large numbers of simulated images. However, higher-speed simulation methods based on linear imaging models, such as the convolution method, are often not accurate enough for use in these applications. We present a method that generates an image from the convolution of an object function and the probe intensity, and then uses a multivariate polynomial fit to a dataset of corresponding multislice and convolution images to correct it. We develop and validate this method using simulated images of Pt and Pt–Mo nanoparticles and find that for these systems, once the polynomial is fit, the method runs about six orders of magnitude faster than parallelized CPU implementations of the multislice method while achieving a 1 − R2 error of 0.010–0.015 and root-mean-square error/standard deviation of dataset being predicted of about 0.1 when compared to full multislice simulations. KW - Scanning transmission electron microscopy KW - Convolution KW - Frozen phonon multislice simulation KW - High-throughput KW - Linear regression ER - TY - JOUR TI - Assessing the Russian Internet Research Agency’s impact on the political attitudes and behaviors of American Twitter users in late 2017 AU - Bail, Christopher A. AU - Guay, Brian AU - Maloney, Emily AU - Combs, Aidan AU - Hillygus, D. Sunshine AU - Merhout, Friedolin AU - Freelon, Deen AU - Volfovsky, Alexander PY - 2025 SP - 243–250 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/110870 LA - eng AB - There is widespread concern that Russia and other countries have launched social-media campaigns designed to increase political divisions in the United States. Though a growing number of studies analyze the strategy of such campaigns, it is not yet known how these efforts shaped the political attitudes and behaviors of Americans. We study this question using longitudinal data that describe the attitudes and online behaviors of 1,239 Republican and Democratic Twitter users from late 2017 merged with nonpublic data about the Russian Internet Research Agency (IRA) from Twitter. Using Bayesian regression tree models, we find no evidence that interaction with IRA accounts substantially impacted 6 distinctive measures of political attitudes and behaviors over a 1-mo period. We also find that interaction with IRA accounts were most common among respondents with strong ideological homophily within their Twitter network, high interest in politics, and high frequency of Twitter usage. Together, these findings suggest that Russian trolls might have failed to sow discord because they mostly interacted with those who were already highly polarized. We conclude by discussing several important limitations of our study—especially our inability to determine whether IRA accounts influenced the 2016 presidential election—as well as its implications for future research on social media influence campaigns, political polarization, and computational social science. KW - misinformation KW - social media KW - political polarization KW - computational social science ER - TY - JOUR TI - Perceived gender and political persuasion : a social media field experiment during the 2020 US Democratic presidential primary election AU - Combs, Aidan AU - Tierney, Graham AU - Alqabandi, Fatima AU - Cornell, Devin AU - Varela, Gabriel AU - Castro Araújo, Andrés AU - Argyle, Lisa P. AU - Bail, Christopher A. AU - Volfovsky, Alexander PY - 2025 SP - 1-14 UR - https://fis.uni-bamberg.de/handle/uniba/110871 LA - eng AB - Women have less influence than men in a variety of settings. Does this result from stereotypes that depict women as less capable, or biased interpretations of gender differences in behavior? We present a field experiment that—unbeknownst to the participants—randomized the gender of avatars assigned to Democrats using a social media platform we created to facilitate discussion about the 2020 Primary Election. We find that misrepresenting a man as a woman undermines his influence, but misrepresenting a woman as a man does not increase hers. We demonstrate that men’s higher resistance to being influenced—and gendered word use patterns—both contribute to this outcome. These findings challenge prevailing wisdom that women simply need to behave more like men to overcome gender discrimination and suggest that narrowing the gap will require simultaneous attention to the behavior of people who identify as women and as men. KW - political persuasion ER - TY - JOUR TI - Affective connotations according to LLMs : implications for meaning measurement and cultural bias AU - Combs, Aidan AU - Dametto, Diego AU - Blaison, Christophe AU - Leung, Renee AU - Malhotra, Aarti AU - Schröder, Tobias AU - Hoey, Jesse AU - Smith-Lovin, Lynn PY - 2025 JO - Cognition and Emotion VL - 0 SP - 1–17 PB - Routledge, Taylor & Francis Group CY - Abingdon SN - 1464-0600 DO - 10.1080/02699931.2025.2568551 UR - https://fis.uni-bamberg.de/handle/uniba/110743 LA - eng ER - TY - CPAPER TI - Which Demographics do LLMs Default to During Annotation? AU - Schäfer, Johannes AU - Combs, Aidan AU - Bagdon, Christopher Doyle AU - Li, Jiahui AU - Probol, Nadine AU - Greschner, Lynn AU - Papay, Sean AU - Menchaca Resendiz, Yarik AU - Velutharambath, Aswathy AU - Wuehrl, Amelie AU - Weber, Sabine AU - Klinger, Roman PY - 2025 T2 - Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics SP - 17331–17348 PB - Association for Computational Linguistics SN - 979-8-89176-251-0 DO - 10.18653/v1/2025.acl-long.848 UR - https://fis.uni-bamberg.de/handle/uniba/109541 LA - eng AB - Demographics and cultural background of annotators influence the labels they assign in text annotation – for instance, an elderly woman might find it offensive to read a message addressed to a “bro”, but a male teenager might find it appropriate. It is therefore important to acknowledge label variations to not underrepresent members of a society. Two research directions developed out of this observation in the context of using large language models (LLM) for data annotations, namely (1) studying biases and inherent knowledge of LLMs and (2) injecting diversity in the output by manipulating the prompt with demographic information. We combine these two strands of research and ask the question to which demographics an LLM resorts when no demographics is given. To answer this question, we evaluate which attributes of human annotators LLMs inherently mimic. Furthermore, we compare non-demographic conditioned prompts and placebo-conditioned prompts (e.g., “you are an annotator who lives in house number 5”) to demographics-conditioned prompts (“You are a 45 year old man and an expert on politeness annotation. How do you rate {instance}”). We study these questions for politeness and offensiveness annotations on the POPQUORN data set, a corpus created in a controlled manner to investigate human label variations based on demographics which has not been used for LLM-based analyses so far. We observe notable influences related to gender, race, and age in demographic prompting, which contrasts with previous studies that found no such effects KW - - ER - TY - CPAPER TI - Donate or Create? : Comparing Data Collection Strategies for Emotion-labeled Multimodal Social Media Posts AU - Bagdon, Christopher Doyle AU - Combs, Aidan AU - Silberer, Carina AU - Klinger, Roman PY - 2025 T2 - Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics SP - 17307–17330 PB - Association for Computational Linguistics SN - 979-8-89176-251-0 DO - 10.18653/v1/2025.acl-long.847 UR - https://fis.uni-bamberg.de/handle/uniba/109544 LA - eng AB - Accurate modeling of subjective phenomena such as emotion expression requires data annotated with authors’ intentions. Commonly such data is collected by asking study participants to donate and label genuine content produced in the real world, or create content fitting particular labels during the study. Asking participants to create content is often simpler to implement and presents fewer risks to participant privacy than data donation. However, it is unclear if and how study-created content may differ from genuine content, and how differences may impact models. We collect study-created and genuine multimodal social media posts labeled for emotion and compare them on several dimensions, including model performance. We find that compared to genuine posts, study-created posts are longer, rely more on their text and less on their images for emotion expression, and focus more on emotion-prototypical events. The samples of participants willing to donate versus create posts are demographically different. Study-created data is valuable to train models that generalize well to genuine data, but realistic effectiveness estimates require genuine data. KW - - ER - TY - CPAPER TI - QoLAS : A Reddit Corpus of Health-Related Quality of Life Aspects of Mental Disorders AU - Greschner, Lynn AU - Wührl, Amelie AU - Klinger, Roman PY - 2025 T2 - Proceedings of the 24th Workshop on Biomedical Language Processing SP - 201–216 PB - Association for Computational Linguistics SN - 979-8-89176-275-6 DO - 10.18653/v1/2025.bionlp-1.18 UR - https://fis.uni-bamberg.de/handle/uniba/109545 LA - eng AB - ‘Quality of Life’ (QoL) refers to a person’s subjective perception of various aspects of their life. For medical practitioners, it is one of the most important concepts for treatment decisions. Therefore, it is essential to understand in which aspects a medical condition affects a patient’s subjective perception of their life. With this paper, we focus on the under-resourced domain of mental health-related QoL, and contribute the first corpus to study and model this concept: We (1) annotate 240 Reddit posts with a set of 11 QoL aspects (such as ‘independence’, ‘mood’, or ‘relationships’) and their sentiment polarity. Based on this novel corpus, we (2) evaluate a pipeline to detect QoL mentions and classify them into aspects using open domain aspect-based sentiment analysis. We find that users frequently discuss healthrelated QoL in their posts, focusing primarily on the aspects ‘relationships’ and ‘selfimage’. Our method reliably predicts such mentions and their sentiment, however, detecting finegrained individual aspects remains challenging. An analysis of a large corpus of automatically labeled data reveals that social media content contains novel aspects pertinent to patients that are not covered by existing QoL taxonomies KW - - ER - TY - CPAPER TI - Regular-pattern-sensitive CRFs for Distant Label Interactions AU - Papay, Sean AU - Klinger, Roman AU - Padó, Sebastian PY - 2025 T2 - Proceedings of the 1st Joint Workshop on Large Language Models and Structure Modeling (XLLM 2025) SP - 26–35 PB - Association for Computational Linguistics SN - 979-8-89176-286-2 DO - 10.18653/v1/2025.xllm-1.4 UR - https://fis.uni-bamberg.de/handle/uniba/109547 LA - eng AB - While LLMs have grown popular in sequence labeling, linear-chain conditional random fields (CRFs) remain a popular alternative with the ability to directly model interactions between labels. However, the Markov assumption limits them to interactions between adjacent labels. Weighted finite state transducers (FSTs), in contrast, can model distant label–label interactions, but exact label inference is intractable in general. In this work, we present regular-pattern-sensitive CRFs (RPCRFs), a method of enriching standard linear-chain CRFs with the ability to learn long-distance label interactions through userspecified patterns. This approach allows users to write regular-expression label patterns concisely specifying which types of interactions the model should take into account, allowing the model to learn from data whether and in which contexts these patterns occur. The result can be interpreted alternatively as a CRF augmented with additional, non-local potentials, or as a finite-state transducer whose structure is defined by a set of easily-interpretable patterns. Critically, exact training and inference are tractable for many pattern sets. We detail how an RPCRF can be automatically constructed from a set of user-specified patterns, and demonstrate the model’s effectiveness on a sequence of three synthetic sequence modeling datasets. KW - - ER - TY - CPAPER TI - iPrOp : Interactive Prompt Optimization for Large Language Models with a Human in the Loop AU - Li, Jiahui AU - Klinger, Roman PY - 2025 T2 - Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics SP - 276–285 PB - Association for Computational Linguistics SN - 979-8-89176-254-1 DO - 10.18653/v1/2025.acl-srw.18 UR - https://fis.uni-bamberg.de/handle/uniba/109548 LA - eng AB - Prompt engineering has made significant contributions to the era of large language models, yet its effectiveness depends on the skills of a prompt author. This paper introduces iPrOp, a novel interactive prompt optimization approach, to bridge manual prompt engineering and automatic prompt optimization while offering users the flexibility to assess evolving prompts. We aim to provide users with task-specific guidance to enhance human engagement in the optimization process, which is structured through prompt variations, informative instances, predictions generated by large language models along with their corresponding explanations, and relevant performance metrics. This approach empowers users to choose and further refine the prompts based on their individual preferences and needs. It can not only assist non-technical domain experts in generating optimal prompts tailored to their specific tasks or domains, but also enable to study the intrinsic parameters that influence the performance of prompt optimization. The evaluation shows that our approach has the capability to generate improved prompts, leading to enhanced task performance. KW - - ER - TY - CPAPER TI - An Analysis of Emotion Communication Channels in Fan-Fiction : Towards Emotional Storytelling AU - Kim, Evgeny AU - Klinger, Roman PY - 2025 SP - 56-64 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108307 LA - eng AB - Centrality of emotion for the stories told by humans is underpinned by numerous studies in literature and psychology. The research in automatic storytelling has recently turned towards emotional storytelling, in which characters’ emotions play an important role in the plot development (Theune et al., 2004; y Perez, 2007; Mendez et al., 2016). However, these studies mainly use emotion to generate propositional statements in the form “A feels affection towards B” or “A confronts B”. At the same time, emotional behavior does not boil down to such propositional descriptions, as humans display complex and highly variable patterns in communicating their emotions, both verbally and non-verbally. In this paper, we analyze how emotions are expressed non-verbally in a corpus of fan fiction short stories. Our analysis shows that stories written by humans convey character emotions along various non-verbal channels. We find that some non-verbal channels, such as facial expressions and voice characteristics of the characters, are more strongly associated with joy, while gestures and body postures are more likely to occur with trust. Based on our analysis, we argue that automatic storytelling systems should take variability of emotion into account when generating descriptions of characters’ emotions. KW - Emotion Communication Channels ER - TY - CPAPER TI - Automatic Section Recognition in Obituaries AU - Sabbatino, Valentino AU - Bostan, Laura Ana Maria AU - Klinger, Roman PY - 2025 SP - 817-825 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108308 LA - eng AB - Obituaries contain information about people’s values across times and cultures, which makes them a useful resource for exploring cultural history. They are typically structured similarly, with sections corresponding to Personal Information, Biographical Sketch, Characteristics, Family, Gratitude, Tribute, Funeral Information and Other aspects of the person. To make this information available for further studies, we propose a statistical model which recognizes these sections. To achieve that, we collect a corpus of 20058 English obituaries from TheDaily Item, Remembering.CA and The London Free Press. The evaluation of our annotation guidelines with three annotators on 1008 obituaries shows a substantial agreement of Fleiss κ = 0.87. Formulated as an automatic segmentation task, a convolutional neural network outperforms bag-of-words and embedding-based BiLSTMs and BiLSTM-CRFs with a micro F1 = 0.81. KW - Obituaries ER - TY - CPAPER TI - GoodNewsEveryone : A Corpus of News Headlines Annotated with Emotions, Semantic Roles, and Reader Perception AU - Bostan, Laura Ana Maria AU - Kim, Evgeny AU - Klinger, Roman PY - 2025 SP - 1554-1566 PB - Otto-Friedrich-Universität CY - Bamberg SN - 979-10-95546-34-4 UR - https://fis.uni-bamberg.de/handle/uniba/108309 LA - eng AB - Most research on emotion analysis from text focuses on the task of emotion classification or emotion intensity regression. Fewer works address emotions as a phenomenon to be tackled with structured learning, which can be explained by the lack of relevant datasets. We fill this gap by releasing a dataset of 5000 English news headlines annotated via crowdsourcing with their associated emotions, the corresponding emotion experiencers and textual cues, related emotion causes and targets, as well as the reader’s perception of the emotion of the headline. This annotation task is comparably challenging, given the large number of classes and roles to be identified. We therefore propose a multiphase annotation procedure in which we first find relevant instances with emotional content and then annotate the more fine-grained aspects. Finally, we develop a baseline for the task of automatic prediction of semantic role structures and discuss the results. The corpus we release enables further research on emotion classification, emotion intensity prediction, emotion cause detection, and supports further qualitative studies. KW - GoodNewsEveryone ER - TY - CPAPER TI - PO-EMO : Conceptualization, Annotation, and Modeling of Aesthetic Emotions in German and English Poetry AU - Haider, Thomas AU - Eger, Steffen AU - Kim, Evgeny AU - Klinger, Roman AU - Menninghaus, Winfried PY - 2025 SP - 1652-1663 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108310 LA - eng AB - Most approaches to emotion analysis of social media, literature, news, and other domains focus exclusively on basic emotion categories as defined by Ekman or Plutchik. However, art (such as literature) enables engagement in a broader range of more complex and subtle emotions. These have been shown to also include mixed emotional responses. We consider emotions in poetry as they are elicited in the reader, rather than what is expressed in the text or intended by the author. Thus, we conceptualize a set of aesthetic emotions that are predictive of aesthetic appreciation in the reader, and allow the annotation of multiple labels per line to capture mixed emotions within their context. We evaluate this novel setting in an annotation experiment both with carefully trained experts and via crowdsourcing. Our annotation with experts leads to an acceptable agreement of k = .70, resulting in a consistent dataset for future large scale analysis. Finally, we conduct first emotion classification experiments based on BERT, showing that identifying aesthetic emotions is challenging in our data, with up to .52 F1-micro on the German subset. Data and resources are available at https://github.com/tnhaider/poetry-emotion. KW - PO-EMO ER - TY - CPAPER TI - Challenges in Emotion Style Transfer : An Exploration with a Lexical Substitution Pipeline AU - Helbig, David AU - Troiano, Enrica AU - Klinger, Roman PY - 2025 SP - 41-50 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108318 LA - eng AB - We propose the task of emotion style transfer, which is particularly challenging, as emotions (here: anger, disgust, fear, joy, sadness, surprise) are on the fence between content and style. To understand the particular difficulties of this task, we design a transparent emotion style transfer pipeline based on three steps: (1) select the words that are promising to be substituted to change the emotion (with a brute-force approach and selection based on the attention mechanism of an emotion classifier), (2) find sets of words as candidates for substituting the words (based on lexical and distributional semantics), and (3) select the most promising combination of substitutions with an objective function which consists of components for content (based on BERT sentence embeddings), emotion (based on an emotion classifier), and fluency (based on a neural language model). This comparably straight-forward setup enables us to explore the task and understand in what cases lexical substitution can vary the emotional load of texts, how changes in content and style interact and if they are at odds. We further evaluate our pipeline quantitatively in an automated and an annotation study based on Tweets and find, indeed, that simultaneous adjustments of content and emotion are conflicting objectives: as we show in a qualitative analysis motivated by Scherer’s emotion component model, this is particularly the case for implicit emotion expressions based on cognitive appraisal or descriptions of bodily reactions. KW - Emotion Style Transfer ER - TY - CHAP TI - Emotion Analysis for Literary Studies : Corpus Creation and Computational Modelling AU - Klinger, Roman AU - Kim, Evgeny AU - Padó, Sebastian PY - 2025 SP - 237-268 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108317 LA - eng AB - Most approaches to emotion analysis in fictional texts focus on detecting the emotion class expressed over the course of a text, either with machine learning-based classification or with dictionaries. These approaches do not consider who experiences the emotion and what triggers it and therefore, as a necessary simplicifaction, aggregate across different characters and events. This constitutes a research gap, as emotions play a crucial role in the interaction between characters and the events they are involved in. We fill this gap with the development of two corpora and associated computational models which represent individual events together with their experiencers and stimuli. The first resource, REMAN (Relational EMotion ANnotation), aims at a fine-grained annotation of all these aspects on the text level. The second corpus, FANFIC, contains complete stories, annotated on the experiencer-stimulus level, i. e., focuses on emotional relations among characters. FANFIC is therefore a character relation corpus while REMAN considers event descriptions in addition. Our experiments show that computational stimuli detection is particularly challenging. Furthermore, predicting roles in joint models has the potential to perform better than separate predictions. These resources provide a starting point for future research on the recognition of emotions and associated entities in text. They support qualitative literary studies and digital humanities research. The corpora are freely available at http://www.ims.uni-stuttgart.de/data/emotion. KW - Emotion Analysis ER - TY - CPAPER TI - Lost in Back-Translation : Emotion Preservation in Neural Machine Translation AU - Troiano, Enrica AU - Klinger, Roman AU - Padó, Sebastian PY - 2025 SP - 4340-4354 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108320 LA - eng AB - Machine translation provides powerful methods to convert text between languages, and is therefore a technology enabling a multilingual world. An important part of communication, however, takes place at the non-propositional level (e.g., politeness, formality, emotions), and it is far from clear whether current MT methods properly translate this information. This paper investigates the specific hypothesis that the non-propositional level of emotions is at least partially lost in MT. We carry out a number of experiments in a back-translation setup and establish that (1) emotions are indeed partially lost during translation; (2) this tendency can be reversed almost completely with a simple re-ranking approach informed by an emotion classifier, taking advantage of diversity in the n-best list; (3) the re-ranking approach can also be applied to change emotions, obtaining a model for emotion style transfer. An in-depth qualitative analysis reveals that there are recurring linguistic changes through which emotions are toned down or amplified, such as change of modality. KW - Emotion Preservation ER - TY - CPAPER TI - Appraisal Theories for Emotion Classification in Text AU - Hofmann, Jan AU - Troiano, Enrica AU - Sassenberg, Kai AU - Klinger, Roman PY - 2025 SP - 125-138 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108321 LA - eng AB - Automatic emotion categorization has been predominantly formulated as text classification in which textual units are assigned to an emotion from a predefined inventory, for instance following the fundamental emotion classes proposed by Paul Ekman (fear, joy, anger, disgust, sadness, surprise) or Robert Plutchik (adding trust, anticipation). This approach ignores existing psychological theories to some degree, which provide explanations regarding the perception of events. For instance, the description that somebody discovers a snake is associated with fear, based on the appraisal as being an unpleasant and non-controllable situation. This emotion reconstruction is even possible without having access to explicit reports of a subjective feeling (for instance expressing this with the words “I am afraid.”). Automatic classification approaches therefore need to learn properties of events as latent variables (for instance that the uncertainty and the mental or physical effort associated with the encounter of a snake leads to fear). With this paper, we propose to make such interpretations of events explicit, following theories of cognitive appraisal of events, and show their potential for emotion classification when being encoded in classification models. Our results show that high quality appraisal dimension assignments in event descriptions lead to an improvement in the classification of discrete emotion categories. We make our corpus of appraisal-annotated emotion-associated event descriptions publicly available. KW - Emotion Classification ER - TY - CPAPER TI - Emotion-Aware, Emotion-Agnostic, or Automatic : Corpus Creation Strategies to Obtain Cognitive Event Appraisal Annotations AU - Hofmann, Jan AU - Troiano, Enrica AU - Klinger, Roman PY - 2025 SP - 160-170 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108339 LA - eng AB - Appraisal theories explain how the cognitive evaluation of an event leads to a particular emotion. In contrast to theories of basic emotions or affect (valence/arousal), this theory has not received a lot of attention in natural language processing. Yet, in psychology it has been proven powerful: Smith and Ellsworth (1985) showed that the appraisal dimensions attention, certainty, anticipated effort, pleasantness, responsibility/control and situational control discriminate between (at least) 15 emotion classes. We study different annotation strategies for these dimensions, based on the event-focused enISEAR corpus (Troiano et al., 2019). We analyze two manual annotation settings: (1) showing the text to annotate while masking the experienced emotion label; (2) revealing the emotion associated with the text. Setting 2 enables the annotators to develop a more realistic intuition of the described event, while Setting 1 is a more standard annotation procedure, purely relying on text. We evaluate these strategies in two ways: by measuring inter-annotator agreement and by fine- tuning RoBERTa to predict appraisal variables. Our results show that knowledge of the emotion increases annotators’ reliability. Further, we evaluate a purely automatic rule-based labeling strategy (inferring appraisal from annotated emotion classes). Training on automatically assigned labels leads to a competitive performance of our classifier, even when tested on manual annotations. This is an indicator that it might be possible to automatically create appraisal corpora for every domain for which emotion corpora already exist. KW - Corpus Creation Strategies ER - TY - CPAPER TI - Emotion Ratings : How Intensity, Annotation Confidence and Agreements are Entangled AU - Troiano, Enrica AU - Padó, Sebastian AU - Klinger, Roman PY - 2025 SP - 40-49 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108340 LA - eng AB - When humans judge the affective content of texts, they also implicitly assess the correctness of such judgment, that is, their confidence. We hypothesize that people’s (in)confidence that they performed well in an annotation task leads to (dis)agreements among each other. If this is true, confidence may serve as a diagnostic tool for systematic differences in annotations. To probe our assumption, we conduct a study on a subset of the Corpus of Contemporary American English, in which we ask raters to distinguish neutral sentences from emotion-bearing ones, while scoring the confidence of their answers. Confidence turns out to approximate inter-annotator disagreements. Further, we find that confidence is correlated to emotion intensity: perceiving stronger affect in text prompts annotators to more certain classification performances. This insight is relevant for modelling studies of intensity, as it opens the question wether automatic regressors or classifiers actually predict intensity, or rather human’s self-perceived confidence. KW - Emotion Ratings ER - TY - CPAPER TI - Experiencers, Stimuli, or Targets : Which Semantic Roles Enable Machine Learning to Infer the Emotions? AU - Oberländer, Laura AU - Reich, Kevin AU - Klinger, Roman PY - 2025 SP - 119–128 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108342 LA - eng AB - Emotion recognition is predominantly formulated as text classification in which textual units are assigned to an emotion from a predefined inventory (e.g., fear, joy, anger, disgust, sadness, surprise, trust, anticipation). More recently, semantic role labeling approaches have been developed to extract structures from the text to answer questions like: “who is described to feel the emotion?” (experiencer), “what causes this emotion?” (stimulus), and at which entity is it directed?” (target). Though it has been shown that jointly modeling stimulus and emotion category prediction is beneficial for both subtasks, it remains unclear which of these semantic roles enables a classifier to infer the emotion. Is it the experiencer, because the identity of a person is biased towards a particular emotion (X is always happy)? Is it a particular target (everybody loves X) or a stimulus (doing X makes everybody sad)? We answer these questions by training emotion classification models on five available datasets annotated with at least one semantic role by masking the fillers of these roles in the text in a controlled manner and find that across multiple corpora, stimuli and targets carry emotion information, while the experiencer might be considered a confounder. Further, we analyze if informing the model about the position of the role improves the classification decision. Particularly on literature corpora we find that the role information improves the emotion classification. KW - Machine Learning ER - TY - CPAPER TI - Multi-Task Learning with Sentiment, Emotion, and Target Detection to Recognize Hate Speech and Offensive Language AU - Plaza-del-Arco, Flor Miriam AU - Halat, Sercan AU - Padó, Sebastian AU - Klinger, Roman PY - 2025 SP - 297-318 PB - Otto-Friedrich-Universität CY - Bamberg T3 - CEUR Workshop Proceedings UR - https://fis.uni-bamberg.de/handle/uniba/108352 LA - eng AB - The recognition of hate speech and offensive language (HOF) is commonly formulated as a classification task which asks models to decide if a text contains HOF. This task is challenging because of the large variety of explicit and implicit ways to verbally attack a target person or group. In this paper, we investigate whether HOF detection can profit by taking into account the relationships between HOF and similar concepts: (a) HOF is related to sentiment analysis because hate speech is typically a negative statement and expresses a negative opinion; (b) it is related to emotion analysis, as expressed hate points to the author experiencing (or pretending to experience) anger while the addressees experience (or are intended to experience) fear. (c) Finally, one constituting element of HOF is the (explicit or implicit) mention of a targeted person or group. On this basis, we hypothesize that HOF detection shows improvements when being modeled jointly with these concepts, in a multi-task learning setup. We base our experiments on existing data sets for each of these concepts (sentiment, emotion, target of HOF) and evaluate our models as a participant (as team IMS-SINAI) in the HASOC FIRE 2021 English Subtask 1A: “Subtask 1A: Identifying Hate, offensive and profane content from the post”. Based on model-selection experiments in which we consider multiple available resources and submissions to the shared task, we find that the combination of the CrowdFlower emotion corpus, the SemEval 2016 Sentiment Corpus, and the OffensEval 2019 target detection data leads to an F1 =.7947 in a multi-head multi-task learning model based on BERT, in comparison to .7895 of a plain BERT model. On the HASOC 2019 test data, this result is more substantial with an increase by 2pp in F1 (from 0.78 F1 to 0.8 F1) and a considerable increase in recall. Across both data sets (2019, 2021), the recall is particularly increased for the class of HOF (6pp for the 2019 data and 3pp for the 2021 data), showing that MTL with emotion, sentiment, and target identification is an appropriate approach for early warning systems that might be deployed in social media platforms. KW - multi-task learning KW - hate and offensive language detection KW - sentiment analysis KW - emotion analysis KW - target classification KW - social media mining ER - TY - CPAPER TI - What Makes Medical Claims (Un)Verifiable? : Analyzing Entity and Relation Properties for Fact Verification AU - Wührl, Amelie AU - Menchaca Resendiz, Yarik AU - Grimminger, Lara AU - Klinger, Roman PY - 2025 SP - 2046–2058 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108355 LA - eng AB - Verifying biomedical claims fails if no evidence can be discovered. In these cases, the fact-checking verdict remains unknown and the claim is unverifiable. To improve this situation, we have to understand if there are any claim properties that impact its verifiability. In this work we assume that entities and relations define the core variables in a biomedical claim’s anatomy and analyze if their properties help us to differentiate verifiable from unverifiable claims. In a study with trained annotation experts we prompt them to find evidence for biomedical claims, and observe how they refine search queries for their evidence search. This leads to the first corpus for scientific fact verification annotated with subject–relation–object triplets, evidence documents, and fact-checking verdicts (the BEAR-FACT corpus). We find (1) that discovering evidence for negated claims (e.g., X–does-not-cause–Y) is particularly challenging. Further, we see that annotators process queries mostly by adding constraints to the search and by normalizing entities to canonical names. (2) We compare our in-house annotations with a small crowdsourcing setting where we employ both medical experts and laypeople. We find that domain expertise does not have a substantial effect on the reliability of annotations. Finally, (3), we demonstrate that it is possible to reliably estimate the success of evidence retrieval purely from the claim text (.82F1), whereas identifying unverifiable claims proves more challenging (.27F1) KW - Medical Claims ER - TY - CPAPER TI - Crowdsourcing and Validating Event-focused Emotion Corpora for German and English AU - Troiano, Enrica AU - Padó, Sebastian AU - Klinger, Roman PY - 2025 SP - 4005-4011 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108306 LA - eng AB - Sentiment analysis has a range of corpora available across multiple languages. For emotion analysis, the situation is more limited, which hinders potential research on crosslingual modeling and the development of predictive models for other languages. In this paper, we fill this gap for German by constructing deISEAR, a corpus designed in analogy to the well-established English ISEAR emotion dataset. Motivated by Scherer’s appraisal theory, we implement a crowdsourcing experiment which consists of two steps. In step 1, participants create descriptions of emotional events for a given emotion. In step 2, five annotators assess the emotion expressed by the texts. We show that transferring an emotion classification model from the original English ISEAR to the German crowdsourced deISEAR via machine translation does not, on average, cause a performance drop. KW - Crowdsourcing ER - TY - CPAPER TI - Investigating the Relationship between Literary Genres and Emotional Plot Development AU - Kim, Evgeny AU - Padó, Sebastian AU - Klinger, Roman PY - 2025 SP - 17-26 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108284 LA - eng AB - Literary genres are commonly viewed as being defined in terms of content and stylistic features. In this paper, we focus on one particular class of lexical features, namely emotion information, and investigate the hypothesis that emotion-related information correlates with particular genres. Using genre classification as a testbed, we compare a model that computes lexicon-based emotion scores globally for complete stories with a model that tracks emotion arcs through stories on a subset of Project Gutenberg with five genres. Our main findings are: (a), the global emotion model is competitive with a large-vocabulary bag-of-words genre classifier (80%F1); (b), the emotion arc model shows a lower performance (59 % F1) but shows complementary behavior to the global model, as indicated by a very good performance of an oracle model (94 % F1) and an improved performance of an ensemble model (84 % F1); (c), genres differ in the extent to which stories follow the same emotional arcs, with particularly uniform behavior for anger (mystery) and fear (adventures, romance, humor, science fiction). KW - Literary Genres KW - Emotional Plot Development ER - TY - CPAPER TI - IMS at EmoInt-2017 : Emotion Intensity Prediction with Affective Norms, Automatically Extended Resources and Deep Learning AU - Köper, Maximilian AU - Kim, Evgeny AU - Klinger, Roman PY - 2025 SP - 50-57 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108285 LA - eng AB - Our submission to the WASSA-2017 shared task on the prediction of emotion intensity in tweets is a supervised learning method with extended lexicons of affective norms. We combine three main information sources in a random forrest regressor, namely (1), manually created resources, (2) automatically extended lexicons, and (3) the output of a neural network (CNN-LSTM) for sentence regression. All three feature sets perform similarly well in isolation (≈ .67 macro average Pearson correlation). The combination achieves .72 on the official test set (ranked 2nd out of 22 participants). Our analysis reveals that performance is increased by providing cross-emotional intensity predictions. The automatic extension of lexicon features benefit from domain specific embeddings. Complementary ratings for affective norms increase the impact of lexicon features. Our resources (ratings for 1.6 million twitter specific words) and our implementation is publicly available at http://www.ims.uni-stuttgart.de/data/ims_emoint. KW - Emotion Intensity Prediction ER - TY - CPAPER TI - Assessing State-of-the-Art Sentiment Models on State-of-the-Art Sentiment Datasets AU - Barnes, Jeremy AU - Klinger, Roman AU - Schulte im Walde, Sabine PY - 2025 SP - 2-12 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108286 LA - eng AB - There has been a good amount of progress in sentiment analysis over the past 10 years, including the proposal of new methods and the creation of benchmark datasets. In some papers, however, there is a tendency to compare models only on one or two datasets, either because of time restraints or because the model is tailored to a specific task. Accordingly, it is hard to understand how well a certain model generalizes across different tasks and datasets. In this paper, we contribute to this situation by comparing several models on six different benchmarks, which belong to different domains and additionally have different levels of granularity (binary, 3-class, 4-class and 5-class). We show that Bi-LSTMs perform well across datasets and that both LSTMs and Bi-LSTMs are particularly good at fine-grained sentiment tasks (i.e., with more than two classes). Incorporating sentiment information into word embeddings during training gives good results for datasets that are lexically similar to the training data. With our experiments, we contribute to a better understanding of the performance of different model architectures on different data sets. Consequently, we detect novel state-of-the-art results on the SenTube datasets. KW - State-of-the-Art Sentiment Models ER - TY - CPAPER TI - Annotation, Modelling and Analysis of Fine-Grained Emotions on a Stance and Sentiment Detection Corpus AU - Schuff, Hendrik AU - Barnes, Jeremy AU - Mohme, Julian AU - Padó, Sebastian AU - Klinger, Roman PY - 2025 SP - 13–23 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108287 LA - eng AB - There is a rich variety of data sets for sentiment analysis (viz., polarity and subjectivity classification). For the more challenging task of detecting discrete emotions following the definitions of Ekman and Plutchik, however, there are much fewer data sets, and notably no resources for the social media domain. This paper contributes to closing this gap by extending the SemEval 2016 stance and sentiment datasetwith emotion annotation. We (a) analyse annotation reliability and annotation merging; (b) investigate the relation between emotion annotation and the other annotation layers (stance, sentiment); (c) report modelling results as a baseline for future work. KW - Fine-Grained Emotions ER - TY - CPAPER TI - Ranking Right-Wing Extremist Social Media Profiles by Similarity to Democratic and Extremist Groups AU - Hartung, Matthias AU - Klinger, Roman AU - Schmidtke, Franziska AU - Vogel, Lars PY - 2025 SP - 24–33 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108288 LA - eng AB - Social media are used by an increasing number of political actors. A small subset of these is interested in pursuing extremist motives such as mobilization, recruiting or radicalization activities. In order to counteract these trends, online providers and state institutions reinforce their monitoring efforts, mostly relying on manual workflows. We propose a machine learning approach to support manual attempts towards identifying right-wing extremist content in German Twitter profiles. Based on a fine-grained conceptualization of right-wing extremism, we frame the task as ranking each individual profile on a continuum spanning different degrees of right-wing extremism, based on a nearest neighbour approach. A quantitative evaluation reveals that our ranking model yields robust performance (up to 0.81 F1 score) when being used for predicting discrete class labels. At the same time, the model provides plausible continuous ranking scores for a small sample of borderline cases at the division of right-wing extremism and New Right political movements. KW - Social Media Profiles ER - TY - CPAPER TI - Frowning Frodo, Wincing Leia, and a Seriously Great Friendship : Learning to Classify Emotional Relationships of Fictional Characters AU - Kim, Evgeny AU - Klinger, Roman PY - 2025 SP - 647-653 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108305 LA - eng AB - The development of a fictional plot is centered around characters who closely interact with each other forming dynamic social networks. In literature analysis, such networks have mostly been analyzed without particular relation types or focusing on roles which the characters take with respect to each other. We argue that an important aspect for the analysis of stories and their development is the emotion between characters. In this paper, we combine these aspects into a unified framework to classify emotional relationships of fictional characters. We formalize it as a new task and describe the annotation of a corpus, based on fan-fiction short stories. The extraction pipeline which we propose consists of character identification (which we treat as given by an oracle here) and the relation classification. For the latter, we provide results using several approaches previously proposed for relation identification with neural methods. The best result of 0.45 F1 is achieved with a GRU with character position indicators on the task of predicting undirected emotion relations in the associated social network graph. KW - Emotional Relationships ER - TY - CPAPER TI - Model Architectures for Quotation Detection AU - Scheible, Christian AU - Klinger, Roman AU - Padó, Sebastian PY - 2025 SP - 1736-1745 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108290 LA - eng AB - Quotation detection is the task of locating spans of quoted speech in text. The state of the art treats this problem as a sequence labeling task and employs linear-chain conditional random fields. We question the efficacy of this choice: The Markov assumption in the model prohibits it from making joint decisions about the begin, end, and internal context of a quotation. We perform an extensive analysis with two new model architectures. We find that (a), simple boundary classification combined with a greedy prediction strategy is competitive with the state of the art; (b), a semi-Markov model significantly outperforms all others, by relaxing the Markov assumption. KW - Model Architectures ER - TY - CPAPER TI - Adversarial Training for Satire Detection : Controlling for Confounding Variables AU - McHardy, Robert AU - Adel, Heike AU - Klinger, Roman PY - 2025 SP - 660-665 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108291 LA - eng AB - The automatic detection of satire vs. regular news is relevant for downstream applications (for instance, knowledge base population) and to improve the understanding of linguistic characteristics of satire. Recent approaches build upon corpora which have been labeled automatically based on article sources. We hypothesize that this encourages the models to learn characteristics for different publication sources (e.g., “The Onion” vs. “The Guardian”) rather than characteristics of satire, leading to poor generalization performance to unseen publication sources. We therefore propose a novel model for satire detection with an adversarial component to control for the confounding variable of publication source. On a large novel data set collected from German news (which we make available to the research community), we observe comparable satire classification performance and, as desired, a considerable drop in publication classification performance with adversarial training. Our analysis shows that the adversarial component is crucial for the model to learn to pay attention to linguistic properties of satire. KW - Satire Detection ER - TY - JOUR TI - Deviations from cultural consensus about occupations : The duality of occupation meanings and Americans’ meaning communities AU - Combs, Aidan AU - Varela, Gabriel AU - Robinson, Dawn T. AU - Smith-Lovin, Lynn AU - Vaisey, Stephen PY - 2025 SP - 153-165 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108383 LA - eng AB - We examine ratings of 642 occupations by a national online sample of U.S respondents in 2019 (Freeland et al., 2020). We analyze the respondents’ ratings of occupations on three dimensions of cultural meaning—evaluation (good versus bad), potency (powerful versus powerless), and activity (lively versus quiet). We take deviations of respondents’ individual ratings from population evaluation, potency and activity estimates, focusing on deviations from consensus rather than consensus itself. Drawing on Breiger's (1974) work on duality, we examine two projections of the initial rectangular matrix of correlated deviations. Our two projections represent (1) the cultural communities that people form when they differ from consensus in similar ways, and (2) the clusters of occupations that move in similar ways across those subcultures. Correlations among the residuals at the person level are indicators of shared subcultural differences from the mainstream—different ways of meaning-making about what is valuable and worthy about occupational work. At the occupation level, the structure represents schemas for which occupations share common elements and move together when those elements are evaluated differently. We use dyad models to investigate what metrics of occupation similarity predict similarity in deviations from consensus. We find that similarity in affective meaning (evaluation, potency and activity), material requirements, rewards, and work characteristics all predict clustering at the occupation level. Demographic composition of occupations is less important. We find that older respondents, White respondents, and higher income respondents tend to discriminate more between occupations on evaluation and potency. Respondents who are more similar in age have more similar patterns of deviations. However, occupation-level variables are in general much stronger predictors of residual structure than respondent-level variables. KW - Duality KW - Occupations KW - Culture KW - Cultural consensus ER - TY - CPAPER TI - Projecting Embeddings for Domain Adaption : Joint Modeling of Sentiment Analysis in Diverse Domains AU - Barnes, Jeremy AU - Klinger, Roman AU - Schulte im Walde, Sabine PY - 2025 SP - 818-830 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108292 LA - eng AB - Domain adaptation for sentiment analysis is challenging due to the fact that supervised classifiers are very sensitive to changes in domain. The two most prominent approaches to this problem are structural correspondence learning and autoencoders. However, they either require long training times or suffer greatly on highly divergent domains. Inspired by recent advances in cross-lingual sentiment analysis, we provide a novel perspective and cast the domain adaptation problem as an embedding projection task. Our model takes as input two mono-domain embedding spaces and learns to project them to a bi-domain space, which is jointly optimized to (1) project across domains and to (2) predict sentiment. We perform domain adaptation experiments on 20 source-target domain pairs for sentiment classification and report novel state-of-the-art results on 11 domain pairs, including the Amazon domain adaptation datasets and SemEval 2013 and 2016 datasets. Our analysis shows that our model performs comparably to state-of-the-art approaches on domains that are similar, while performing significantly better on highly divergent domains. Our code is available at https://github.com/jbarnesspain/domain_blse KW - Projecting Embeddings ER - TY - CPAPER TI - DERE : A Task and Domain-Independent Slot Filling Framework for Declarative Relation Extraction AU - Adel, Heike AU - Bostan, Laura Ana Maria AU - Papay, Sean AU - Padó, Sebastian AU - Klinger, Roman PY - 2025 SP - 42–47 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108297 LA - eng AB - Most machine learning systems for natural language processing are tailored to specific tasks. As a result, comparability of models across tasks is missing and their applicability to new tasks is limited. This affects end users without machine learning experience as well as model developers. To address these limitations, we present DERE, a novel framework for declarative specification and compilation of template-based information extraction. It uses a generic specification language for the task and for data annotations in terms of spans and frames. This formalism enables the representation of a large variety of natural language processing challenges. The backend can be instantiated by different models, following different paradigms. The clear separation of frame specification and model backend will ease the implementation of new models and the evaluation of different models across different tasks. Furthermore, it simplifies transfer learning, joint learning across tasks and/or domains as well as the assessment of model generalizability. DERE is available as open-source software. KW - DERE ER - TY - CPAPER TI - Who Feels What and Why? : Annotation of a Literature Corpus with Semantic Roles of Emotions AU - Kim, Evgeny AU - Klinger, Roman PY - 2025 SP - 1345-1359 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108294 LA - eng AB - Most approaches to emotion analysis in fictional texts focus on detecting the emotion expressed in text. We argue that this is a simplification which leads to an overgeneralized interpretation of the results, as it does not take into account who experiences an emotion and why. Emotions play a crucial role in the interaction between characters and the events they are involved in. Until today, no specific corpora that capture such an interaction were available for literature. We aim at filling this gap and present a publicly available corpus based on Project Gutenberg, REMAN (Relational EMotion ANnotation), manually annotated for spans which correspond to emotion trigger phrases and entities/events in the roles of experiencers, targets, and causes of the emotion. We provide baseline results for the automatic prediction of these relational structures and show that emotion lexicons are not able to encompass the high variability of emotion expressions and demonstrate that statistical models benefit from joint modeling of emotions with its roles in all subtasks. The corpus that we provide enables future research on the recognition of emotions and associated entities in text. It supports qualitative literary studies and digital humanities. The corpus is available at http://www.ims.uni-stuttgart.de/data/reman . KW - Literature Corpus ER - TY - RPRT TI - DFG Final Report for Automatic Fact Checking for Biomedical Information in Social Media and Scientific Literature (FIBISS), project number 667374 AU - Klinger, Roman AU - Wührl, Amelie PY - 2025 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/107867 LA - eng AB - Research into methods for the automatic verification of facts, i.e., computational models that can distinguish correct information from misinformation or disinformation, is largely focused on the news domain and on the analysis of posts in social media. Among other things, texts are checked for their truthfulness. This can be done by analyzing linguistic features that suggest an intention to deceive or by comparing them with other sources that make comparable statements in terms of content. Most studies focus on politically relevant areas. The biomedical domain is also an area of particular social relevance. In social media, various actors and medical laypersons share reports on treatment methods, successes and failures, such as the (disproven) method of treating viral infections with deworming agents or disinfectants. There are also reports on (disproven) links between treatments and adverse effects, such as the causation of autism by vaccination. However, the biomedical domain, unlike other areas relevant for automated fact checking, benefits from a large resource of reliable scientific articles. The aim of the FIBISS project was therefore to develop and evaluate methods that can extract biomedical claims in social media and compare them with reliable sources. One challenge here is that social media does not typically use technical language, so different vocabularies have to be combined. The approach in FIBISS was therefore to develop generalizing information extraction methods. In the course of the project, large language models also became prominent as a further methodological approach. The project was therefore adapted to optimize general representations of claims in such a way that they are suitable for comparison using automatic fact-checking procedures. As a result, we contribute text corpora that are used to develop and evaluate automated biomedical fact-checking systems. We propose methods that automatically reformulate claims so that they are suitable to be automatically verified. Furthermore, we present approaches that can automatically assess the credibility of claims, even independently of existing evidence. KW - Projektabschlussbericht KW - DFG KW - Final project report ER - TY - CPAPER TI - Strategies for political-statement segmentation and labelling in unstructured text AU - Nikolaev, Dmitry AU - Papay, Sean PY - 2025 T2 - Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities SP - 437–451 PB - Association for Computational Linguistics CY - Stroudsburg, PA DO - 10.18653/v1/2025.nlp4dh-1.38 UR - https://fis.uni-bamberg.de/handle/uniba/108001 LA - eng AB - Analysis of parliamentary speeches and political-party manifestos has become an integral area of computational study of political texts. While speeches have been overwhelmingly analysed using unsupervised methods, a large corpus of manifestos with by-statement political-stance labels has been created by the participants of the MARPOR project. It has been recently shown that these labels can be predicted by a neural model; however, the current approach relies on provided statement boundaries, limiting out-of-domain applicability. In this work, we propose and test a range of unified split-and-label frameworks—based on linear-chain CRFs, fine-tuned text-to-text models, and the combination of in-context learning with constrained decoding—that can be used to jointly segment and classify statements from raw textual data. We show that our approaches achieve competitive accuracy when applied to raw text of political manifestos, and then demonstrate the research potential of our method by applying it to the records of the UK House of Commons and tracing the political trajectories of four major parties in the last three decades. KW - manifesto KW - corpus KW - annotation ER - TY - RPRT TI - DFG Final Report for Automatic Fact Checking for Biomedical Information in Social Media and Scientific Literature (FIBISS), project number 667374 AU - Klinger, Roman AU - Wührl, Amelie PY - 2025 PB - Technische Informationsbibliothek CY - Hannover DO - 10.34657/17863 UR - https://fis.uni-bamberg.de/handle/uniba/107866 LA - eng AB - Research into methods for the automatic verification of facts, i.e., computational models that can distinguish correct information from misinformation or disinformation, is largely focused on the news domain and on the analysis of posts in social media. Among other things, texts are checked for their truthfulness. This can be done by analyzing linguistic features that suggest an intention to deceive or by comparing them with other sources that make comparable statements in terms of content. Most studies focus on politically relevant areas. The biomedical domain is also an area of particular social relevance. In social media, various actors and medical laypersons share reports on treatment methods, successes and failures, such as the (disproven) method of treating viral infections with deworming agents or disinfectants. There are also reports on (disproven) links between treatments and adverse effects, such as the causation of autism by vaccination. However, the biomedical domain, unlike other areas relevant for automated fact checking, benefits from a large resource of reliable scientific articles. The aim of the FIBISS project was therefore to develop and evaluate methods that can extract biomedical claims in social media and compare them with reliable sources. One challenge here is that social media does not typically use technical language, so different vocabularies have to be combined. The approach in FIBISS was therefore to develop generalizing information extraction methods. In the course of the project, large language models also became prominent as a further methodological approach. The project was therefore adapted to optimize general representations of claims in such a way that they are suitable for comparison using automatic fact-checking procedures. As a result, we contribute text corpora that are used to develop and evaluate automated biomedical fact-checking systems. We propose methods that automatically reformulate claims so that they are suitable to be automatically verified. Furthermore, we present approaches that can automatically assess the credibility of claims, even independently of existing evidence. KW - Projektabschlussbericht KW - DFG KW - Final project report ER - TY - CPAPER TI - Fearful Falcons and Angry Llamas : Emotion Category Annotations of Arguments by Humans and LLMs AU - Greschner, Lynn AU - Klinger, Roman PY - 2025 T2 - Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities SP - 628–646 PB - Association for Computational Linguistics (ACL) CY - Stroudsburg, PA SN - 979-8-89176-234-3 DO - 10.18653/v1/2025.nlp4dh-1.52 UR - https://fis.uni-bamberg.de/handle/uniba/107836 LA - eng AB - Arguments evoke emotions, influencing the effect of the argument itself. Not only the emotional intensity but also the category influences the argument’s effects, for instance, the willingness to adapt stances. While binary emotionality has been studied in argumentative texts, there is no work on discrete emotion categories (e.g., ‘anger’) in such data. To fill this gap, we crowdsource subjective annotations of emotion categories in a German argument corpus and evaluate automatic LLM-based labeling methods. Specifically, we compare three prompting strategies (zero-shot, one-shot, chain-of-thought) on three large instruction-tuned language models (Falcon-7b-instruct, Llama-3.1-8B-instruct, GPT-4o-mini). We further vary the definition of the output space to be binary (is there emotionality in the argument?), closed-domain (which emotion from a given label set is in the argument?), or open-domain (which emotion is in the argument?). We find that emotion categories enhance the prediction of emotionality in arguments, emphasizing the need for discrete emotion annotations in arguments. Across all prompt settings and models, automatic predictions show a high recall but low precision for predicting anger and fear, indicating a strong bias toward negative emotions. KW - argument KW - emotion KW - corpus KW - large language model ER - TY - JOUR TI - Deviations from cultural consensus about occupations : The duality of occupation meanings and Americans’ meaning communities AU - Combs, Aidan AU - Varela, Gabriel AU - Robinson, Dawn T. AU - Smith-Lovin, Lynn AU - Vaisey, Stephen PY - 2025 JO - Social networks : an international journal of structural analysis VL - 82 SP - 153-165 PB - Elsevier Science CY - Amsterdam [u.a.] SN - 0378-8733 DO - 10.1016/j.socnet.2025.04.003 UR - https://fis.uni-bamberg.de/handle/uniba/107736 LA - eng AB - We examine ratings of 642 occupations by a national online sample of U.S respondents in 2019 (Freeland et al., 2020). We analyze the respondents’ ratings of occupations on three dimensions of cultural meaning—evaluation (good versus bad), potency (powerful versus powerless), and activity (lively versus quiet). We take deviations of respondents’ individual ratings from population evaluation, potency and activity estimates, focusing on deviations from consensus rather than consensus itself. Drawing on Breiger's (1974) work on duality, we examine two projections of the initial rectangular matrix of correlated deviations. Our two projections represent (1) the cultural communities that people form when they differ from consensus in similar ways, and (2) the clusters of occupations that move in similar ways across those subcultures. Correlations among the residuals at the person level are indicators of shared subcultural differences from the mainstream—different ways of meaning-making about what is valuable and worthy about occupational work. At the occupation level, the structure represents schemas for which occupations share common elements and move together when those elements are evaluated differently. We use dyad models to investigate what metrics of occupation similarity predict similarity in deviations from consensus. We find that similarity in affective meaning (evaluation, potency and activity), material requirements, rewards, and work characteristics all predict clustering at the occupation level. Demographic composition of occupations is less important. We find that older respondents, White respondents, and higher income respondents tend to discriminate more between occupations on evaluation and potency. Respondents who are more similar in age have more similar patterns of deviations. However, occupation-level variables are in general much stronger predictors of residual structure than respondent-level variables. KW - Duality KW - Occupations KW - Culture KW - Cultural consensus ER - TY - CPAPER TI - MOPO: Multi-Objective Prompt Optimization for Affective Text Generation AU - Menchaca Resendiz, Yarik AU - Klinger, Roman PY - 2025 T2 - Proceedings of the 31st International Conference on Computational Linguistics SP - 5588–5606 PB - Association for Computational Linguistics CY - Stroudsburg, PA SN - 979-8-89176-196-4 UR - https://fis.uni-bamberg.de/handle/uniba/105971 LA - eng ER - TY - THES TI - Task generality in relation extraction AU - Papay, Sean PY - 2024 PB - Universitätsbibliothek der Universität Stuttgart CY - Stuttgart DO - https://doi.org/10.18419/opus-14421 UR - https://fis.uni-bamberg.de/handle/uniba/115309 LA - eng ER - TY - CPAPER TI - Actor Identification in Discourse : A Challenge for LLMs? AU - Barić, Ana AU - Padó, Sebastian AU - Papay, Sean PY - 2024 T2 - Proceedings of the 5th Workshop on Computational Approaches to Discourse (CODI 2024) SP - 64–70 PB - Association for Computational Linguistics DO - 10.18653/v1/2024.codi-1.6 UR - https://fis.uni-bamberg.de/handle/uniba/115293 LA - eng AB - The identification of political actors who put forward claims in public debate is a crucial step in the construction of discourse networks, which are helpful to analyze societal debates. Actor identification is, however, rather challenging: Often, the locally mentioned speaker of a claim is only a pronoun (“He proposed that [claim]”), so recovering the canonical actor name requires discourse understanding. We compare a traditional pipeline of dedicated NLP components (similar to those applied to the related task of coreference) with a LLM, which appears a good match for this generation task. Evaluating on a corpus of German actors in newspaper reports, we find surprisingly that the LLM performs worse. Further analysis reveals that the LLM is very good at identifying the right reference, but struggles to generate the correct canonical form. This points to an underlying issue in LLMs with controlling generated output. Indeed, a hybrid model combining the LLM with a classifier to normalize its output substantially outperforms both initial models. KW - Actor Identification ER - TY - CPAPER TI - Automatic Analysis of Political Debates and Manifestos : Successes and Challenges AU - Ceron, Tanise AU - Barić, Ana AU - Blessing, André AU - Haunss, Sebastian AU - Kuhn, Jonas AU - Lapesa, Gabriella AU - Padó, Sebastian AU - Papay, Sean AU - Zauchner, Patricia F. PY - 2024 T2 - Robust argumentation machines : first international conference, RATIO 2024, Bielefeld, Germany, June 5-7, 2024 : proceedings SP - 71–88 PB - Springer Nature CY - Cham, Switzerland SN - 978-3-031-63536-6 DO - 10.1007/978-3-031-63536-6_5 UR - https://fis.uni-bamberg.de/handle/uniba/115294 LA - eng AB - The opinions of political actors (e.g., politicians, parties, organizations) expressed through claims are the core elements of political debates and decision-making. Political actors communicate through different channels: parties publish manifestos for major elections, while individual actors make statements on a day-to-day basis as reflected in the media. These two channels offer different approaches for analysis: Manifestos, on the one hand, are useful to characterize the parties’ positions at a global ideological level over time. In contrast, individual statements can be collected to analyze debates in particular policy domains on a fine-grained level, in terms of individual actors and claims. In this article, we summarize a series of studies we have carried out. We apply NLP-driven (semi-)automatic analyses on these two channels and compare their potentials and challenges. The fine-grained analysis yields rich insights into the communication but comes at the cost of three challenges: (a) a substantial hunger for manual annotation, introducing practical hurdles for analysis both within and across languages; (b) difficulties in claim classification arising from the uneven frequency distribution over the theory-based annotation schemas; (c) the need to map actor mentions onto canonical versions. Manifesto-based analysis avoids these challenges to a substantial extent when a more coarse-grained analysis of party positions is sufficient. We highlight the benefits and challenges of both approaches, and conclude by outlining perspectives for addressing the challenges in future research. KW - Claim identification KW - discourse network analysis KW - party positioning KW - argument mining ER - TY - CPAPER TI - Sentence‑Final Particle de in Mandarin as an Informativity Maximizer AU - Chen, Jun AU - Papay, Sean PY - 2024 T2 - Selected Reflections in Language, Logic, and Information : ESSLLI 2019, ESSLLI 2020 and ESSLLI 2021 Student Sessions, Selected Papers SP - 24–43 PB - Springer Nature Switzerland CY - Cham SN - 978-3-031-50627-7 DO - 10.1007/978-3-031-50628-4_3 UR - https://fis.uni-bamberg.de/handle/uniba/115198 LA - eng ER - TY - CPAPER TI - Dealing with Controversy : An Emotion and Coping Strategy Corpus Based on Role Playing AU - Troiano, Enrica AU - Labat, Sofie AU - Stranisci, Marco Antonio AU - Damiano, Rossana AU - Patti, Viviana AU - Klinger, Roman PY - 2024 T2 - Findings of the Association for Computational Linguistics : EMNLP 2024 SP - 1634–1658 PB - Association for Computational Linguistics CY - Miami, Florida UR - https://fis.uni-bamberg.de/handle/uniba/104594 LA - eng ER - TY - CPAPER TI - How Entangled is Factuality and Deception in German? AU - Velutharambath, Aswathy AU - Wuehrl, Amelie AU - Klinger, Roman PY - 2024 T2 - Findings of the Association for Computational Linguistics : EMNLP 2024 SP - 9538–9554 PB - Association for Computational Linguistics UR - https://fis.uni-bamberg.de/handle/uniba/104595 LA - eng AB - The statement “The earth is flat” is factually inaccurate, but if someone truly believes and argues in its favor, it is not deceptive. Research on deception detection and fact checking often conflates factual accuracy with the truthfulness of statements. This assumption makes it difficult to (a) study subtle distinctions and interactions between the two and (b) gauge their effects on downstream tasks. The belief-based deception framework disentangles these properties by defining texts as deceptive when there is a mismatch between what people say and what they truly believe. In this study, we assess if presumed patterns of deception generalize to German language texts. We test the effectiveness of computational models in detecting deception using an established corpus of belief-based argumentation. Finally, we gauge the impact of deception on the downstream task of fact checking and explore if this property confounds verification models. Surprisingly, our analysis finds no correlation with established cues of deception. Previous work claimed that computational models can outperform humans in deception detection accuracy, however, our experiments show that both traditional and state-of-the-art models struggle with the task, performing no better than random guessing. For fact checking, we find that natural language inference-based verification performs worse on non-factual and deceptive content, while prompting large language models for the same task is less sensitive to these properties. KW - Factuality KW - Deception ER - TY - CPAPER TI - Understanding Fine-grained Distortions in Reports of Scientific Findings AU - Wührl, Amelie AU - Wright, Dustin AU - Klinger, Roman AU - Augenstein, Isabelle PY - 2024 T2 - Findings of the Association for Computational Linguistics ACL 2024 SP - 6175–6191 PB - Association for Computational Linguistics CY - Bangkok, Thailand and virtual meeting DO - 10.18653/v1/2024.findings-acl.369 UR - https://fis.uni-bamberg.de/handle/uniba/98199 LA - eng AB - Distorted science communication harms individuals and society as it can lead to unhealthy behavior change and decrease trust in scientific institutions. Given the rapidly increasing volume of science communication in recent years, a fine-grained understanding of how findings from scientific publications are reported to the general public, and methods to detect distortions from the original work automatically, are crucial. Prior work focused on individual aspects of distortions or worked with unpaired data. In this work, we make three foundational contributions towards addressing this problem: (1) annotating 1,600 instances of scientific findings from academic papers paired with corresponding findings as reported in news articles and tweets wrt. four characteristics: causality, certainty, generality and sensationalism; (2) establishing baselines for automatically detecting these characteristics; and (3) analyzing the prevalence of changes in these characteristics in both human-annotated and large-scale unlabeled data. Our results show that scientific findings frequently undergo subtle distortions when reported. Tweets distort findings more often than science news reports. Detecting fine-grained distortions automatically poses a challenging task. In our experiments, fine-tuned task-specific models consistently outperform few-shot LLM prompting. KW - Fine-grained Distortions ER - TY - CPAPER TI - Entity-Level Sentiment : More than the Sum of Its Parts AU - Rønningstad, Egil AU - Klinger, Roman AU - Øvrelid, Lilja AU - Velldal, Erik PY - 2024 T2 - Proceedings of the 14th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis SP - 84–96 PB - Association for Computational Linguistics CY - Bangkok, Thailand DO - 10.18653/v1/2024.wassa-1.8 UR - https://fis.uni-bamberg.de/handle/uniba/98198 LA - eng AB - In sentiment analysis of longer texts, there may be a variety of topics discussed, of entities mentioned, and of sentiments expressed regarding each entity. We find a lack of studies exploring how such texts express their sentiment towards each entity of interest, and how these sentiments can be modelled. In order to better understand how sentiment regarding persons and organizations (each entity in our scope) is expressed in longer texts, we have collected a dataset of expert annotations where the overall sentiment regarding each entity is identified, together with the sentence-level sentiment for these entities separately. We show that the reader’s perceived sentiment regarding an entity often differs from an arithmetic aggregation of sentiments at the sentence level. Only 70% of the positive and 55% of the negative entities receive a correct overall sentiment label when we aggregate the (human-annotated) sentiment labels for the sentences where the entity is mentioned. Our dataset reveals the complexity of entity-specific sentiment in longer texts, and allows for more precise modelling and evaluation of such sentiment expressions. KW - Entity-Level Sentiment ER - TY - CPAPER TI - Hierarchical Adversarial Correction to Mitigate Identity Term Bias in Toxicity Detection AU - Schäfer, Johannes AU - Heid, Ulrich AU - Klinger, Roman PY - 2024 T2 - Proceedings of the 14th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis SP - 35–51 PB - Association for Computational Linguistics CY - Bangkok, Thailand DO - 10.18653/v1/2024.wassa-1.4 UR - https://fis.uni-bamberg.de/handle/uniba/98197 LA - eng AB - Corpora that are the fundament for toxicity detection contain such expressions typically directed against a target individual or group, e.g., people of a specific gender or ethnicity. Prior work has shown that the target identity mention can constitute a confounding variable. As an example, a model might learn that Christians are always mentioned in the context of hate speech. This misguided focus can lead to a limited generalization to newly emerging targets that are not found in the training data. In this paper, we hypothesize and subsequently show that this issue can be mitigated by considering targets on different levels of specificity. We distinguish levels of (1) the existence of a target, (2) a class (e.g., that the target is a religious group), or (3) a specific target group (e.g., Christians or Muslims). We define a target label hierarchy based on these three levels and then exploit this hierarchy in an adversarial correction for the lowest level (i.e. (3)) while maintaining some basic target features. This approach does not lower the toxicity detection performance but increases the generalization to targets not being available at training time. KW - Hierarchical Adversarial Correction ER - TY - CPAPER TI - IMS_medicALY at #SMM4H 2024 : Detecting Impacts of Outdoor Spaces on Social Anxiety with Data Augmented Ensembling AU - Wührl, Amelie AU - Greschner, Lynn AU - Menchaca Resendiz, Yarik AU - Klinger, Roman PY - 2024 T2 - Proceedings of The 9th Social Media Mining for Health Research and Applications (SMM4H 2024) Workshop and Shared Tasks SP - 83–87 PB - Association for Computational Linguistics CY - Bangkok, Thailand UR - https://fis.uni-bamberg.de/handle/uniba/98196 LA - eng AB - Many individuals affected by Social Anxiety Disorder turn to social media platforms to share their experiences and seek advice. This includes discussing the potential benefits of engaging with outdoor environments. As part of #SMM4H 2024, Shared Task 3 focuses on classifying the effects of outdoor spaces on social anxiety symptoms in Reddit posts. In our contribution to the task, we explore the effectiveness of domain-specific models (trained on social media data – SocBERT) against general domain models (trained on diverse datasets – BERT, RoBERTa, GPT-3.5) in predicting the sentiment related to outdoor spaces. Further, we assess the benefits of augmenting sparse human-labeled data with synthetic training instances and evaluate the complementary strengths of domain-specific and general classifiers using an ensemble model. Our results show that (1) fine-tuning small, domain-specific models generally outperforms large general language models in most cases. Only one large language model (GPT-4) exhibits performance comparable to the fine-tuned models (52% F1). Further, we find that (2) synthetic data does improve the performance of fine-tuned models in some cases, and (3) models do not appear to complement each other in our ensemble setup. KW - Data Augmented Ensembling ER - TY - CPAPER TI - CoVERT : A Corpus of Fact-checked Biomedical COVID-19 Tweets AU - Mohr, Isabelle AU - Wührl, Amelie AU - Klinger, Roman PY - 2024 SP - 244-257 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96462 LA - eng AB - During the first two years of the COVID-19 pandemic, large volumes of biomedical information concerning this new disease have been published on social media. Some of this information can pose a real danger, particularly when false information is shared, for instance recommendations how to treat diseases without professional medical advice. Therefore, automatic fact-checking resources and systems developed specifically for medical domain are crucial. While existing fact-checking resources cover COVID-19 related information in news or quantify the amount of misinformation in tweets, there is no dataset providing fact-checked COVID-19 related Twitter posts with detailed annotations for biomedical entities, relations and relevant evidence. We contribute CoVERT, a fact-checked corpus of tweets with a focus on the domain of biomedicine and COVID-19 related (mis)information. The corpus consists of 300 tweets, each annotated with named entities and relations. We employ a novel crowdsourcing methodology to annotate all tweets with fact-checking labels and supporting evidence, which crowdworkers search for online. This methodology results in substantial inter-annotator agreement. Furthermore, we use the retrieved evidence extracts as part of a fact-checking pipeline, finding that the real-world evidence is more useful than the knowledge directly available in pretrained language models. KW - CoVERT ER - TY - CPAPER TI - Exploring fine-tuned embeddings that model intensifiers for emotion analysis AU - Bostan, Laura AU - Klinger, Roman PY - 2024 SP - 25-34 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/95816 LA - eng AB - Adjective phrases like “a little bit surprised”, “completely shocked”, or “not stunned at all” are not handled properly by current state-of-the-art emotion classification and intensity prediction systems. Based on this finding, we analyze differences between embeddings used by these systems in regard to their capability of handling such cases and argue that intensifiers in context of emotion words need special treatment, as is established for sentiment polarity classification, but not for more fine-grained emotion prediction. To resolve this issue, we analyze different aspects of a post-processing pipeline which enriches the word representations of such phrases. This includes expansion of semantic spaces at the phrase level and sub-word level followed by retrofitting to emotion lexicons. We evaluate the impact of these steps with ‘A La Carte and Bag-of-Substrings extensions based on pretrained GloVe,Word2vec, and fastText embeddings against a crowd-sourced corpus of intensity annotations for tweets containing our focus phrases. We show that the fastText-based models do not gain from handling these specific phrases under inspection. For Word2vec embeddings, we show that our post-processing pipeline improves the results by up to 8% on a novel dataset densly populated with intensifiers while it does not decrease the performance on the established EmoInt dataset. KW - fine-tuned embeddings ER - TY - CPAPER TI - IEST : WASSA-2018 Implicit Emotions Shared Task AU - Klinger, Roman AU - De Clercq, Orphée AU - Mohammad, Saif M. AU - Balahur, Alexandra PY - 2024 SP - 31-42 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/95817 LA - eng AB - Past shared tasks on emotions use data with both overt expressions of emotions (I am so happy to see you!) as well as subtle expressions where the emotions have to be inferred, for instance from event descriptions. Further, most datasets do not focus on the cause or the stimulus of the emotion. Here, for the first time, we propose a shared task where systems have to predict the emotions in a large automatically labeled dataset of tweets without access to words denoting emotions. Based on this intention, we call this the Implicit Emotion Shared Task (IEST) because the systems have to infer the emotion mostly from the context. Every tweet has an occurrence of an explicit emotion word that is masked. The tweets are collected in a manner such that they are likely to include a description of the cause of the emotion – the stimulus. Altogether, 30 teams submitted results which range from macro F1 scores of 21 % to 71 %. The baseline (Max-Ent bag of words and bigrams) obtains an F1 score of 60 % which was available to the participants during the development phase. A study with human annotators suggests that automatic methods outperform human predictions, possibly by honing into subtle textual clues not used by humans. Corpora, resources, and results are available at the shared task website at http://implicitemotions.wassa2018.com. KW - WASSA-2018 ER - TY - CPAPER TI - Topic Bias in Emotion Classification AU - Wegge, Maximilian AU - Klinger, Roman PY - 2024 SP - 89-103 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/95822 LA - eng AB - Emotion corpora are typically sampled based on keyword/hashtag search or by asking study participants to generate textual instances. In any case, these corpora are not uniform samples representing the entirety of a domain. We hypothesize that this practice of data acquision leads to unrealistic correlations between overrepresented topics in these corpora that harm the generalizability of models. Such topic bias could lead to wrong predictions for instances like “I organized the service for my aunt’s funeral.” when funeral events are overpresented for instances labeled with sadness, despite the emotion of pride being more appropriate here. In this paper, we study this topic bias both from the data and the modeling perspective. We first label a set of emotion corpora automatically via topic modeling and show that emotions in fact correlate with specific topics. Further, we see that emotion classifiers are confounded by such topics. Finally, we show that the established debiasing method of adversarial correction via gradient reversal mitigates the issue. Our work points out issues with existing emotion corpora and that more representative resources are required for fair evaluation of models predicting affective concepts from text. KW - emotion classification ER - TY - CPAPER TI - Where are We in Event-centric Emotion Analysis? : Bridging Emotion Role Labeling and Appraisal-based Approaches AU - Klinger, Roman PY - 2024 SP - 1-17 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/95825 LA - eng AB - The term emotion analysis in text subsumes various natural language processing tasks which have in common the goal to enable computers to understand emotions. Most popular is emotion classification in which one or multiple emotions are assigned to a predefined textual unit. While such setting is appropriate for identifying the reader’s or author’s emotion, emotion role labeling adds the perspective of mentioned entities and extracts text spans that correspond to the emotion cause. The underlying emotion theories agree on one important point; that an emotion is caused by some internal or external event and comprises several subcomponents, including the subjective feeling and a cognitive evaluation. We therefore argue that emotions and events are related in two ways. (1) Emotions are events; and this perspective is the fundament in natural language processing for emotion role labeling. (2) Emotions are caused by events; a perspective that is made explicit with research how to incorporate psychological appraisal theories in NLP models to interpret events. These two research directions, role labeling and (event-focused) emotion classification, have by and large been tackled separately. In this paper, we contextualize both perspectives and discuss open research questions. KW - emotion analysis ER - TY - CPAPER TI - Affective Natural Language Generation of Event Descriptions through Fine-grained Appraisal Conditions AU - Menchaca Resendiz, Yarik AU - Klinger, Roman PY - 2024 SP - 375-387 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/95926 LA - eng AB - Models for affective text generation have shown a remarkable progress, but they commonly rely only on basic emotion theories or valance/arousal values as conditions. This is appropriate when the goal is to create explicit emotion statements (“The kid is happy.”). Emotions are, however, commonly communicated implicitly. For instance, the emotional interpretation of an event (“Their dog died.”) does often not require an explicit emotion statement. In psychology, appraisal theories explain the link between a cognitive evaluation of an event and the potentially developed emotion. They put the assessment of the situation on the spot, for instance regarding the own control or the responsibility for what happens. We hypothesize and subsequently show that including appraisal variables as conditions in a generation framework comes with two advantages. (1) The generation model is informed in greater detail about what makes a specific emotion and what properties it has. This leads to text generation that better fulfills the condition. (2) The variables of appraisal allow a user to perform a more fine-grained control of the generated text, by stating properties of a situation instead of only providing the emotion category. Our Bart and T5-based experiments with 7 emotions (Anger, Disgust, Fear, Guilt, Joy, Sadness, Shame), and 7 appraisals (Attention, Responsibility, Control, Circumstance, Pleasantness, Effort, Certainty) show that (1) adding appraisals during training improves the accurateness of the generated texts by 10 pp in F1. Further, (2) the texts with appraisal variables are longer and contain more details. This exemplifies the greater control for users. KW - Affective Natural Language Generation ER - TY - CPAPER TI - Automatic Emotion Experiencer Recognition AU - Wegge, Maximilian AU - Klinger, Roman PY - 2024 SP - 1-7 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/95930 LA - eng AB - The most prominent subtask in emotion analysis is emotion classifcation; to assign a category to a textual unit, for instance a social media post. Many research questions from the social sciences do, however, not only require the detection of the emotion of an author of a post but to understand who is ascribed an emotion in text. This task is tackled by emotion role labeling which aims at extracting who is described in text to experience an emotion, why, and towards whom. This could, however, be considered overly sophisticated if the main question to answer is who feels which emotion. A targeted approach for such setup is to classify emotion experiencer mentions (aka “emoters”) regarding the emotion they presumably perceive. This task is similar to named entity recognition of person names with the difference that not every mentioned entity name is an emoter. While, very recently, data with emoter annotations has been made available, no experiments have yet been performed to detect such mentions. With this paper, we provide baseline experiments to understand how challenging the task is. We further evaluate the impact on experiencer-specifc emotion categorization and appraisal detection in a pipeline, when gold mentions are not available. We show that experiencer detection in text is a challenging task, with a precision of .82 and a recall of .56 (F1 =.66). These results motivate future work of jointly modeling emoter spans and emotion/appraisal predictions. KW - Automatic Emotion Experiencer Recognition ER - TY - CPAPER TI - UNIDECOR : A Unified Deception Corpus for Cross-Corpus Deception Detection AU - Velutharambath, Aswathy AU - Klinger, Roman PY - 2024 SP - 39-51 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/95939 LA - eng AB - Verbal deception has been studied in psychology, forensics, and computational linguistics for a variety of reasons, like understanding behaviour patterns, identifying false testimonies, and detecting deception in online communication. Varying motivations across research fields lead to differences in the domain choices to study and in the conceptualization of deception, making it hard to compare models and build robust deception detection systems for a given language. With this paper, we improve this situation by surveying available English deception datasets which include domains like social media reviews, court testimonials, opinion statements on specific topics, and deceptive dialogues from online strategy games. We consolidate these datasets into a single unified corpus. Based on this resource, we conduct a correlation analysis of linguistic cues of deception across datasets to understand the differences and perform cross-corpus modeling experiments which show that a cross-domain generalization is challenging to achieve. The unified deception corpus (UNIDECOR) can be obtained from https://www.ims.uni-stuttgart.de/data/unidecor. KW - UNIDECOR ER - TY - CPAPER TI - Natural Language Inference Prompts for Zero-shot Emotion Classification in Text across Corpora AU - Plaza-del-Arco, Flor Miriam AU - Martín-Valdivia, María-Teresa AU - Klinger, Roman PY - 2024 SP - 6805-6817 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96460 LA - eng AB - Within textual emotion classification, the set of relevant labels depends on the domain and application scenario and might not be known at the time of model development. This conflicts with the classical paradigm of supervised learning in which the labels need to be predefined. A solution to obtain a model with a flexible set of labels is to use the paradigm of zero-shot learning as a natural language inference task, which in addition adds the advantage of not needing any labeled training data. This raises the question how to prompt a natural language inference model for zero-shot learning emotion classification. Options for prompt formulations include the emotion name anger alone or the statement “This text expresses anger”. With this paper, we analyze how sensitive a natural language inference-based zero-shot-learning classifier is to such changes to the prompt under consideration of the corpus: How carefully does the prompt need to be selected? We perform experiments on an established set of emotion datasets presenting different language registers according to different sources (tweets, events, blogs) with three natural language inference models and show that indeed the choice of a particular prompt formulation needs to fit to the corpus. We show that this challenge can be tackled with combinations of multiple prompts. Such ensemble is more robust across corpora than individual prompts and shows nearly the same performance as the individual best prompt for a particular corpus. KW - Natural Language Inference Prompts ER - TY - CPAPER TI - An Analysis of Annotated Corpora for Emotion Classification in Text AU - Bostan, Laura Ana Maria AU - Klinger, Roman PY - 2024 SP - 2104-2119 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96461 LA - eng AB - Several datasets have been annotated and published for classification of emotions. They differ in several ways: (1) the use of different annotation schemata (e. g., discrete label sets, including joy, anger, fear, or sadness or continuous values including valence, or arousal), (2) the domain, and, (3) the file formats. This leads to several research gaps: supervised models often only use a limited set of available resources. Additionally, no previous work has compared emotion corpora in a systematic manner. We aim at contributing to this situation with a survey of the datasets, and aggregate them in a common file format with a common annotation schema. Based on this aggregation, we perform the first cross-corpus classification experiments in the spirit of future research enabled by this paper, in order to gain insight and a better understanding of differences of models inferred from the data. This work also simplifies the choice of the most appropriate resources for developing a model for a novel domain. One result from our analysis is that a subset of corpora is better classified with models trained on a different corpus. For none of the corpora, training on all data altogether is better than using a subselection of the resources. Our unified corpus is available at http://www.ims.uni-stuttgart.de/data/unifyemotion. KW - Emotion Classification ER - TY - CPAPER TI - Emotion Analysis from Texts AU - Štajner, Sanja AU - Klinger, Roman PY - 2024 SP - 7-12 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96132 LA - eng AB - Emotion analysis in text is an area of research that encompasses a set of various natural language processing (NLP) tasks, including classification and regression settings, as well as structured prediction tasks like role labelling or stimulus detection. In this tutorial, we provide an overview of research from emotion psychology which sets the ground for choosing adequate NLP methodology, and present existing resources and classification methods used for emotion analysis in texts. We further discuss appraisal theories and how events can be interpreted regarding their presumably caused emotion and briefly introduce emotion role labelling. In addition to these technical topics, we discuss the use cases of emotion analysis in text, their societal impact, ethical considerations, as well as the main challenges in the field. KW - emotion analysis ER - TY - CPAPER TI - Orthonormal Explicit Topic Analysis for Cross-Lingual Document Matching AU - McCrae, John Philip AU - Cimiano, Philipp AU - Klinger, Roman PY - 2024 SP - 1732-1740 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96521 LA - eng AB - Cross-lingual topic modelling has applications in machine translation, word sense disambiguation and terminology alignment. Multilingual extensions of approaches based on latent (LSI), generative (LDA, PLSI) as well as explicit (ESA) topic modelling can induce an interlingual topic space allowing documents in different languages to be mapped into the same space and thus to be compared across languages. In this paper, we present a novel approach that combines latent and explicit topic modelling approaches in the sense that it builds on a set of explicitly defined topics, but then computes latent relations between these. Thus, the method combines the benefits of both explicit and latent topic modelling approaches. We show that on a cross-lingual mate retrieval task, our model significantly outperforms LDA, LSI, and ESA, as well as a baseline that translates every word in a document into the target language. KW - Cross-Lingual Document Matching ER - TY - JOUR TI - Dimensional Modeling of Emotions in Text with Appraisal Theories : Corpus Creation, Annotation Reliability, and Prediction AU - Troiano, Enrica AU - Oberländer, Laura AU - Klinger, Roman PY - 2024 SP - 1-72 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96522 LA - eng AB - The most prominent tasks in emotion analysis are to assign emotions to texts and to understand how emotions manifest in language. An important observation for natural language processing is that emotions can be communicated implicitly by referring to events alone, appealing to an empathetic, intersubjective understanding of events, even without explicitly mentioning an emotion name. In psychology, the class of emotion theories known as appraisal theories aims at explaining the link between events and emotions. Appraisals can be formalized as variables that measure a cognitive evaluation by people living through an event that they consider relevant. They include the assessment if an event is novel, if the person considers themselves to be responsible, if it is in line with their own goals, and so forth. Such appraisals explain which emotions are developed based on an event, for example, that a novel situation can induce surprise or one with uncertain consequences could evoke fear. We analyze the suitability of appraisal theories for emotion analysis in text with the goal of understanding if appraisal concepts can reliably be reconstructed by annotators, if they can be predicted by text classifiers, and if appraisal concepts help to identify emotion categories. To achieve that, we compile a corpus by asking people to textually describe events that triggered particular emotions and to disclose their appraisals. Then, we ask readers to reconstruct emotions and appraisals from the text. This set-up allows us to measure if emotions and appraisals can be recovered purely from text and provides a human baseline to judge a model’s performance measures. Our comparison of text classification methods to human annotators shows that both can reliably detect emotions and appraisals with similar performance. Therefore, appraisals constitute an alternative computational emotion analysis paradigm and further improve the categorization of emotions in text with joint models. KW - Dimensional Modeling KW - Emotion ER - TY - JOUR TI - Detecting miRNA Mentions and Relations in Biomedical Literature [version 3; peer review: 2 approved, 1 approved with reservations] AU - Bagewadi, Shweta AU - Bobić, Tamara AU - Hofmann-Apitius, Martin AU - Fluck, Juliane AU - Klinger, Roman PY - 2024 SP - 1-36 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96523 LA - eng AB - Introduction: MicroRNAs (miRNAs) have demonstrated their potential as post-transcriptional gene expression regulators, participating in a wide spectrum of regulatory events such as apoptosis, differentiation, and stress response. Apart from the role of miRNAs in normal physiology, their dysregulation is implicated in a vast array of diseases. Dissection of miRNA-related associations are valuable for contemplating their mechanism in diseases, leading to the discovery of novel miRNAs for disease prognosis, diagnosis, and therapy. Motivation: Apart from databases and prediction tools, miRNA-related information is largely available as unstructured text. Manual retrieval of these associations can be labor-intensive due to steadily growing number of publications. Additionally, most of the published miRNA entity recognition methods are keyword based, further subjected to manual inspection for retrieval of relations. Despite the fact that several databases host miRNA-associations derived from text, lower sensitivity and lack of published details for miRNA entity recognition and associated relations identification has motivated the need for developing comprehensive methods that are freely available for the scientific community. Additionally, the lack of a standard corpus for miRNA-relations has caused difficulty in evaluating the available systems. We propose methods to automatically extract mentions of miRNAs, species, genes/proteins, disease, and relations from scientific literature. Our generated corpora, along with dictionaries, and miRNA regular expression are freely available for academic purposes. To our knowledge, these resources are the most comprehensive developed so far. Results: The identification of specific miRNA mentions reaches a recall of 0.94 and precision of 0.93. Extraction of miRNA-disease and miRNA-gene relations lead to an F1 score of up to 0.76. A comparison of the information extracted by our approach to the databases miR2Disease and miRSel for the extraction of Alzheimer's disease related relations shows the capability of our proposed methods in identifying correct relations with improved sensitivity. The published resources and described methods can help the researchers for maximal retrieval of miRNA-relations and generation of miRNA-regulatory networks. Availability: The training and test corpora, annotation guidelines, developed dictionaries, and supplementary files are available at http://www.scai.fraunhofer.de/mirna-corpora.html KW - MicroRNAs KW - prediction algorithms KW - corpus ER - TY - JOUR TI - A Survey on Sentiment and Emotion Analysis for Computational Literary Studies AU - Kim, Evgeny AU - Klinger, Roman PY - 2024 SP - 1-23 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96526 LA - eng AB - Emotionen sind ein wesentlicher Bestandteil fesselnder Erzählungen: Literatur erzählt uns von Menschen mit Zielen, Wünschen, Leidenschaften und Absichten. Die Analyse von Emotionen ist Teil des breiteren und größeren Feldes der Sentimentanalyse und findet in der Literaturwissenschaft zunehmend Beachtung. In der Vergangenheit wurde die affektive Dimension der Literatur hauptsächlich im Rahmen der literarischen Hermeneutik untersucht. Mit dem Aufkommen der Digital Humanities (DH) als Forschungsfeld, haben jedoch einige Studien über Emotionen im literarischen Kontext eine computergestützte Wendung genommen. In Anbetracht der Tatsache, dass sich die DH als Feld noch im Aufbau befindet, kann diese Forschungsrichtung als relativ neu bezeichnet werden. In dieser Übersicht bieten wir einen Überblick über die bestehende Forschung zur Emotionsanalyse in der Literatur. Die untersuchte Forschungsliteratur befasst sich mit einer Vielzahl von Themen, darunter die Veränderungen der emotionalen Konnotation im Verlauf eines Texts, die Netzwerkanalyse eines literarischen Textes und das Verstehen der Emotionalität von Texten, neben anderen Themen. Basierend auf diesem Überblick weisen wir auf eine Reihe von verbleibenden Herausforderungen hin, die vielversprechende zukünftige Forschungsrichtungen darstellen. KW - Emotion Analysis ER - TY - JOUR TI - From theories on styles to their transfer in text : Bridging the gap with a hierarchical survey AU - Troiano, Enrica AU - Velutharambath, Aswathy AU - Klinger, Roman PY - 2024 SP - 849-908 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96527 LA - eng AB - Humans are naturally endowed with the ability to write in a particular style. They can, for instance, rephrase a formal letter in an informal way, convey a literal message with the use of figures of speech or edit a novel by mimicking the style of some well-known authors. Automating this form of creativity constitutes the goal of style transfer. As a natural language generation task, style transfer aims at rewriting existing texts, and specifically, it creates paraphrases that exhibit some desired stylistic attributes. From a practical perspective, it envisions beneficial applications, like chatbots that modulate their communicative style to appear empathetic, or systems that automatically simplify technical articles for a non-expert audience. Several style-aware paraphrasing methods have attempted to tackle style transfer. A handful of surveys give a methodological overview of the field, but they do not support researchers to focus on specific styles. With this paper, we aim at providing a comprehensive discussion of the styles that have received attention in the transfer task. We organize them in a hierarchy, highlighting the challenges for the definition of each of them and pointing out gaps in the current research landscape. The hierarchy comprises two main groups. One encompasses styles that people modulate arbitrarily, along the lines of registers and genres. The other group corresponds to unintentionally expressed styles, due to an author’s personal characteristics. Hence, our review shows how these groups relate to one another and where specific styles, including some that have not yet been explored, belong in the hierarchy. Moreover, we summarize the methods employed for different stylistic families, hinting researchers towards those that would be the most fitting for future research. KW - Natural Language Generation KW - Translation Technology KW - Style Transfer KW - Paraphrasing ER - TY - CPAPER TI - Named Entity Recognition with Combinations of Conditional Random Fields AU - Klinger, Roman AU - Hofmann-Apitius, Martin AU - Fluck, Juliane AU - Friedrich, Christoph M. PY - 2024 SP - 89-91 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96533 LA - eng AB - The Gene Mention task is a Named Entity Recognition (NER) task for labeling gene and gene product names in biomedical text. To deal with acceptable alternatives additionally to the gold standard, we use combinations of Conditional Random Fields (CRF) together with a normalizing tagger. This process isfollowed by a postprocessing step including an acronym disambiguation based on Latent Semantic Analysis(LSA). For robust model selection we apply 50-fold Bootstrapping to obtain an average F-Score of 84.58 % on the trainingset and 86.33 % on the test set. KW - named entity recognition KW - text mining KW - data mining KW - conditional random fields KW - multi model approach ER - TY - CPAPER TI - User's Choice of Precision and Recall in Named Entity Recognition AU - Klinger, Roman AU - Friedrich, Christoph M. PY - 2024 SP - 192-196 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96826 LA - eng AB - Conditional Random Fields are commonly trained to maximize likelihood. The corresponding Fβ measure, the weighted harmonic mean of preci-sion and recall, which is established for evaluation in information retrieval and text mining, is not necessarily the optimal result for the user’s choice of β. Some approaches have been published to optimize multivariate measures like Fβ to overcome this inconsistency. The limitation is that constraints like the value of β have to be known at training time. This publication proposes a method of multiobjective optimization of both precision and recall based on a preceding likelihood training. The output is an estimation of pareto-optimal solutions from which the user can select the best for the actual application. Evaluated on two publicly available data sets in the field of named entity recognition, nearly all Fβ values are superior to those resulting from log-likelihood training KW - Named Entity Recognition KW - Conditional Random Fields KW - Multi-Objective Optimization KW - NSGA-II KW - Fβ measure KW - Recall KW - Precision ER - TY - CPAPER TI - Bi-directional Inter-dependencies of Subjective Expressions and Targets and their Value for a Joint Model AU - Klinger, Roman AU - Cimiano, Philipp PY - 2024 SP - 848-854 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96827 LA - eng AB - Opinion mining is often regarded as a classification or segmentation task, involving the prediction of i) subjective expressions, ii) their target and iii) their polarity. Intuitively, these three variables are bidirectionally interdependent, but most work has either attempted to predict them in isolation or proposing pipeline-based approaches that cannot model the bidirectional interaction between these variables. Towards better understanding the interaction between these variables, we propose a model that allows for analyzing the relation of target and subjective phrases in both directions, thus providing an upper bound for the impact of a joint model in comparison to a pipeline model. We report results on two public datasets (cameras and cars), showing that our model outperforms state-of-the-art models, as well as on a new dataset consisting of Twitter posts KW - Joint Model ER - TY - CPAPER TI - The USAGE review corpus for fine grained multi lingual opinion analysis AU - Klinger, Roman AU - Cimiano, Philipp PY - 2024 SP - 2211-2218 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96828 LA - eng AB - Opinion mining has received wide attention in recent years. Models for this task are typically trained or evaluated with a manually annotated dataset. However, fine-grained annotation of sentiments including information about aspects and their evaluation is very labour-intensive. The data available so far is limited. Contributing to this situation, this paper describes the Bielefeld University Sentiment Analysis Corpus for German and English (USAGE), which we offer freely to the community and which contains the annotation of product reviews from Amazon with both aspects and subjective phrases. It provides information on segments in the text which denote an aspect or a subjective evaluative phrase which refers to the aspect. Relations and coreferences are explicitly annotated. This dataset contains 622 English and 611 German reviews, allowing to investigate how to port sentiment analysis systems across languages and domains. We describe the methodology how the corpus was created and provide statistics including inter-annotator agreement. We further provide figures for a baseline system and results for German and English as well as in a cross-domain setting. The results are encouraging in that they show that aspects and phrases can be extracted robustly without the need of tuning to a particular type of products. KW - sentiment analysis KW - corpus KW - product reviews ER - TY - CPAPER TI - Towards Opinion Mining from Reviews for the Prediction of Product Rankings AU - Kessler, Wiltrud AU - Klinger, Roman AU - Kuhn, Jonas PY - 2024 SP - 51-57 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96829 LA - eng AB - Opinion mining aims at summarizing the content of reviews for a specific brand, product, or manufacturer. However, the actual desire of a user is often one step further: Produce a ranking corresponding to specific needs such that a selection process is supported. In this work, we aim towards closing this gap. We present the task to rank products based on sentiment information and discuss necessary steps towards addressing this task. This includes, on the one hand, the identification of gold rankings as a fundament for an objective function and evaluation and, on the other hand, methods to rank products based on review information. To demonstrate early results on that task, we employ real world examples of rankings as gold standard that are of interest to potential customers as well as product managers, in our case the sales ranking provided by Amazon.com and the quality ranking by Snapsort.com. As baseline methods, we use the average star ratings and review frequencies. Our best text-based approximation of the sales ranking achieves a Spearman’s correlation coefficient of ρ = 0.23. On the Snapsort data, a ranking based on extracting comparisons leads to ρ = 0.51. In addition, we show that aspect-specific rankings can be used to measure the impact of specific aspects on the ranking. KW - Opinion Mining ER - TY - CPAPER TI - Towards Gene Recognition from Rare and Ambiguous Abbreviations using a Filtering Approach AU - Hartung, Matthias AU - Klinger, Roman AU - Zwick, Matthias AU - Cimiano, Philipp PY - 2024 SP - 118-127 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96830 LA - eng AB - Retrieving information about highly ambiguous gene/protein homonyms is a challenge, in particular where their non-protein meanings are more frequent than their protein meaning (e. g., SAH or HF). Due to their limited coverage in common benchmarking data sets, the performance of existing gene/protein recognition tools on these problematic cases is hard to assess. We uniformly sample a corpus of eight ambiguous gene/protein abbreviations from MEDLINEr and provide manual annotations for each mention of these abbreviations.1 Based on this resource, we show that available gene recognition tools such as conditional random fields (CRF) trained on BioCreative 2 NER data or GNAT tend to underperform on this phenomenon. We propose to extend existing gene recognition approaches by combining a CRF and a support vector machine. In a cross- entity evaluation and without taking any entity-specific information into account, our model achieves a gain of 6 points F1-Measure over our best baseline which checks for the occurrence of a long form of the abbreviation and more than 9 points over all existing tools investigated. KW - Gene Recognition ER - TY - CPAPER TI - An Impact Analysis of Features in a Classification Approach to Irony Detection in Product Reviews AU - Buschmeier, Konstantin AU - Cimiano, Philipp AU - Klinger, Roman PY - 2024 SP - 42-49 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96831 LA - eng AB - Irony is an important device in human communication, both in everyday spoken conversations as well as in written texts including books, websites, chats, reviews, and Twitter messages among others. Specific cases of irony and sarcasm have been studied in different contexts but, to the best of our knowledge, only recently the first publicly available corpus including annotations about whether a text is ironic or not has been published by Filatova (2012). How- ever, no baseline for classification of ironic or sarcastic reviews has been provided. With this paper, we aim at closing this gap. We formulate the problem as a supervised classification task and evaluate different classifiers, reaching an F1-measure of up to 74 % using logistic regression. We analyze the impact of a number of features which have been proposed in previous research as well as combinations of them. KW - Irony Detection ER - TY - CPAPER TI - Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference AU - Kadiķis, Emīls AU - Srivastav, Vaibhav AU - Klinger, Roman PY - 2024 SP - 6031-6037 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96472 LA - eng AB - The task of natural language inference (NLI), to decide if a hypothesis entails or contradicts a premise, received considerable attention in recent years. All competitive systems build on top of contextualized representations and make use of transformer architectures for learning an NLI model. When somebody is faced with a particular NLI task, they need to select the best model that is available. This is a time-consuming and resource-intense endeavour. To solve this practical problem, we propose a simple method for predicting the performance without actually fine-tuning the model. We do this by testing how well the pre-trained models perform on the aNLI task when just comparing sentence embeddings with cosine similarity to what kind of performance is achieved when training a classifier on top of these embeddings. We show that the accuracy of the cosine similarity approach correlates strongly with the accuracy of the classification approach with a Pearson correlation coefficient of 0.65. Since the similarity is orders of magnitude faster to compute on a given dataset (less than a minute vs. hours), our method can lead to significant time savings in the process of model selection. KW - Abductive Natural Language Inference ER - TY - CPAPER TI - SCARE : the Sentiment Corpus of App Reviews with Fine-grained Annotations in German AU - Sänger, Mario AU - Kemmerer, Steffen AU - Adolphs, Peter AU - Klinger, Roman AU - Leser, Ulf PY - 2024 SP - 1114–1121 PB - Otto-Friedrich-Universität CY - Bamberg SN - 978-2-9517408-9-1 UR - https://fis.uni-bamberg.de/handle/uniba/96474 LA - eng AB - The automatic analysis of texts containing opinions of users about, e.g., products or political views has gained attention within the last decades. However, previous work on the task of analyzing user reviews about mobile applications in app stores is limited. Publicly available corpora do not exist, such that a comparison of different methods and models is difficult. We fill this gap by contributing the Sentiment Corpus of App Reviews (SCARE), which contains fine-grained annotations of application aspects, subjective (evaluative) phrases and relations between both. This corpus consists of 1,760 annotated application reviews from the Google Play Store with 2,487 aspects and 3,959 subjective phrases. We describe the process and methodology how the corpus was created. The Fleiss Kappa between four annotators reveals an agreement of 0.72. We provide a strong baseline with a linear-chain conditional random field and word-embedding features with a performance of 0.62 for aspect detection and 0.63 for the extraction of subjective phrases. The corpus is available to the research community to support the development of sentiment analysis methods on mobile application reviews. KW - Sentiment Corpus of App Reviews ER - TY - CPAPER TI - Items from Psychometric Tests as Training Data for Personality Profiling Models of Twitter Users AU - Kreuter, Anne AU - Sassenberg, Kai AU - Klinger, Roman PY - 2024 SP - 315-323 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96473 LA - eng AB - Machine-learned models for author profiling in social media often rely on data acquired via self-reporting-based psychometric tests (questionnaires) filled out by social media users. This is an expensive but accurate data collection strategy. Another, less costly alternative, which leads to potentially more noisy and biased data, is to rely on labels inferred from publicly available information in the profiles of the users, for instance self-reported diagnoses or test results. In this paper, we explore a third strategy, namely to directly use a corpus of items from validated psychometric tests as training data. Items from psychometric tests often consist of sentences from an I-perspective (e.g., ‘I make friends easily.’). Such corpora of test items constitute ‘small data’, but their availability for many concepts is a rich resource. We investigate this approach for personality profiling, and evaluate BERT classifiers fine-tuned on such psychometric test items for the big five personality traits (openness, conscientiousness, extraversion, agreeableness, neuroticism) and analyze various augmentation strategies regarding their potential to address the challenges coming with such a small corpus. Our evaluation on a publicly available Twitter corpus shows a comparable performance to in-domain training for 4/5 personality traits with T5-based data augmentation. KW - Personality Profiling Models ER - TY - CPAPER TI - An Entity-based Claim Extraction Pipeline for Real-world Biomedical Fact-checking AU - Wührl, Amelie AU - Grimminger, Lara AU - Klinger, Roman PY - 2024 SP - 29-37 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96455 LA - eng AB - Existing fact-checking models for biomedical claims are typically trained on synthetic or well-worded data and hardly transfer to social media content. This mismatch can be mitigated by adapting the social media input to mimic the focused nature of common training claims. To do so, Wührl and Klinger (2022a) propose to extract concise claims based on medical entities in the text. However, their study has two limitations: First, it relies on gold-annotated entities. Therefore, its feasibility for a real-world application cannot be assessed since this requires detecting relevant entities automatically. Second, they represent claim entities with the original tokens. This constitutes a terminology mismatch which potentially limits the fact-checking performance. To understand both challenges, we propose a claim extraction pipeline for medical tweets that incorporates named entity recognition and terminology normalization via entity linking. We show that automatic NER does lead to a performance drop in comparison to using gold annotations but the fact-checking performance still improves considerably over inputting the unchanged tweets. Normalizing entities to their canonical forms does, however, not improve the performance. KW - entity-based Claim Extraction Pipeline ER - TY - CPAPER TI - Experiencer-Specific Emotion and Appraisal Prediction AU - Wegge, Maximilian AU - Troiano, Enrica AU - Oberlaender, Laura Ana Maria AU - Klinger, Roman PY - 2024 SP - 25-32 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96456 LA - eng AB - Emotion classification in NLP assigns emotions to texts, such as sentences or paragraphs. With texts like “I felt guilty when he cried”, focusing on the sentence level disregards the standpoint of each participant in the situation: the writer (“I”) and the other entity (“he”) could in fact have different affective states. The emotions of different entities have been considered only partially in emotion semantic role labeling, a task that relates semantic roles to emotion cue words. Proposing a related task, we narrow the focus on the experiencers of events, and assign an emotion (if any holds) to each of them. To this end, we represent each emotion both categorically and with appraisal variables, as a psychological access to explaining why a person develops a particular emotion. On an event description corpus, our experiencer-aware models of emotions and appraisals outperform the experiencer-agnostic baselines, showing that disregarding event participants is an oversimplification for the emotion detection task. KW - Emotion ER - TY - CPAPER TI - Bilingual Sentiment Embeddings : Joint Projection of Sentiment Across Languages AU - Barnes, Jeremy AU - Klinger, Roman AU - Schulte im Walde, Sabine PY - 2024 SP - 2483-2493 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96457 LA - eng AB - Sentiment analysis in low-resource languages suffers from a lack of annotated corpora to estimate high-performing models. Machine translation and bilingual word embeddings provide some relief through cross-lingual sentiment approaches. However, they either require large amounts of parallel data or do not sufficiently capture sentiment information. We introduce Bilingual Sentiment Embeddings (BLSE), which jointly represent sentiment information in a source and target language. This model only requires a small bilingual lexicon, a source-language corpus annotated for sentiment, and monolingual word embeddings for each language. We perform experiments on three language combinations (Spanish, Catalan, Basque) for sentence-level cross-lingual sentiment classification and find that our model significantly outperforms state-of-the-art methods on four out of six experimental setups, as well as capturing complementary information to machine translation. Our analysis of the resulting embedding space provides evidence that it represents sentiment information in the resource-poor target language without any annotated data in that language. KW - Bilingual Sentiment Embeddings ER - TY - CPAPER TI - Automatically Selected Skip Edges in Conditional Random Fields for Named Entity Recognition AU - Klinger, Roman PY - 2024 SP - 580-585 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96382 LA - eng AB - Incorporating distant information via manually selected skip chain templates has been shown to be beneficial for the performance of conditional random field models in contrast to a simple linear chain based structure (Sutton and McCallum, 2007; Galley, 2006; Liu et al., 2010). The set of properties to be captured by a template is typically manually chosen with respect to the application domain. In this paper, a search strategy to find meaningful skip chains independent from the application domain is proposed. From a huge set of potentially beneficial templates, some can be shown to have a positive im- pact on the performance. The search for a meaningful graphical structure demonstrates the usefulness of the approach with an increase of nearly 2 % F1 measure on a publicly available data set (Klinger et al., 2008). KW - Automatically Selected Skip Edges ER - TY - JOUR TI - On the Relationship between Frames and Emotionality in Text AU - Troiano, Enrica AU - Klinger, Roman AU - Padó, Sebastian PY - 2024 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96383 LA - eng AB - Emotions, which are responses to salient events, can be realized in text implicitly, for instance with mere references to facts (e.g., “That was the beginning of a long war”). Interpreting affective meanings thus relies on the readers’ background knowledge, but that is hardly modeled in computational emotion analysis. Much work in the field is focused on the word level and treats individual lexical units as the fundamental emotion cues in written communication. We shift our attention to word relations. We leverage Frame Semantics, a prominent theory for the description of predicate-argument structures, which matches the study of emotions: frames build on a “semantics of understanding” whose assumptions rely precisely on people’s world knowledge. Our overarching question is whether and to what extent the events that are represented by frames possess an emotion meaning. To carry out a large corpus-based correspondence analysis, we automatically annotate texts with emotions as well as with FrameNet frames and roles, and we analyze the correlations between them. Our main finding is that substantial groups of frames have an emotional import. With an extensive qualitative analysis, we show that they capture several properties of emotions that are purported by theories from psychology. These observations boost insights on the two strands of research that we bring together: emotion analysis can profit from the event-based perspective of frame semantics; in return, frame semantics gains a better grip of its position vis-à-vis emotions, an integral part of word meanings. KW - Emotionality ER - TY - CPAPER TI - “splink” is happy and “phrouth” is scary : Emotion Intensity Analysis for Nonsense Words AU - Sabbatino, Valentino AU - Troiano, Enrica AU - Schweitzer, Antje AU - Klinger, Roman PY - 2024 SP - 37-50 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96471 LA - eng AB - People associate affective meanings to words - “death” is scary and sad while “party” is connotated with surprise and joy. This raises the question if the association is purely a product of the learned affective imports inherent to semantic meanings, or is also an effect of other features of words, e.g., morphological and phonological patterns. We approach this question with an annotation-based analysis leveraging nonsense words. Specifically, we conduct a best-worst scaling crowdsourcing study in which participants assign intensity scores for joy, sadness, anger, disgust, fear, and surprise to 272 non-sense words and, for comparison of the results to previous work, to 68 real words. Based on this resource, we develop character-level and phonology-based intensity regressors. We evaluate them on both nonsense words and real words (making use of the NRC emotion intensity lexicon of 7493 words), across six emotion categories. The analysis of our data reveals that some phonetic patterns show clear differences between emotion intensities. For instance, s as a first phoneme contributes to joy, sh to surprise, p as last phoneme more to disgust than to anger and fear. In the modelling experiments, a regressor trained on real words from the NRC emotion intensity lexicon shows a higher performance (r = 0.17) than regressors that aim at learning the emotion connotation purely from nonsense words. We conclude that humans do associate affective meaning to words based on surface patterns, but also based on similarities to existing words (“juy” to “joy”, or “flike” to “like”). KW - Emotion Intensity Analysis ER - TY - CPAPER TI - Entity-based Claim Representation Improves Fact-Checking of Medical Content in Tweets AU - Wührl, Amelie AU - Klinger, Roman PY - 2024 SP - 187-198 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96458 LA - eng AB - False medical information on social media poses harm to people’s health. While the need for biomedical fact-checking has been recognized in recent years, user-generated medical content has received comparably little attention. At the same time, models for other text genres might not be reusable, because the claims they have been trained with are substantially different. For instance, claims in the SciFact dataset are short and focused: “Side effects associated with antidepressants increases risk of stroke”. In contrast, social media holds naturally-occurring claims, often embedded in additional context: "‘If you take antidepressants like SSRIs, you could be at risk of a condition called serotonin syndrome’ Serotonin syndrome nearly killed me in 2010. Had symptoms of stroke and seizure.” This showcases the mismatch between real-world medical claims and the input that existing fact-checking systems expect. To make user-generated content checkable by existing models, we propose to reformulate the social-media input in such a way that the resulting claim mimics the claim characteristics in established datasets. To accomplish this, our method condenses the claim with the help of relational entity information and either compiles the claim out of an entity-relation-entity triple or extracts the shortest phrase that contains these elements. We show that the reformulated input improves the performance of various fact-checking models as opposed to checking the tweet text in its entirety. KW - Entity-based Claim Representation ER - TY - CPAPER TI - Towards Multimodal Emotion Recognition in German Speech Events in Cars using Transfer Learning AU - Cevher, Deniz AU - Zepf, Sebastian AU - Klinger, Roman PY - 2024 SP - 79-90 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96459 LA - eng AB - The recognition of emotions by humans is a complex process which considers multiple interacting signals such as facial expressions and both prosody and semantic content of utterances. Commonly, research on automatic recognition of emotions is, with few exceptions, limited to one modality. We describe an in-car experiment for emotion recognition from speech interactions for three modalities: the audio signal of a spoken interaction, the visual signal of the driver’s face, and the manually transcribed content of utterances of the driver. We use off-the-shelf tools for emotion detection in audio and face and compare that to a neural transfer learning approach for emotion recognition from text which utilizes existing resources from other domains. We see that transfer learning enables models based on out-of-domain corpora to per- form well. This method contributes up to 10 percentage points in F1, with up to 76 micro-average F1 across the emotions joy, annoyance and insecurity. Our findings also indicate that off-the-shelf-tools analyzing face and audio are not ready yet for emotion detection in in-car speech interactions without further adjustments. KW - Multimodal Emotion Recognition ER - TY - CPAPER TI - Recovering Patient Journeys : A Corpus of Biomedical Entities and Relations on Twitter (BEAR) AU - Wührl, Amelie AU - Klinger, Roman PY - 2024 SP - 4439-4450 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96463 LA - eng AB - Text mining and information extraction for the medical domain has focused on scientific text generated by researchers. However, their access to individual patient experiences or patient-doctor interactions is limited. On social media, doctors, patients and their relatives also discuss medical information. Individual information provided by laypeople complements the knowledge available in scientific text. It reflects the patient’s journey making the value of this type of data twofold: It offers direct access to people’s perspectives, and it might cover information that is not available elsewhere, including self-treatment or self-diagnose. Named entity recognition and relation extraction are methods to structure information that is available in unstructured text. However, existing medical social media corpora focused on a comparably small set of entities and relations. In contrast, we provide rich annotation layers to model patients’ experiences in detail. The corpus consists of medical tweets annotated with a fine-grained set of medical entities and relations between them, namely 14 entity (incl. environmental factors, diagnostics, biochemical processes, patients’ quality-of-life descriptions, pathogens, medical conditions, and treatments) and 20 relation classes (incl. prevents, influences, interactions, causes). The dataset consists of 2,100 tweets with approx. 6,000 entities and 2,200 relations. KW - Patient Journeys ER - TY - CPAPER TI - Instance Selection Improves Cross-Lingual Model Training for Fine-Grained Sentiment Analysis AU - Klinger, Roman AU - Cimiano, Philipp PY - 2024 SP - 153-163 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96621 LA - eng AB - Scarcity of annotated corpora for many languages is a bottleneck for training finegrained sentiment analysis models that can tag aspects and subjective phrases. We propose to exploit statistical machine translation to alleviate the need for training data by projecting annotated data in a source language to a target language such that a supervised fine-grained sentiment analysis system can be trained. To avoid a negative influence of poor-quality translations, we propose a filtering approach based on machine translation quality estimation measures to select only high-quality sentence pairs for projection. We evaluate on the language pair German/English on a corpus of product reviews annotated for both languages and compare to in-target-language training. Projection without any filtering leads to 23 % F1 in the task of detecting aspect phrases, compared to 41 % F1 for in-target-language training. Our approach obtains up to 47 % F1. Further, we show that the detection of subjective phrases is competitive to in-target-language training without filtering. KW - Instance Selection ER - TY - CPAPER TI - Hate Towards the Political Opponent : A Twitter Corpus Study of the 2020 US Elections on the Basis of Offensive Speech and Stance Detection AU - Grimminger, Lara AU - Klinger, Roman PY - 2024 SP - 171-180 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96622 LA - eng AB - The 2020 US Elections have been, more than ever before, characterized by social media campaigns and mutual accusations. We investigate in this paper if this manifests also in online communication of the supporters of the candidates Biden and Trump, by uttering hateful and offensive communication. We formulate an annotation task, in which we join the tasks of hateful/offensive speech detection and stance detection, and annotate 3000 Tweets from the campaign period, if they express a particular stance towards a candidate. Next to the established classes of favorable and against, we add mixed and neutral stances and also annotate if a candidate is mentioned with- out an opinion expression. Further, we an- notate if the tweet is written in an offensive style. This enables us to analyze if supporters of Joe Biden and the Democratic Party communicate differently than supporters of Donald Trump and the Republican Party. A BERT baseline classifier shows that the detection if somebody is a supporter of a candidate can be performed with high quality (.89 F1 for Trump and .91 F1 for Biden), while the detection that somebody expresses to be against a candidate is more challenging (.79 F1 and .64 F1, respectively). The automatic detection of hate/offensive speech remains challenging (with .53 F1). Our corpus is publicly available and constitutes a novel resource for computational modelling of offensive language under consideration of stances. KW - Twitter Corpus Study ER - TY - CPAPER TI - x-enVENT : A Corpus of Event Descriptions with Experiencer-specific Emotion and Appraisal Annotations AU - Troiano, Enrica AU - Oberlaender, Laura Ana Maria AU - Wegge, Maximilian AU - Klinger, Roman PY - 2024 SP - 1365-1375 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96469 LA - eng AB - Emotion classification is often formulated as the task to categorize texts into a predefined set of emotion classes. So far, this task has been the recognition of the emotion of writers and readers, as well as that of entities mentioned in the text. We argue that a classification setup for emotion analysis should be performed in an integrated manner, including the different semantic roles that participate in an emotion episode. Based on appraisal theories in psychology, which treat emotions as reactions to events, we compile an English corpus of written event descriptions. The descriptions depict emotion-eliciting circumstances, and they contain mentions of people who responded emotionally. We annotate all experiencers, including the original author, with the emotions they likely felt. In addition, we link them to the event they found salient (which can be different for different experiencers in a text) by annotating event properties, or appraisals (e.g., the perceived event undesirability, the uncertainty of its outcome). Our analysis reveals patterns in the co-occurrence of people’s emotions in interaction. Hence, this richly-annotated resource provides useful data to study emotions and event evaluations from the perspective of different roles, and it enables the development of experiencer-specific emotion and appraisal classification systems. KW - emotion analysis KW - corpus KW - affective computing KW - role labeling KW - emotion experiencer KW - appraisal theories KW - events ER - TY - JOUR TI - Prevention or Promotion? : Predicting Author's Regulatory Focus AU - Velutharambath, Aswathy AU - Sassenberg, Kai AU - Klinger, Roman PY - 2024 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/95826 LA - eng AB - People differ fundamentally in what motivates them to pursue a goal and how they approach it. For instance, some people seek growth and show eagerness, whereas others prefer security and are vigilant. The concept of regulatory focus is employed in psychology, to explain and predict this goal-directed behavior of humans underpinned by two unique motivational systems – the promotion and the prevention system. Traditionally, text analysis methods using closed-vocabularies are employed to assess the distinctive linguistic patterns associated with the two systems. From an NLP perspective, automatically detecting the regulatory focus of individuals from text provides valuable insights into the behavioral inclinations of the author, finding its applications in areas like marketing or health communication. However, the concept never made an impactful debut in computational linguistics research. To bridge this gap we introduce the novel task of regulatory focus classification from text and present two complementary German datasets – (1) experimentally generated event descriptions and (2) manually annotated short social media texts used for evaluating the generalizability of models on real-world data. First, we conduct a correlation analysis to verify if the linguistic footprints of regulatory focus reported in psychology studies are observable and to what extent in our datasets. For automatic classification, we compare closed-vocabulary-based analyses with a state-of-the-art BERT-based text classification model and observe that the latter outperforms lexicon-based approaches on experimental data and is notably better on out-of-domain Twitter data. KW - Author's Regulatory Focus ER - TY - CPAPER TI - Emotion-Conditioned Text Generation through Automatic Prompt Optimization AU - Menchaca Resendiz, Yarik AU - Klinger, Roman PY - 2024 SP - 24-30 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/95827 LA - eng AB - Conditional natural language generation methods often require either expensive fine-tuning or training a large language model from scratch. Both are unlikely to lead to good results without a substantial amount of data and computational resources. Prompt learning without changing the parameters of a large language model presents a promising alternative. It is a cost-effective approach, while still achieving competitive results. While this procedure is now established for zero- and few-shot text classification and structured prediction, it has received limited attention in conditional text generation. We present the first automatic prompt optimization approach for emotion-conditioned text generation with instruction-fine-tuned models. Our method uses an iterative optimization procedure that changes the prompt by adding, removing, or replacing tokens. As objective function, we only require a text classifier that measures the realization of the conditional variable in the generated text. We evaluate the method on emotion-conditioned text generation with a focus on event reports and compare it to manually designed prompts that also act as the seed for the optimization procedure. The optimized prompts achieve 0.75 macro-average F1 to fulfill the emotion condition in contrast to manually designed seed prompts with only 0.22 macro-average F1. KW - Emotion-Conditioned Text Generation ER - TY - CPAPER TI - EmoProgress : Cumulated Emotion Progression Analysis in Dreams and Customer Service Dialogues AU - Wemmer, Eileen AU - Labat, Sofie AU - Klinger, Roman PY - 2024 T2 - Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) SP - 5660–5677 PB - ELRA and ICCL UR - https://fis.uni-bamberg.de/handle/uniba/95874 LA - eng AB - Emotion analysis often involves the categorization of isolated textual units, but these are parts of longer discourses, like dialogues or stories. This leads to two different established emotion classification setups: (1) Classification of a longer text into one or multiple emotion categories. (2) Classification of the parts of a longer text (sentences or utterances), either (2a) with or (2b) without consideration of the context. None of these settings, does, however, enable to answer the question which emotion is presumably experienced at a specific moment in time. For instance, a customer’s request of “My computer broke.” would be annotated with anger. This emotion persists in a potential follow-up reply “It is out of warranty.” which would also correspond to the global emotion label. An alternative reply “We will send you a new one.” might, in contrast, lead to relief. Modeling these label relations requires classification of textual parts under consideration of the past, but without access to the future. Consequently, we propose a novel annotation setup for emotion categorization corpora, in which the annotations reflect the emotion up to the annotated sentence. We ensure this by uncovering the textual parts step-by-step to the annotator, asking for a label in each step. This perspective is important to understand the final, global emotion, while having access to the individual sentence’s emotion contributions to this final emotion. In modeling experiments, we use these data to check if the context is indeed required to automatically predict such cumulative emotion progressions. KW - Emotion ER - TY - CPAPER TI - Can Factual Statements Be Deceptive? : The DeFaBel Corpus of Belief-based Deception AU - Velutharambath, Aswathy AU - Wührl, Amelie AU - Klinger, Roman PY - 2024 T2 - Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) SP - 2708-2723 PB - ELRA and ICCL UR - https://fis.uni-bamberg.de/handle/uniba/95819 LA - eng AB - If a person firmly believes in a non-factual statement, such as “The Earth is flat”, and argues in its favor, there is no inherent intention to deceive. As the argumentation stems from genuine belief, it may be unlikely to exhibit the linguistic properties associated with deception or lying. This interplay of factuality, personal belief, and intent to deceive remains an understudied area. Disentangling the influence of these variables in argumentation is crucial to gain a better understanding of the linguistic properties attributed to each of them. To study the relation between deception and factuality, based on belief, we present the DeFaBel corpus, a crowd-sourced resource of belief-based deception. To create this corpus, we devise a study in which participants are instructed to write arguments supporting statements like “eating watermelon seeds can cause indigestion”, regardless of its factual accuracy or their personal beliefs about the statement. In addition to the generation task, we ask them to disclose their belief about the statement. The collected instances are labelled as deceptive if the arguments are in contradiction to the participants’ personal beliefs. Each instance in the corpus is thus annotated (or implicitly labelled) with personal beliefs of the author, factuality of the statement, and the intended deceptiveness. The DeFaBel corpus contains 1031 texts in German, out of which 643 are deceptive and 388 are non-deceptive. It is the first publicly available corpus for studying deception in German. In our analysis, we find that people are more confident in the persuasiveness of their arguments when the statement is aligned with their belief, but surprisingly less confident when they are generating arguments in favor of facts. The DeFaBel corpus can be obtained from https://www.ims.uni-stuttgart.de/data/defabel . KW - Factual Statements ER - TY - CPAPER TI - “You are an expert annotator” : Automatic Best–Worst-Scaling Annotations for Emotion Intensity Modeling AU - Bagdon, Christopher Doyle AU - Karmalkar, Prathamesh AU - Gurulingappa, Harsha AU - Klinger, Roman PY - 2024 T2 - Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies SP - 7917–7929 PB - Association for Computational Linguistics CY - Mexico City, Mexico UR - https://fis.uni-bamberg.de/handle/uniba/95820 LA - eng ER - TY - CPAPER TI - English Prompts are Better for NLI-based Zero-Shot Emotion Classification than Target-Language Prompts AU - Bareiss, Patrick AU - Klinger, Roman AU - Barnes, Jeremy PY - 2024 T2 - WWW '24: Companion Proceedings of the ACM on Web Conference 2024 SP - 1318–1326 CY - New York SN - 979-8-4007-0172-6 DO - 10.1145/3589335.3651902 UR - https://fis.uni-bamberg.de/handle/uniba/95270 LA - eng ER - TY - CPAPER TI - What Makes Medical Claims (Un)Verifiable? : Analyzing Entity and Relation Properties for Fact Verification AU - Wührl, Amelie AU - Menchaca Resendiz, Yarik AU - Grimminger, Lara AU - Klinger, Roman PY - 2024 T2 - Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics SP - 2046–2058 PB - Association for Computational Linguistics UR - https://fis.uni-bamberg.de/handle/uniba/93870 LA - eng AB - Verifying biomedical claims fails if no evidence can be discovered. In these cases, the fact-checking verdict remains unknown and the claim is unverifiable. To improve this situation, we have to understand if there are any claim properties that impact its verifiability. In this work we assume that entities and relations define the core variables in a biomedical claim’s anatomy and analyze if their properties help us to differentiate verifiable from unverifiable claims. In a study with trained annotation experts we prompt them to find evidence for biomedical claims, and observe how they refine search queries for their evidence search. This leads to the first corpus for scientific fact verification annotated with subject–relation–object triplets, evidence documents, and fact-checking verdicts (the BEAR-FACT corpus). We find (1) that discovering evidence for negated claims (e.g., X–does-not-cause–Y) is particularly challenging. Further, we see that annotators process queries mostly by adding constraints to the search and by normalizing entities to canonical names. (2) We compare our in-house annotations with a small crowdsourcing setting where we employ both medical experts and laypeople. We find that domain expertise does not have a substantial effect on the reliability of annotations. Finally, (3), we demonstrate that it is possible to reliably estimate the success of evidence retrieval purely from the claim text (.82F1), whereas identifying unverifiable claims proves more challenging (.27F1) KW - Medical Claims ER - TY - CPAPER TI - Topic Bias in Emotion Classification AU - Wegge, Maximilian AU - Klinger, Roman PY - 2024 T2 - Proceedings of the Ninth Workshop on Noisy and User-generated Text (W-NUT 2024) SP - 89-103 PB - Association for Computational Linguistics CY - San Ġiljan, Malta UR - https://fis.uni-bamberg.de/handle/uniba/93869 LA - eng AB - Emotion corpora are typically sampled based on keyword/hashtag search or by asking study participants to generate textual instances. In any case, these corpora are not uniform samples representing the entirety of a domain. We hypothesize that this practice of data acquision leads to unrealistic correlations between overrepresented topics in these corpora that harm the generalizability of models. Such topic bias could lead to wrong predictions for instances like “I organized the service for my aunt’s funeral.” when funeral events are overpresented for instances labeled with sadness, despite the emotion of pride being more appropriate here. In this paper, we study this topic bias both from the data and the modeling perspective. We first label a set of emotion corpora automatically via topic modeling and show that emotions in fact correlate with specific topics. Further, we see that emotion classifiers are confounded by such topics. Finally, we show that the established debiasing method of adversarial correction via gradient reversal mitigates the issue. Our work points out issues with existing emotion corpora and that more representative resources are required for fair evaluation of models predicting affective concepts from text. KW - emotion classification ER - TY - CPAPER TI - An Entity-based Claim Extraction Pipeline for Real-world Fact-checking AU - Wührl, Amelie AU - Grimminger, Lara AU - Klinger, Roman PY - 2023 T2 - Proceedings of the Sixth Fact Extraction and VERification Workshop (FEVER) SP - 29–37 PB - ACL CY - Dubrovnik, Croatia DO - 10.18653/v1/2023.fever-1.3 UR - https://fis.uni-bamberg.de/handle/uniba/116502 LA - eng ER - TY - CPAPER TI - Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis ED - Barnes, Jeremy ED - Clercq, Orphee De ED - Klinger, Roman PY - 2023 PB - Association for Computational Linguistics CY - Toronto, Canada SN - 978-1-959429-87-6 UR - https://fis.uni-bamberg.de/handle/uniba/116449 LA - eng ER - TY - CHAP TI - Emotionsklassifikation in Texten unter Berücksichtigung des Komponentenprozessmodells AU - Klinger, Roman PY - 2023 SP - 131-154 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96381 LA - deu AB - Ein wichtiger Bestandteil unserer alltäglichen Kommunikation, neben der Mitteilung und Beschreibung von Ereignissen und Fakten, ist der Ausdruck von Emotionen, welcher auch Bestandteil von Hassrede ist: Es wird zum Beispiel Wut zum Ausdruck gebracht, was wiederum bei den Betroffenen Angst, Traurigkeit oder vielleicht auch Überraschung auslösen kann. In der maschinellen Verarbeitung von Sprache haben sich in der letzten Zeit einige konkrete Aufgaben, welche Teil der Emotionsanalyse in Text sind, herauskristallisiert. Diese sind zum einen Klassifikationsaufgaben (welche Emotion drückt ein Text aus?) und zum anderen relationale Strukturlernaufgaben (welche Wörter bezeichnen die Person, die eine Emotion fühlt und welche Wörter lassen auf die Ursache der Emotion schließen?). Wir verschaffen uns in diesem Kapitel einen kurzen Überblick über das Feld und diskutieren im Anschluss etwas genauer, wie sich die Beschreibungen von Emotionen in verschiedenen Domänen unterscheiden und wie Ereignisbeschreibungen mit Hilfe psychologischer Theorien mit Emotionen zusammengebracht werden können. Insbesondere analysieren wir auf Basis des Emotions-Komponenten-Prozessmodells, auf welche Komponenten von Emotionen (subjektives Gefühl, kognitive Evaluation, Körperreaktion, Ausdruck, Motivation) Autor:innen zugreifen, und stellen fest, dass diese Verteilung zwischen sozialen Medien und Literatur unterschiedlich ist. In beiden Domänen spielt aber die kognitive Komponente zur Interpretation von Emotionen eine wichtige Rolle. Dies zeigt auf, dass insbesondere der Ereignisinterpretation Aufmerksamkeit geschenkt werden muss, um implizit kommunizierte Emotionen aufzudecken. Dies motiviert uns, Emotionen mit Hilfe der Appraisaltheorien zu analysieren, welche den Zusammenhang zwischen kognitiven Prozessen und Emotionen erklären. Zu beiden Konzepten – dem Komponentenmodell und den Appraisaltheorien – präsentieren wir Textkorpora und Klassifikationsmodelle. KW - Emotionsklassifikation ER - TY - CPAPER TI - Cross-lingual Inference with A Chinese Entailment Graph AU - Li, Tianyi AU - Weber, Sabine AU - Hosseini, Mohammad Javad AU - Guillou, Liane AU - Steedman, Mark PY - 2022 T2 - Findings of the Association for Computational Linguistics: ACL 2022 SP - 1214–1233 PB - Association for Computational Linguistics DO - 10.18653/v1/2022.findings-acl.96 UR - https://fis.uni-bamberg.de/handle/uniba/97224 LA - eng AB - Predicate entailment detection is a crucial task for question-answering from text, where previous work has explored unsupervised learning of entailment graphs from typed open relation triples. In this paper, we present the first pipeline for building Chinese entailment graphs, which involves a novel high-recall open relation extraction (ORE) method and the first Chinese fine-grained entity typing dataset under the FIGER type ontology. Through experiments on the Levy-Holt dataset, we verify the strength of our Chinese entailment graph, and reveal the cross-lingual complementarity: on the parallel Levy-Holt dataset, an ensemble of Chinese and English entailment graphs outperforms both monolingual graphs, and raises unsupervised SOTA by 4.7 AUC points. KW - Chinese Entailment Graph ER - TY - JOUR TI - From theories on styles to their transfer in text : Bridging the gap with a hierarchical survey AU - Troiano, Enrica AU - Velutharambath, Aswathy AU - Klinger, Roman PY - 2022 JO - Natural Language Engineering VL - 29 IS - 4 SP - 849-908 PB - Cambridge University Press CY - Cambridge SN - 1351-3249 DO - 10.1017/S1351324922000407 UR - https://fis.uni-bamberg.de/handle/uniba/93895 LA - eng AB - Humans are naturally endowed with the ability to write in a particular style. They can, for instance, rephrase a formal letter in an informal way, convey a literal message with the use of figures of speech or edit a novel by mimicking the style of some well-known authors. Automating this form of creativity constitutes the goal of style transfer. As a natural language generation task, style transfer aims at rewriting existing texts, and specifically, it creates paraphrases that exhibit some desired stylistic attributes. From a practical perspective, it envisions beneficial applications, like chatbots that modulate their communicative style to appear empathetic, or systems that automatically simplify technical articles for a non-expert audience. Several style-aware paraphrasing methods have attempted to tackle style transfer. A handful of surveys give a methodological overview of the field, but they do not support researchers to focus on specific styles. With this paper, we aim at providing a comprehensive discussion of the styles that have received attention in the transfer task. We organize them in a hierarchy, highlighting the challenges for the definition of each of them and pointing out gaps in the current research landscape. The hierarchy comprises two main groups. One encompasses styles that people modulate arbitrarily, along the lines of registers and genres. The other group corresponds to unintentionally expressed styles, due to an author’s personal characteristics. Hence, our review shows how these groups relate to one another and where specific styles, including some that have not yet been explored, belong in the hierarchy. Moreover, we summarize the methods employed for different stylistic families, hinting researchers towards those that would be the most fitting for future research. KW - Natural Language Generation KW - Translation Technology KW - Style Transfer KW - Paraphrasing ER - TY - CPAPER TI - Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference AU - Kadiķis, Emīls AU - Srivastav, Vaibhav AU - Klinger, Roman PY - 2022 T2 - Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies SP - 6031-6037 PB - Association for Computational Linguistics CY - Seattle DO - 10.18653/v1/2022.naacl-main.441 UR - https://fis.uni-bamberg.de/handle/uniba/93889 LA - eng AB - The task of natural language inference (NLI), to decide if a hypothesis entails or contradicts a premise, received considerable attention in recent years. All competitive systems build on top of contextualized representations and make use of transformer architectures for learning an NLI model. When somebody is faced with a particular NLI task, they need to select the best model that is available. This is a time-consuming and resource-intense endeavour. To solve this practical problem, we propose a simple method for predicting the performance without actually fine-tuning the model. We do this by testing how well the pre-trained models perform on the aNLI task when just comparing sentence embeddings with cosine similarity to what kind of performance is achieved when training a classifier on top of these embeddings. We show that the accuracy of the cosine similarity approach correlates strongly with the accuracy of the classification approach with a Pearson correlation coefficient of 0.65. Since the similarity is orders of magnitude faster to compute on a given dataset (less than a minute vs. hours), our method can lead to significant time savings in the process of model selection. KW - Abductive Natural Language Inference ER - TY - CPAPER TI - “splink” is happy and “phrouth” is scary : Emotion Intensity Analysis for Nonsense Words AU - Sabbatino, Valentino AU - Troiano, Enrica AU - Schweitzer, Antje AU - Klinger, Roman PY - 2022 T2 - Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis SP - 37-50 PB - Association for Computational Linguistics CY - Dublin DO - 10.18653/v1/2022.wassa-1.4 UR - https://fis.uni-bamberg.de/handle/uniba/93888 LA - eng AB - People associate affective meanings to words - “death” is scary and sad while “party” is connotated with surprise and joy. This raises the question if the association is purely a product of the learned affective imports inherent to semantic meanings, or is also an effect of other features of words, e.g., morphological and phonological patterns. We approach this question with an annotation-based analysis leveraging nonsense words. Specifically, we conduct a best-worst scaling crowdsourcing study in which participants assign intensity scores for joy, sadness, anger, disgust, fear, and surprise to 272 non-sense words and, for comparison of the results to previous work, to 68 real words. Based on this resource, we develop character-level and phonology-based intensity regressors. We evaluate them on both nonsense words and real words (making use of the NRC emotion intensity lexicon of 7493 words), across six emotion categories. The analysis of our data reveals that some phonetic patterns show clear differences between emotion intensities. For instance, s as a first phoneme contributes to joy, sh to surprise, p as last phoneme more to disgust than to anger and fear. In the modelling experiments, a regressor trained on real words from the NRC emotion intensity lexicon shows a higher performance (r = 0.17) than regressors that aim at learning the emotion connotation purely from nonsense words. We conclude that humans do associate affective meaning to words based on surface patterns, but also based on similarities to existing words (“juy” to “joy”, or “flike” to “like”). KW - Emotion Intensity Analysis ER - TY - CPAPER TI - x-enVENT : A Corpus of Event Descriptions with Experiencer-specific Emotion and Appraisal Annotations AU - Troiano, Enrica AU - Oberlaender, Laura Ana Maria AU - Wegge, Maximilian AU - Klinger, Roman PY - 2022 T2 - Proceedings of the Thirteenth Language Resources and Evaluation Conference SP - 1365-1375 PB - European Language Resources Association CY - Marseille UR - https://fis.uni-bamberg.de/handle/uniba/93887 LA - eng AB - Emotion classification is often formulated as the task to categorize texts into a predefined set of emotion classes. So far, this task has been the recognition of the emotion of writers and readers, as well as that of entities mentioned in the text. We argue that a classification setup for emotion analysis should be performed in an integrated manner, including the different semantic roles that participate in an emotion episode. Based on appraisal theories in psychology, which treat emotions as reactions to events, we compile an English corpus of written event descriptions. The descriptions depict emotion-eliciting circumstances, and they contain mentions of people who responded emotionally. We annotate all experiencers, including the original author, with the emotions they likely felt. In addition, we link them to the event they found salient (which can be different for different experiencers in a text) by annotating event properties, or appraisals (e.g., the perceived event undesirability, the uncertainty of its outcome). Our analysis reveals patterns in the co-occurrence of people’s emotions in interaction. Hence, this richly-annotated resource provides useful data to study emotions and event evaluations from the perspective of different roles, and it enables the development of experiencer-specific emotion and appraisal classification systems. KW - emotion analysis KW - corpus KW - affective computing KW - role labeling KW - emotion experiencer KW - appraisal theories KW - events ER - TY - CPAPER TI - CoVERT : A Corpus of Fact-checked Biomedical COVID-19 Tweets AU - Mohr, Isabelle AU - Wührl, Amelie AU - Klinger, Roman PY - 2022 T2 - Proceedings of the Thirteenth Language Resources and Evaluation Conference SP - 244-257 PB - European Language Resources Association CY - Marseille UR - https://fis.uni-bamberg.de/handle/uniba/93885 LA - eng AB - During the first two years of the COVID-19 pandemic, large volumes of biomedical information concerning this new disease have been published on social media. Some of this information can pose a real danger, particularly when false information is shared, for instance recommendations how to treat diseases without professional medical advice. Therefore, automatic fact-checking resources and systems developed specifically for medical domain are crucial. While existing fact-checking resources cover COVID-19 related information in news or quantify the amount of misinformation in tweets, there is no dataset providing fact-checked COVID-19 related Twitter posts with detailed annotations for biomedical entities, relations and relevant evidence. We contribute CoVERT, a fact-checked corpus of tweets with a focus on the domain of biomedicine and COVID-19 related (mis)information. The corpus consists of 300 tweets, each annotated with named entities and relations. We employ a novel crowdsourcing methodology to annotate all tweets with fact-checking labels and supporting evidence, which crowdworkers search for online. This methodology results in substantial inter-annotator agreement. Furthermore, we use the retrieved evidence extracts as part of a fact-checking pipeline, finding that the real-world evidence is more useful than the knowledge directly available in pretrained language models. KW - CoVERT ER - TY - CPAPER TI - Natural Language Inference Prompts for Zero-shot Emotion Classification in Text across Corpora AU - Plaza-del-Arco, Flor Miriam AU - Martín-Valdivia, María-Teresa AU - Klinger, Roman PY - 2022 T2 - Proceedings of the 29th International Conference on Computational Linguistics SP - 6805-6817 PB - International Committee on Computational Linguistics CY - Gyeongju UR - https://fis.uni-bamberg.de/handle/uniba/93883 LA - eng AB - Within textual emotion classification, the set of relevant labels depends on the domain and application scenario and might not be known at the time of model development. This conflicts with the classical paradigm of supervised learning in which the labels need to be predefined. A solution to obtain a model with a flexible set of labels is to use the paradigm of zero-shot learning as a natural language inference task, which in addition adds the advantage of not needing any labeled training data. This raises the question how to prompt a natural language inference model for zero-shot learning emotion classification. Options for prompt formulations include the emotion name anger alone or the statement “This text expresses anger”. With this paper, we analyze how sensitive a natural language inference-based zero-shot-learning classifier is to such changes to the prompt under consideration of the corpus: How carefully does the prompt need to be selected? We perform experiments on an established set of emotion datasets presenting different language registers according to different sources (tweets, events, blogs) with three natural language inference models and show that indeed the choice of a particular prompt formulation needs to fit to the corpus. We show that this challenge can be tackled with combinations of multiple prompts. Such ensemble is more robust across corpora than individual prompts and shows nearly the same performance as the individual best prompt for a particular corpus. KW - Natural Language Inference Prompts ER - TY - CPAPER TI - Claim Detection in Biomedical Twitter Posts AU - Wührl, Amelie AU - Klinger, Roman PY - 2021 SP - 131-142 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108351 LA - eng AB - Social media contains unfiltered and unique information, which is potentially of great value, but, in the case of misinformation, can also do great harm. With regards to biomedical topics, false information can be particularly dangerous. Methods of automatic fact-checking and fake news detection address this problem, but have not been applied to the biomedical domain in social media yet. We aim to fill this research gap and annotate a corpus of 1200 tweets for implicit and explicit biomedical claims (the latter also with span annotations for the claim phrase). With this corpus, which we sample to be related to COVID-19, measles, cystic fibrosis, and depression, we develop baseline models which detect tweets that contain a claim automatically. Our analyses reveal that biomedical tweets are densely populated with claims (45 % in a corpus sampled to contain 1200 tweets focused on the domains mentioned above). Baseline classification experiments with embedding-based classifiers and BERT-based transfer learning demonstrate that the detection is challenging, however, shows acceptable performance for the identification of explicit expressions of claims. Implicit claim tweets are more challenging to detect. KW - Biomedical Twitter Posts ER - TY - CPAPER TI - Zero-Shot Cross-Lingual Transfer is a Hard Baseline to Beat in German Fine-Grained Entity Typing AU - Weber, Sabine AU - Steedman, Mark PY - 2021 T2 - Proceedings of the Second Workshop on Insights from Negative Results in NLP SP - 42–48 PB - Association for Computational Linguistics DO - 10.18653/v1/2021.insights-1.7 UR - https://fis.uni-bamberg.de/handle/uniba/97228 LA - eng AB - The training of NLP models often requires large amounts of labelled training data, which makes it difficult to expand existing models to new languages. While zero-shot cross-lingual transfer relies on multilingual word embeddings to apply a model trained on one language to another, Yarowski and Ngai (2001) propose the method of annotation projection to generate training data without manual annotation. This method was successfully used for the tasks of named entity recognition and coarse-grained entity typing, but we show that it is outperformed by zero-shot cross-lingual transfer when applied to the similar task of fine-grained entity typing. In our study of fine-grained entity typing with the FIGER type ontology for German, we show that annotation projection amplifies the English model’s tendency to underpredict level 2 labels and is beaten by zero-shot cross-lingual transfer on three novel test sets. KW - Zero-Shot Cross-Lingual Transfer ER - TY - CPAPER TI - Multi-Task Learning with Sentiment, Emotion, and Target Detection to Recognize Hate Speech and Offensive Language AU - Plaza-del-Arco, Flor Miriam AU - Halat, Sercan AU - Klinger, Roman AU - Padó, Sebastian PY - 2021 T2 - Working Notes of FIRE 2021 : Forum for Information Retrieval Evaluation Gandhinagar, India, December 13-17, 2021 SP - 297-318 T3 - CEUR Workshop Proceedings UR - https://fis.uni-bamberg.de/handle/uniba/93896 LA - eng AB - The recognition of hate speech and offensive language (HOF) is commonly formulated as a classification task which asks models to decide if a text contains HOF. This task is challenging because of the large variety of explicit and implicit ways to verbally attack a target person or group. In this paper, we investigate whether HOF detection can profit by taking into account the relationships between HOF and similar concepts: (a) HOF is related to sentiment analysis because hate speech is typically a negative statement and expresses a negative opinion; (b) it is related to emotion analysis, as expressed hate points to the author experiencing (or pretending to experience) anger while the addressees experience (or are intended to experience) fear. (c) Finally, one constituting element of HOF is the (explicit or implicit) mention of a targeted person or group. On this basis, we hypothesize that HOF detection shows improvements when being modeled jointly with these concepts, in a multi-task learning setup. We base our experiments on existing data sets for each of these concepts (sentiment, emotion, target of HOF) and evaluate our models as a participant (as team IMS-SINAI) in the HASOC FIRE 2021 English Subtask 1A: “Subtask 1A: Identifying Hate, offensive and profane content from the post”. Based on model-selection experiments in which we consider multiple available resources and submissions to the shared task, we find that the combination of the CrowdFlower emotion corpus, the SemEval 2016 Sentiment Corpus, and the OffensEval 2019 target detection data leads to an F1 =.7947 in a multi-head multi-task learning model based on BERT, in comparison to .7895 of a plain BERT model. On the HASOC 2019 test data, this result is more substantial with an increase by 2pp in F1 (from 0.78 F1 to 0.8 F1) and a considerable increase in recall. Across both data sets (2019, 2021), the recall is particularly increased for the class of HOF (6pp for the 2019 data and 3pp for the 2021 data), showing that MTL with emotion, sentiment, and target identification is an appropriate approach for early warning systems that might be deployed in social media platforms. KW - multi-task learning KW - hate and offensive language detection KW - sentiment analysis KW - emotion analysis KW - target classification KW - social media mining ER - TY - CPAPER TI - Emotion Recognition under Consideration of the Emotion Component Process Model AU - Casel, Felix AU - Heindl, Amelie AU - Klinger, Roman PY - 2021 T2 - Proceedings of the 17th Conference on Natural Language Processing (KONVENS 2021) SP - 49-61 PB - KONVENS 2021 Organizers UR - https://fis.uni-bamberg.de/handle/uniba/93898 LA - eng ER - TY - CPAPER TI - Claim Detection in Biomedical Twitter Posts AU - Wührl, Amelie AU - Klinger, Roman PY - 2021 T2 - Proceedings of the 20th Workshop on Biomedical Language Processing SP - 131-142 PB - Association for Computational Linguistics DO - 10.18653/v1/2021.bionlp-1.15 UR - https://fis.uni-bamberg.de/handle/uniba/93899 LA - eng AB - Social media contains unfiltered and unique information, which is potentially of great value, but, in the case of misinformation, can also do great harm. With regards to biomedical topics, false information can be particularly dangerous. Methods of automatic fact-checking and fake news detection address this problem, but have not been applied to the biomedical domain in social media yet. We aim to fill this research gap and annotate a corpus of 1200 tweets for implicit and explicit biomedical claims (the latter also with span annotations for the claim phrase). With this corpus, which we sample to be related to COVID-19, measles, cystic fibrosis, and depression, we develop baseline models which detect tweets that contain a claim automatically. Our analyses reveal that biomedical tweets are densely populated with claims (45 % in a corpus sampled to contain 1200 tweets focused on the domains mentioned above). Baseline classification experiments with embedding-based classifiers and BERT-based transfer learning demonstrate that the detection is challenging, however, shows acceptable performance for the identification of explicit expressions of claims. Implicit claim tweets are more challenging to detect. KW - Biomedical Twitter Posts ER - TY - CPAPER TI - Emotion Ratings : How Intensity, Annotation Confidence and Agreements are Entangled AU - Troiano, Enrica AU - Padó, Sebastian AU - Klinger, Roman PY - 2021 T2 - Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis SP - 40-49 PB - Association for Computational Linguistics UR - https://fis.uni-bamberg.de/handle/uniba/93902 LA - eng AB - When humans judge the affective content of texts, they also implicitly assess the correctness of such judgment, that is, their confidence. We hypothesize that people’s (in)confidence that they performed well in an annotation task leads to (dis)agreements among each other. If this is true, confidence may serve as a diagnostic tool for systematic differences in annotations. To probe our assumption, we conduct a study on a subset of the Corpus of Contemporary American English, in which we ask raters to distinguish neutral sentences from emotion-bearing ones, while scoring the confidence of their answers. Confidence turns out to approximate inter-annotator disagreements. Further, we find that confidence is correlated to emotion intensity: perceiving stronger affect in text prompts annotators to more certain classification performances. This insight is relevant for modelling studies of intensity, as it opens the question wether automatic regressors or classifiers actually predict intensity, or rather human’s self-perceived confidence. KW - Emotion Ratings ER - TY - CPAPER TI - Hate Towards the Political Opponent : A Twitter Corpus Study of the 2020 US Elections on the Basis of Offensive Speech and Stance Detection AU - Grimminger, Lara AU - Klinger, Roman PY - 2021 T2 - Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis SP - 171-180 PB - Association for Computational Linguistics UR - https://fis.uni-bamberg.de/handle/uniba/93903 LA - eng AB - The 2020 US Elections have been, more than ever before, characterized by social media campaigns and mutual accusations. We investigate in this paper if this manifests also in online communication of the supporters of the candidates Biden and Trump, by uttering hateful and offensive communication. We formulate an annotation task, in which we join the tasks of hateful/offensive speech detection and stance detection, and annotate 3000 Tweets from the campaign period, if they express a particular stance towards a candidate. Next to the established classes of favorable and against, we add mixed and neutral stances and also annotate if a candidate is mentioned with- out an opinion expression. Further, we an- notate if the tweet is written in an offensive style. This enables us to analyze if supporters of Joe Biden and the Democratic Party communicate differently than supporters of Donald Trump and the Republican Party. A BERT baseline classifier shows that the detection if somebody is a supporter of a candidate can be performed with high quality (.89 F1 for Trump and .91 F1 for Biden), while the detection that somebody expresses to be against a candidate is more challenging (.79 F1 and .64 F1, respectively). The automatic detection of hate/offensive speech remains challenging (with .53 F1). Our corpus is publicly available and constitutes a novel resource for computational modelling of offensive language under consideration of stances. KW - Twitter Corpus Study ER - TY - JOUR TI - Assessing the Russian Internet Research Agency’s impact on the political attitudes and behaviors of American Twitter users in late 2017 AU - Bail, Christopher A. AU - Guay, Brian AU - Maloney, Emily AU - Combs, Aidan AU - Hillygus, D. Sunshine AU - Merhout, Friedolin AU - Freelon, Deen AU - Volfovsky, Alexander PY - 2020 JO - Proceedings of the National Academy of Sciences of the United States of America VL - 117 IS - 1 SP - 243–250 PB - National Acad. of Sciences CY - Washington, DC SN - 0027-8424 DO - 10.1073/pnas.1906420116 UR - https://fis.uni-bamberg.de/handle/uniba/97249 LA - eng AB - There is widespread concern that Russia and other countries have launched social-media campaigns designed to increase political divisions in the United States. Though a growing number of studies analyze the strategy of such campaigns, it is not yet known how these efforts shaped the political attitudes and behaviors of Americans. We study this question using longitudinal data that describe the attitudes and online behaviors of 1,239 Republican and Democratic Twitter users from late 2017 merged with nonpublic data about the Russian Internet Research Agency (IRA) from Twitter. Using Bayesian regression tree models, we find no evidence that interaction with IRA accounts substantially impacted 6 distinctive measures of political attitudes and behaviors over a 1-mo period. We also find that interaction with IRA accounts were most common among respondents with strong ideological homophily within their Twitter network, high interest in politics, and high frequency of Twitter usage. Together, these findings suggest that Russian trolls might have failed to sow discord because they mostly interacted with those who were already highly polarized. We conclude by discussing several important limitations of our study—especially our inability to determine whether IRA accounts influenced the 2016 presidential election—as well as its implications for future research on social media influence campaigns, political polarization, and computational social science. KW - misinformation KW - social media KW - political polarization KW - computational social science ER - TY - GEN TI - Strukturierte Modellierung von Affekt in Text AU - Klinger, Roman PY - 2020 CY - Stuttgart DO - 10.18419/opus-10994 UR - https://fis.uni-bamberg.de/handle/uniba/93891 LA - deu ER - TY - CPAPER TI - Experiencers, Stimuli, or Targets : Which Semantic Roles Enable Machine Learning to Infer the Emotions? AU - Oberländer, Laura AU - Reich, Kevin AU - Klinger, Roman PY - 2020 T2 - Proceedings of the Third Workshop on Computational Modeling of People's Opinions, Personality, and Emotion's in Social Media SP - 119–128 PB - Association for Computational Linguistics UR - https://fis.uni-bamberg.de/handle/uniba/93900 LA - eng AB - Emotion recognition is predominantly formulated as text classification in which textual units are assigned to an emotion from a predefined inventory (e.g., fear, joy, anger, disgust, sadness, surprise, trust, anticipation). More recently, semantic role labeling approaches have been developed to extract structures from the text to answer questions like: “who is described to feel the emotion?” (experiencer), “what causes this emotion?” (stimulus), and at which entity is it directed?” (target). Though it has been shown that jointly modeling stimulus and emotion category prediction is beneficial for both subtasks, it remains unclear which of these semantic roles enables a classifier to infer the emotion. Is it the experiencer, because the identity of a person is biased towards a particular emotion (X is always happy)? Is it a particular target (everybody loves X) or a stimulus (doing X makes everybody sad)? We answer these questions by training emotion classification models on five available datasets annotated with at least one semantic role by masking the fillers of these roles in the text in a controlled manner and find that across multiple corpora, stimuli and targets carry emotion information, while the experiencer might be considered a confounder. Further, we analyze if informing the model about the position of the role improves the classification decision. Particularly on literature corpora we find that the role information improves the emotion classification. KW - Machine Learning ER - TY - CPAPER TI - A Computational Analysis of Financial and Environmental Narratives within Financial Reports and its Value for Investors AU - Armbrust, Felix AU - Schäfer, Henry AU - Klinger, Roman PY - 2020 T2 - Proceedings of the 1st Joint Workshop on Financial Narrative Processing and MultiLing Financial Summarisation SP - 181-194 PB - COLING UR - https://fis.uni-bamberg.de/handle/uniba/93901 LA - eng ER - TY - CPAPER TI - Appraisal Theories for Emotion Classification in Text AU - Hofmann, Jan AU - Troiano, Enrica AU - Sassenberg, Kai AU - Klinger, Roman PY - 2020 T2 - Proceedings of the 28th International Conference on Computational Linguistics SP - 125-138 PB - International Committee on Computational Linguistics DO - 10.18653/v1/2020.coling-main.11 UR - https://fis.uni-bamberg.de/handle/uniba/93905 LA - eng AB - Automatic emotion categorization has been predominantly formulated as text classification in which textual units are assigned to an emotion from a predefined inventory, for instance following the fundamental emotion classes proposed by Paul Ekman (fear, joy, anger, disgust, sadness, surprise) or Robert Plutchik (adding trust, anticipation). This approach ignores existing psychological theories to some degree, which provide explanations regarding the perception of events. For instance, the description that somebody discovers a snake is associated with fear, based on the appraisal as being an unpleasant and non-controllable situation. This emotion reconstruction is even possible without having access to explicit reports of a subjective feeling (for instance expressing this with the words “I am afraid.”). Automatic classification approaches therefore need to learn properties of events as latent variables (for instance that the uncertainty and the mental or physical effort associated with the encounter of a snake leads to fear). With this paper, we propose to make such interpretations of events explicit, following theories of cognitive appraisal of events, and show their potential for emotion classification when being encoded in classification models. Our results show that high quality appraisal dimension assignments in event descriptions lead to an improvement in the classification of discrete emotion categories. We make our corpus of appraisal-annotated emotion-associated event descriptions publicly available. KW - Emotion Classification ER - TY - CPAPER TI - Lost in Back-Translation : Emotion Preservation in Neural Machine Translation AU - Troiano, Enrica AU - Klinger, Roman AU - Padó, Sebastian PY - 2020 T2 - Proceedings of the 28th International Conference on Computational Linguistics SP - 4340-4354 PB - International Committee on Computational Linguistics DO - 10.18653/v1/2020.coling-main.384 UR - https://fis.uni-bamberg.de/handle/uniba/93906 LA - eng AB - Machine translation provides powerful methods to convert text between languages, and is therefore a technology enabling a multilingual world. An important part of communication, however, takes place at the non-propositional level (e.g., politeness, formality, emotions), and it is far from clear whether current MT methods properly translate this information. This paper investigates the specific hypothesis that the non-propositional level of emotions is at least partially lost in MT. We carry out a number of experiments in a back-translation setup and establish that (1) emotions are indeed partially lost during translation; (2) this tendency can be reversed almost completely with a simple re-ranking approach informed by an emotion classifier, taking advantage of diversity in the n-best list; (3) the re-ranking approach can also be applied to change emotions, obtaining a model for emotion style transfer. An in-depth qualitative analysis reveals that there are recurring linguistic changes through which emotions are toned down or amplified, such as change of modality. KW - Emotion Preservation ER - TY - CPAPER TI - Dissecting Span Identification Tasks with Performance Prediction AU - Papay, Sean AU - Klinger, Roman AU - Padó, Sebastian PY - 2020 T2 - Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) SP - 4881-4895 PB - Association for Computational Linguistics DO - 10.18653/v1/2020.emnlp-main.396 UR - https://fis.uni-bamberg.de/handle/uniba/93908 LA - eng AB - Span identification (in short, span ID) tasks such as chunking, NER, or code-switching detection, ask models to identify and classify relevant spans in a text. Despite being a staple of NLP, and sharing a common structure, there is little insight on how these tasks’ properties influence their difficulty, and thus little guidance on what model families work well on span ID tasks, and why. We analyze span ID tasks via performance prediction, estimating how well neural architectures do on different tasks. Our contributions are: (a) we identify key properties of span ID tasks that can inform performance prediction; (b) we carry out a large-scale experiment on English data, building a model to predict performance for unseen span ID tasks that can support architecture choices; (c), we investigate the parameters of the meta model, yielding new insights on how model and task properties interact to affect span ID performance. We find, e.g., that span frequency is especially important for LSTMs, and that CRFs help when spans are infrequent and boundaries non-distinctive. KW - Span Identification Tasks ER - TY - CPAPER TI - Challenges in Emotion Style Transfer : An Exploration with a Lexical Substitution Pipeline AU - Helbig, David AU - Troiano, Enrica AU - Klinger, Roman PY - 2020 T2 - Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media SP - 41-50 PB - Association for Computational Linguistics DO - 10.18653/v1/2020.socialnlp-1.6 UR - https://fis.uni-bamberg.de/handle/uniba/93911 LA - eng AB - We propose the task of emotion style transfer, which is particularly challenging, as emotions (here: anger, disgust, fear, joy, sadness, surprise) are on the fence between content and style. To understand the particular difficulties of this task, we design a transparent emotion style transfer pipeline based on three steps: (1) select the words that are promising to be substituted to change the emotion (with a brute-force approach and selection based on the attention mechanism of an emotion classifier), (2) find sets of words as candidates for substituting the words (based on lexical and distributional semantics), and (3) select the most promising combination of substitutions with an objective function which consists of components for content (based on BERT sentence embeddings), emotion (based on an emotion classifier), and fluency (based on a neural language model). This comparably straight-forward setup enables us to explore the task and understand in what cases lexical substitution can vary the emotional load of texts, how changes in content and style interact and if they are at odds. We further evaluate our pipeline quantitatively in an automated and an annotation study based on Tweets and find, indeed, that simultaneous adjustments of content and emotion are conflicting objectives: as we show in a qualitative analysis motivated by Scherer’s emotion component model, this is particularly the case for implicit emotion expressions based on cognitive appraisal or descriptions of bodily reactions. KW - Emotion Style Transfer ER - TY - CPAPER TI - PO-EMO : Conceptualization, Annotation, and Modeling of Aesthetic Emotions in German and English Poetry AU - Haider, Thomas AU - Eger, Steffen AU - Kim, Evgeny AU - Klinger, Roman AU - Menninghaus, Winfried PY - 2020 T2 - Proceedings of the Twelfth Language Resources and Evaluation Conference SP - 1652-1663 PB - European Language Resources Association CY - Paris UR - https://fis.uni-bamberg.de/handle/uniba/93912 LA - eng AB - Most approaches to emotion analysis of social media, literature, news, and other domains focus exclusively on basic emotion categories as defined by Ekman or Plutchik. However, art (such as literature) enables engagement in a broader range of more complex and subtle emotions. These have been shown to also include mixed emotional responses. We consider emotions in poetry as they are elicited in the reader, rather than what is expressed in the text or intended by the author. Thus, we conceptualize a set of aesthetic emotions that are predictive of aesthetic appreciation in the reader, and allow the annotation of multiple labels per line to capture mixed emotions within their context. We evaluate this novel setting in an annotation experiment both with carefully trained experts and via crowdsourcing. Our annotation with experts leads to an acceptable agreement of k = .70, resulting in a consistent dataset for future large scale analysis. Finally, we conduct first emotion classification experiments based on BERT, showing that identifying aesthetic emotions is challenging in our data, with up to .52 F1-micro on the German subset. Data and resources are available at https://github.com/tnhaider/poetry-emotion. KW - PO-EMO ER - TY - CPAPER TI - GoodNewsEveryone : A Corpus of News Headlines Annotated with Emotions, Semantic Roles, and Reader Perception AU - Bostan, Laura Ana Maria AU - Kim, Evgeny AU - Klinger, Roman PY - 2020 T2 - Proceedings of the Twelfth Language Resources and Evaluation Conference SP - 1554-1566 PB - European Language Resources Association CY - Paris SN - 979-10-95546-34-4 UR - https://fis.uni-bamberg.de/handle/uniba/93913 LA - eng AB - Most research on emotion analysis from text focuses on the task of emotion classification or emotion intensity regression. Fewer works address emotions as a phenomenon to be tackled with structured learning, which can be explained by the lack of relevant datasets. We fill this gap by releasing a dataset of 5000 English news headlines annotated via crowdsourcing with their associated emotions, the corresponding emotion experiencers and textual cues, related emotion causes and targets, as well as the reader’s perception of the emotion of the headline. This annotation task is comparably challenging, given the large number of classes and roles to be identified. We therefore propose a multiphase annotation procedure in which we first find relevant instances with emotional content and then annotate the more fine-grained aspects. Finally, we develop a baseline for the task of automatic prediction of semantic role structures and discuss the results. The corpus we release enables further research on emotion classification, emotion intensity prediction, emotion cause detection, and supports further qualitative studies. KW - GoodNewsEveryone ER - TY - CPAPER TI - Automatic Section Recognition in Obituaries AU - Sabbatino, Valentino AU - Bostan, Laura Ana Maria AU - Klinger, Roman PY - 2020 T2 - Proceedings of the Twelfth Language Resources and Evaluation Conference SP - 817-825 PB - European Language Resources Association UR - https://fis.uni-bamberg.de/handle/uniba/93914 LA - eng AB - Obituaries contain information about people’s values across times and cultures, which makes them a useful resource for exploring cultural history. They are typically structured similarly, with sections corresponding to Personal Information, Biographical Sketch, Characteristics, Family, Gratitude, Tribute, Funeral Information and Other aspects of the person. To make this information available for further studies, we propose a statistical model which recognizes these sections. To achieve that, we collect a corpus of 20058 English obituaries from TheDaily Item, Remembering.CA and The London Free Press. The evaluation of our annotation guidelines with three annotators on 1008 obituaries shows a substantial agreement of Fleiss κ = 0.87. Formulated as an automatic segmentation task, a convolutional neural network outperforms bag-of-words and embedding-based BiLSTMs and BiLSTM-CRFs with a micro F1 = 0.81. KW - Obituaries ER - TY - JOUR TI - Fast approximate STEM image simulations from a machine learning model AU - Combs, Aidan H. AU - Maldonis, Jason J. AU - Feng, Jie AU - Xu, Zhongnan AU - Voyles, Paul M. AU - Morgan, Dane PY - 2019 JO - Advanced Structural and Chemical Imaging VL - 5 IS - 1, 2 SP - 1-10 PB - Springer Science and Business Media LLC CY - Cham SN - 2198-0926 DO - 10.1186/s40679-019-0064-2 UR - https://fis.uni-bamberg.de/handle/uniba/97250 LA - eng AB - Accurate quantum mechanical scanning transmission electron microscopy image simulation methods such as the multislice method require computation times that are too large to use in applications in high-resolution materials imaging that require very large numbers of simulated images. However, higher-speed simulation methods based on linear imaging models, such as the convolution method, are often not accurate enough for use in these applications. We present a method that generates an image from the convolution of an object function and the probe intensity, and then uses a multivariate polynomial fit to a dataset of corresponding multislice and convolution images to correct it. We develop and validate this method using simulated images of Pt and Pt–Mo nanoparticles and find that for these systems, once the polynomial is fit, the method runs about six orders of magnitude faster than parallelized CPU implementations of the multislice method while achieving a 1 − R2 error of 0.010–0.015 and root-mean-square error/standard deviation of dataset being predicted of about 0.1 when compared to full multislice simulations. KW - Scanning transmission electron microscopy KW - Convolution KW - Frozen phonon multislice simulation KW - High-throughput KW - Linear regression ER - TY - CPAPER TI - Construction and Alignment of Multilingual Entailment Graphs for Semantic Inference AU - Weber, Sabine AU - Steedman, Mark PY - 2019 T2 - Proceedings of the 2019 Workshop on Widening NLP SP - 77–79 PB - Association for Computational Linguistics UR - https://fis.uni-bamberg.de/handle/uniba/97229 LA - eng ER - TY - CPAPER TI - An Analysis of Emotion Communication Channels in Fan-Fiction : Towards Emotional Storytelling AU - Kim, Evgeny AU - Klinger, Roman PY - 2019 T2 - Proceedings of the Second Workshop on Storytelling SP - 56-64 PB - Association for Computational Linguistics DO - 10.18653/v1/W19-3406 UR - https://fis.uni-bamberg.de/handle/uniba/93917 LA - eng AB - Centrality of emotion for the stories told by humans is underpinned by numerous studies in literature and psychology. The research in automatic storytelling has recently turned towards emotional storytelling, in which characters’ emotions play an important role in the plot development (Theune et al., 2004; y Perez, 2007; Mendez et al., 2016). However, these studies mainly use emotion to generate propositional statements in the form “A feels affection towards B” or “A confronts B”. At the same time, emotional behavior does not boil down to such propositional descriptions, as humans display complex and highly variable patterns in communicating their emotions, both verbally and non-verbally. In this paper, we analyze how emotions are expressed non-verbally in a corpus of fan fiction short stories. Our analysis shows that stories written by humans convey character emotions along various non-verbal channels. We find that some non-verbal channels, such as facial expressions and voice characteristics of the characters, are more strongly associated with joy, while gestures and body postures are more likely to occur with trust. Based on our analysis, we argue that automatic storytelling systems should take variability of emotion into account when generating descriptions of characters’ emotions. KW - Emotion Communication Channels ER - TY - CPAPER TI - Crowdsourcing and Validating Event-focused Emotion Corpora for German and English AU - Troiano, Enrica AU - Klinger, Roman AU - Padó, Sebastian PY - 2019 T2 - Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics SP - 4005-4011 PB - Association for Computational Linguistics DO - 10.18653/v1/P19-1391 UR - https://fis.uni-bamberg.de/handle/uniba/93919 LA - eng AB - Sentiment analysis has a range of corpora available across multiple languages. For emotion analysis, the situation is more limited, which hinders potential research on crosslingual modeling and the development of predictive models for other languages. In this paper, we fill this gap for German by constructing deISEAR, a corpus designed in analogy to the well-established English ISEAR emotion dataset. Motivated by Scherer’s appraisal theory, we implement a crowdsourcing experiment which consists of two steps. In step 1, participants create descriptions of emotional events for a given emotion. In step 2, five annotators assess the emotion expressed by the texts. We show that transferring an emotion classification model from the original English ISEAR to the German crowdsourced deISEAR via machine translation does not, on average, cause a performance drop. KW - Crowdsourcing ER - TY - CPAPER TI - Adversarial Training for Satire Detection : Controlling for Confounding Variables AU - McHardy, Robert AU - Adel, Heike AU - Klinger, Roman PY - 2019 T2 - Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies SP - 660-665 PB - Association for Computational Linguistics DO - 10.18653/v1/N19-1069 UR - https://fis.uni-bamberg.de/handle/uniba/93922 LA - eng AB - The automatic detection of satire vs. regular news is relevant for downstream applications (for instance, knowledge base population) and to improve the understanding of linguistic characteristics of satire. Recent approaches build upon corpora which have been labeled automatically based on article sources. We hypothesize that this encourages the models to learn characteristics for different publication sources (e.g., “The Onion” vs. “The Guardian”) rather than characteristics of satire, leading to poor generalization performance to unseen publication sources. We therefore propose a novel model for satire detection with an adversarial component to control for the confounding variable of publication source. On a large novel data set collected from German news (which we make available to the research community), we observe comparable satire classification performance and, as desired, a considerable drop in publication classification performance with adversarial training. Our analysis shows that the adversarial component is crucial for the model to learn to pay attention to linguistic properties of satire. KW - Satire Detection ER - TY - JOUR TI - Embedding Projection for Targeted Cross-Lingual Sentiment : Model Comparisons and a Real-World Study AU - Barnes, Jeremy AU - Klinger, Roman PY - 2019 JO - Journal of artificial intelligence research : JAIR VL - 66 SP - 691-742 PB - Morgan Kaufmann Publ. CY - San Francisco, Calif. SN - 1076-9757 DO - 10.1613/jair.1.11561 UR - https://fis.uni-bamberg.de/handle/uniba/93916 LA - eng ER - TY - JOUR TI - Learning soft domain constraints in a factor graph model for template-based information extraction AU - ter Horst, Hendrik AU - Cimiano, Philipp AU - Brazda, Nicole AU - Müller, Hans Werner AU - Klinger, Roman AU - Hartung, Matthias PY - 2019 JO - Data & Knowledge Engineering VL - 125 SP - 101764 PB - Elsevier CY - Amsterdam [u.a.] SN - 0169-023X DO - 10.1016/j.datak.2019.101764 UR - https://fis.uni-bamberg.de/handle/uniba/93918 LA - eng ER - TY - CPAPER TI - Towards Multimodal Emotion Recognition in German Speech Events in Cars using Transfer Learning AU - Cevher, Deniz AU - Zepf, Sebastian AU - Klinger, Roman PY - 2019 T2 - Proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019) SP - 79-90 PB - German Society for Computational Linguistics & Language Technology UR - https://fis.uni-bamberg.de/handle/uniba/93915 LA - eng AB - The recognition of emotions by humans is a complex process which considers multiple interacting signals such as facial expressions and both prosody and semantic content of utterances. Commonly, research on automatic recognition of emotions is, with few exceptions, limited to one modality. We describe an in-car experiment for emotion recognition from speech interactions for three modalities: the audio signal of a spoken interaction, the visual signal of the driver’s face, and the manually transcribed content of utterances of the driver. We use off-the-shelf tools for emotion detection in audio and face and compare that to a neural transfer learning approach for emotion recognition from text which utilizes existing resources from other domains. We see that transfer learning enables models based on out-of-domain corpora to per- form well. This method contributes up to 10 percentage points in F1, with up to 76 micro-average F1 across the emotions joy, annoyance and insecurity. Our findings also indicate that off-the-shelf-tools analyzing face and audio are not ready yet for emotion detection in in-car speech interactions without further adjustments. KW - Multimodal Emotion Recognition ER - TY - CPAPER TI - Exploring fine-tuned embeddings that model intensifiers for emotion analysis AU - Bostan, Laura AU - Klinger, Roman PY - 2019 T2 - Proceedings of the Tenth Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis SP - 25-34 PB - Association for Computational Linguistics CY - Minneapolis DO - 10.18653/v1/W19-1304 UR - https://fis.uni-bamberg.de/handle/uniba/93954 LA - eng AB - Adjective phrases like “a little bit surprised”, “completely shocked”, or “not stunned at all” are not handled properly by current state-of-the-art emotion classification and intensity prediction systems. Based on this finding, we analyze differences between embeddings used by these systems in regard to their capability of handling such cases and argue that intensifiers in context of emotion words need special treatment, as is established for sentiment polarity classification, but not for more fine-grained emotion prediction. To resolve this issue, we analyze different aspects of a post-processing pipeline which enriches the word representations of such phrases. This includes expansion of semantic spaces at the phrase level and sub-word level followed by retrofitting to emotion lexicons. We evaluate the impact of these steps with ‘A La Carte and Bag-of-Substrings extensions based on pretrained GloVe,Word2vec, and fastText embeddings against a crowd-sourced corpus of intensity annotations for tweets containing our focus phrases. We show that the fastText-based models do not gain from handling these specific phrases under inspection. For Word2vec embeddings, we show that our post-processing pipeline improves the results by up to 8% on a novel dataset densly populated with intensifiers while it does not decrease the performance on the established EmoInt dataset. KW - fine-tuned embeddings ER - TY - CPAPER TI - A Marker Passing Approach to Winograd Schemas AU - Fähndrich, Johannes AU - Weber, Sabine AU - Kanthak, Hannes PY - 2018 T2 - Semantic technology : 8th joint international conference, JIST 2018, Awaji, Japan, November 26–28, 2018, proceedings SP - 165-181 PB - Springer International Publishing CY - Cham, Switzerland SN - 9783030042837 DO - 10.1007/978-3-030-04284-4_12 UR - https://fis.uni-bamberg.de/handle/uniba/97230 LA - eng ER - TY - CPAPER TI - Different German and English Coreference Resolution Models for Multi-domain Content Curation Scenarios AU - Srivastava, Ankit AU - Weber, Sabine AU - Bourgonje, Peter AU - Rehm, Georg PY - 2018 T2 - Language Technologies for the Challenges of the Digital Age : 27th International Conference, GSCL 2017, Berlin, Germany, September 13-14, 2017, Proceedings SP - 48-61 PB - Springer International Publishing CY - Cham SN - 9783319737058 DO - 10.1007/978-3-319-73706-5_5 UR - https://fis.uni-bamberg.de/handle/uniba/97231 LA - eng AB - Coreference Resolution is the process of identifying all words and phrases in a text that refer to the same entity. It has proven to be a useful intermediary step for a number of natural language processing applications. In this paper, we describe three implementations for performing coreference resolution: rule-based, statistical, and projection-based (from English to German). After a comparative evaluation on benchmark datasets, we conclude with an application of these systems on German and English texts from different scenarios in digital curation such as an archive of personal letters, excerpts from a museum exhibition, and regional news articles. KW - Multi-domain Content Curation Scenarios ER - TY - CPAPER TI - DERE : A Task and Domain-Independent Slot Filling Framework for Declarative Relation Extraction AU - Adel, Heike AU - Papay, Sean AU - Klinger, Roman AU - Padó, Sebastian AU - Bostan, Laura Ana Maria PY - 2018 T2 - Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing : System Demonstrations SP - 42–47 PB - Association for Computational Linguistics DO - 10.18653/v1/D18-2008 UR - https://fis.uni-bamberg.de/handle/uniba/93960 LA - eng AB - Most machine learning systems for natural language processing are tailored to specific tasks. As a result, comparability of models across tasks is missing and their applicability to new tasks is limited. This affects end users without machine learning experience as well as model developers. To address these limitations, we present DERE, a novel framework for declarative specification and compilation of template-based information extraction. It uses a generic specification language for the task and for data annotations in terms of spans and frames. This formalism enables the representation of a large variety of natural language processing challenges. The backend can be instantiated by different models, following different paradigms. The clear separation of frame specification and model backend will ease the implementation of new models and the evaluation of different models across different tasks. Furthermore, it simplifies transfer learning, joint learning across tasks and/or domains as well as the assessment of model generalizability. DERE is available as open-source software. KW - DERE ER - TY - CPAPER TI - Who Feels What and Why? : Annotation of a Literature Corpus with Semantic Roles of Emotions AU - Kim, Evgeny AU - Klinger, Roman PY - 2018 T2 - Proceedings of the 27th International Conference on Computational Linguistics SP - 1345-1359 PB - Association for Computational Linguistics UR - https://fis.uni-bamberg.de/handle/uniba/93964 LA - eng AB - Most approaches to emotion analysis in fictional texts focus on detecting the emotion expressed in text. We argue that this is a simplification which leads to an overgeneralized interpretation of the results, as it does not take into account who experiences an emotion and why. Emotions play a crucial role in the interaction between characters and the events they are involved in. Until today, no specific corpora that capture such an interaction were available for literature. We aim at filling this gap and present a publicly available corpus based on Project Gutenberg, REMAN (Relational EMotion ANnotation), manually annotated for spans which correspond to emotion trigger phrases and entities/events in the roles of experiencers, targets, and causes of the emotion. We provide baseline results for the automatic prediction of these relational structures and show that emotion lexicons are not able to encompass the high variability of emotion expressions and demonstrate that statistical models benefit from joint modeling of emotions with its roles in all subtasks. The corpus that we provide enables future research on the recognition of emotions and associated entities in text. It supports qualitative literary studies and digital humanities. The corpus is available at http://www.ims.uni-stuttgart.de/data/reman . KW - Literature Corpus ER - TY - CPAPER TI - Projecting Embeddings for Domain Adaption : Joint Modeling of Sentiment Analysis in Diverse Domains AU - Barnes, Jeremy AU - Klinger, Roman AU - Schulte im Walde, Sabine PY - 2018 T2 - Proceedings of the 27th International Conference on Computational Linguistics SP - 818-830 PB - Association for Computational Linguistics UR - https://fis.uni-bamberg.de/handle/uniba/93965 LA - eng AB - Domain adaptation for sentiment analysis is challenging due to the fact that supervised classifiers are very sensitive to changes in domain. The two most prominent approaches to this problem are structural correspondence learning and autoencoders. However, they either require long training times or suffer greatly on highly divergent domains. Inspired by recent advances in cross-lingual sentiment analysis, we provide a novel perspective and cast the domain adaptation problem as an embedding projection task. Our model takes as input two mono-domain embedding spaces and learns to project them to a bi-domain space, which is jointly optimized to (1) project across domains and to (2) predict sentiment. We perform domain adaptation experiments on 20 source-target domain pairs for sentiment classification and report novel state-of-the-art results on 11 domain pairs, including the Amazon domain adaptation datasets and SemEval 2013 and 2016 datasets. Our analysis shows that our model performs comparably to state-of-the-art approaches on domains that are similar, while performing significantly better on highly divergent domains. Our code is available at https://github.com/jbarnesspain/domain_blse KW - Projecting Embeddings ER - TY - CPAPER TI - SANTO: A Web-based Annotation Tool for Ontology-driven Slot Filling AU - Hartung, Matthias AU - ter Horst, Hendrik AU - Grimm, Frank AU - Diekmann, Tim AU - Klinger, Roman AU - Cimiano, Philipp PY - 2018 T2 - Proceedings of ACL 2018, System Demonstrations SP - 68-73 PB - Association for Computational Linguistics DO - 10.18653/v1/P18-4012 UR - https://fis.uni-bamberg.de/handle/uniba/93966 LA - eng ER - TY - CPAPER TI - On the Semantic Similarity of Disease Mentions in MEDLINE and Twitter AU - Thorne, Camilo AU - Klinger, Roman PY - 2018 T2 - Natural Language Processing and Information Systems : 23rd International Conference on Applications of Natural Language to Information Systems, NLDB 2018, Paris, France, June 13-15, 2018, Proceedings SP - 324–332 PB - Springer International Publishing CY - Cham SN - 978-3-319-91946-1 DO - 10.1007/978-3-319-91947-8_34 UR - https://fis.uni-bamberg.de/handle/uniba/93970 LA - eng ER - TY - CPAPER TI - A Reporting Tool for Relational Visualization and Analysis of Character Mentions in Literature AU - Barth, Florian AU - Kim, Evgeny AU - Murr, Sandra AU - Klinger, Roman PY - 2018 T2 - Kritik der digitalen Vernunft : Konferenzabstracts : Universität zu Köln, 26. Februar bis 2. März 2018 SN - 978-3-946275-02-2 UR - https://fis.uni-bamberg.de/handle/uniba/93971 LA - eng ER - TY - CPAPER TI - Assessing the Impact of Single and Pairwise Slot Constraints in a Factor Graph Model for Template-Based Information Extraction AU - ter Horst, Hendrik AU - Hartung, Matthias AU - Klinger, Roman AU - Brazda, Nicole AU - Müller, Hans Werner AU - Cimiano, Philipp PY - 2018 T2 - Natural Language Processing and Information Systems : 23rd International Conference on Applications of Natural Language to Information Systems, NLDB 2018, Paris, France, June 13-15, 2018, Proceedings SP - 179-190 PB - Springer International Publishing CY - Cham T3 - Lecture Notes in Computer Science SN - 978-3-319-91947-8 DO - 10.1007/978-3-319-91947-8_18 UR - https://fis.uni-bamberg.de/handle/uniba/93969 LA - eng ER - TY - CPAPER TI - An Empirical Analysis of the Role of Amplifiers, Downtoners, and Negations in Emotion Classification in Microblogs AU - Strohm, Florian AU - Klinger, Roman PY - 2018 T2 - 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) PB - IEEE CY - Piscataway, NJ SN - 978-1-5386-5090-5 DO - 10.1109/DSAA.2018.00087 UR - https://fis.uni-bamberg.de/handle/uniba/93961 LA - eng ER - TY - JOUR TI - What You Use, Not What You Do : Automatic Classification and Similarity Detection of Recipes AU - Kicherer, Hanna AU - Dittrich, Marcel AU - Grebe, Lukas AU - Scheible, Christian AU - Klinger, Roman PY - 2018 JO - Data and Knowledge Engineering VL - 117 IS - September SP - 252-263 PB - Elsevier CY - Amsterdam [u.a.] SN - 1872-6933 DO - 10.1016/j.datak.2018.04.004 UR - https://fis.uni-bamberg.de/handle/uniba/93968 LA - eng ER - TY - CPAPER TI - An Analysis of Annotated Corpora for Emotion Classification in Text AU - Bostan, Laura Ana Maria AU - Klinger, Roman PY - 2018 T2 - Proceedings of the 27th International Conference on Computational Linguistics SP - 2104-2119 PB - Association for Computational Linguistics CY - Santa Fe, New Mexico UR - https://fis.uni-bamberg.de/handle/uniba/93962 LA - eng AB - Several datasets have been annotated and published for classification of emotions. They differ in several ways: (1) the use of different annotation schemata (e. g., discrete label sets, including joy, anger, fear, or sadness or continuous values including valence, or arousal), (2) the domain, and, (3) the file formats. This leads to several research gaps: supervised models often only use a limited set of available resources. Additionally, no previous work has compared emotion corpora in a systematic manner. We aim at contributing to this situation with a survey of the datasets, and aggregate them in a common file format with a common annotation schema. Based on this aggregation, we perform the first cross-corpus classification experiments in the spirit of future research enabled by this paper, in order to gain insight and a better understanding of differences of models inferred from the data. This work also simplifies the choice of the most appropriate resources for developing a model for a novel domain. One result from our analysis is that a subset of corpora is better classified with models trained on a different corpus. For none of the corpora, training on all data altogether is better than using a subselection of the resources. Our unified corpus is available at http://www.ims.uni-stuttgart.de/data/unifyemotion. KW - Emotion Classification ER - TY - CPAPER TI - Bilingual Sentiment Embeddings : Joint Projection of Sentiment Across Languages AU - Barnes, Jeremy AU - Klinger, Roman AU - Schulte im Walde, Sabine PY - 2018 T2 - Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) SP - 2483-2493 PB - Association for Computational Linguistics CY - Melbourne DO - 10.18653/v1/P18-1231 UR - https://fis.uni-bamberg.de/handle/uniba/93967 LA - eng AB - Sentiment analysis in low-resource languages suffers from a lack of annotated corpora to estimate high-performing models. Machine translation and bilingual word embeddings provide some relief through cross-lingual sentiment approaches. However, they either require large amounts of parallel data or do not sufficiently capture sentiment information. We introduce Bilingual Sentiment Embeddings (BLSE), which jointly represent sentiment information in a source and target language. This model only requires a small bilingual lexicon, a source-language corpus annotated for sentiment, and monolingual word embeddings for each language. We perform experiments on three language combinations (Spanish, Catalan, Basque) for sentence-level cross-lingual sentiment classification and find that our model significantly outperforms state-of-the-art methods on four out of six experimental setups, as well as capturing complementary information to machine translation. Our analysis of the resulting embedding space provides evidence that it represents sentiment information in the resource-poor target language without any annotated data in that language. KW - Bilingual Sentiment Embeddings ER - TY - CPAPER TI - IEST : WASSA-2018 Implicit Emotions Shared Task AU - Klinger, Roman AU - De Clercq, Orphée AU - Mohammad, Saif M. AU - Balahur, Alexandra PY - 2018 T2 - Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis SP - 31-42 PB - Association for Computational Linguistics CY - Brüssel DO - 10.18653/v1/W18-6206 UR - https://fis.uni-bamberg.de/handle/uniba/93958 LA - eng AB - Past shared tasks on emotions use data with both overt expressions of emotions (I am so happy to see you!) as well as subtle expressions where the emotions have to be inferred, for instance from event descriptions. Further, most datasets do not focus on the cause or the stimulus of the emotion. Here, for the first time, we propose a shared task where systems have to predict the emotions in a large automatically labeled dataset of tweets without access to words denoting emotions. Based on this intention, we call this the Implicit Emotion Shared Task (IEST) because the systems have to infer the emotion mostly from the context. Every tweet has an occurrence of an explicit emotion word that is masked. The tweets are collected in a manner such that they are likely to include a description of the cause of the emotion – the stimulus. Altogether, 30 teams submitted results which range from macro F1 scores of 21 % to 71 %. The baseline (Max-Ent bag of words and bigrams) obtains an F1 score of 60 % which was available to the participants during the development phase. A study with human annotators suggests that automatic methods outperform human predictions, possibly by honing into subtle textual clues not used by humans. Corpora, resources, and results are available at the shared task website at http://implicitemotions.wassa2018.com. KW - WASSA-2018 ER - TY - CPAPER TI - Teaching Computational Aspects in the Digital Humanities Program at University of Stuttgart : Intentions and Experiences AU - Reiter, Nils AU - Schulz, Sarah AU - Kremer, Gerhard AU - Klinger, Roman AU - Viehhauser, Gabriel AU - Kuhn, Jonas PY - 2017 T2 - Proceedings of the Workshop on Teaching NLP for Digital Humanities (Teach4DH 2017) co-located with GSCL 2017 SP - 43-48 T3 - CEUR Workshop Proceedings UR - https://fis.uni-bamberg.de/handle/uniba/93973 LA - eng ER - TY - CPAPER TI - Towards Confidence Estimation for typed Protein-Protein Relation Extraction AU - Thorne, Camilo AU - Klinger, Roman PY - 2017 T2 - Proceedings of the Biomedical NLP Workshop associated with RANLP 2017 SP - 55–63 PB - Association for Computational Linguistics DO - 10.26615/978-954-452-044-1_008 UR - https://fis.uni-bamberg.de/handle/uniba/93974 LA - eng ER - TY - CPAPER TI - Ranking Right-Wing Extremist Social Media Profiles by Similarity to Democratic and Extremist Groups AU - Hartung, Matthias AU - Klinger, Roman AU - Schmidtke, Franziska AU - Vogel, Lars PY - 2017 T2 - Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis SP - 24–33 PB - Association for Computational Linguistics DO - 10.18653/v1/W17-5204 UR - https://fis.uni-bamberg.de/handle/uniba/93975 LA - eng AB - Social media are used by an increasing number of political actors. A small subset of these is interested in pursuing extremist motives such as mobilization, recruiting or radicalization activities. In order to counteract these trends, online providers and state institutions reinforce their monitoring efforts, mostly relying on manual workflows. We propose a machine learning approach to support manual attempts towards identifying right-wing extremist content in German Twitter profiles. Based on a fine-grained conceptualization of right-wing extremism, we frame the task as ranking each individual profile on a continuum spanning different degrees of right-wing extremism, based on a nearest neighbour approach. A quantitative evaluation reveals that our ranking model yields robust performance (up to 0.81 F1 score) when being used for predicting discrete class labels. At the same time, the model provides plausible continuous ranking scores for a small sample of borderline cases at the division of right-wing extremism and New Right political movements. KW - Social Media Profiles ER - TY - CPAPER TI - Assessing State-of-the-Art Sentiment Models on State-of-the-Art Sentiment Datasets AU - Barnes, Jeremy AU - Klinger, Roman AU - Schulte im Walde, Sabine PY - 2017 T2 - Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis SP - 2–12 PB - Association for Computational Linguistics DO - 10.18653/v1/W17-5202 UR - https://fis.uni-bamberg.de/handle/uniba/93977 LA - eng AB - There has been a good amount of progress in sentiment analysis over the past 10 years, including the proposal of new methods and the creation of benchmark datasets. In some papers, however, there is a tendency to compare models only on one or two datasets, either because of time restraints or because the model is tailored to a specific task. Accordingly, it is hard to understand how well a certain model generalizes across different tasks and datasets. In this paper, we contribute to this situation by comparing several models on six different benchmarks, which belong to different domains and additionally have different levels of granularity (binary, 3-class, 4-class and 5-class). We show that Bi-LSTMs perform well across datasets and that both LSTMs and Bi-LSTMs are particularly good at fine-grained sentiment tasks (i.e., with more than two classes). Incorporating sentiment information into word embeddings during training gives good results for datasets that are lexically similar to the training data. With our experiments, we contribute to a better understanding of the performance of different model architectures on different data sets. Consequently, we detect novel state-of-the-art results on the SenTube datasets. KW - State-of-the-Art Sentiment Models ER - TY - CPAPER TI - IMS at EmoInt-2017 : Emotion Intensity Prediction with Affective Norms, Automatically Extended Resources and Deep Learning AU - Köper, Maximilian AU - Kim, Evgeny AU - Klinger, Roman PY - 2017 T2 - Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis SP - 50-57 PB - Association for Computational Linguistics DO - 10.18653/v1/W17-5206 UR - https://fis.uni-bamberg.de/handle/uniba/93978 LA - eng AB - Our submission to the WASSA-2017 shared task on the prediction of emotion intensity in tweets is a supervised learning method with extended lexicons of affective norms. We combine three main information sources in a random forrest regressor, namely (1), manually created resources, (2) automatically extended lexicons, and (3) the output of a neural network (CNN-LSTM) for sentence regression. All three feature sets perform similarly well in isolation (≈ .67 macro average Pearson correlation). The combination achieves .72 on the official test set (ranked 2nd out of 22 participants). Our analysis reveals that performance is increased by providing cross-emotional intensity predictions. The automatic extension of lexicon features benefit from domain specific embeddings. Complementary ratings for affective norms increase the impact of lexicon features. Our resources (ratings for 1.6 million twitter specific words) and our implementation is publicly available at http://www.ims.uni-stuttgart.de/data/ims_emoint. KW - Emotion Intensity Prediction ER - TY - CPAPER TI - Investigating the Relationship between Literary Genres and Emotional Plot Development AU - Kim, Evgeny AU - Padó, Sebastian AU - Klinger, Roman PY - 2017 T2 - Proceedings of the Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature SP - 17-26 PB - Association for Computational Linguistics DO - 10.18653/v1/W17-2203 UR - https://fis.uni-bamberg.de/handle/uniba/93979 LA - eng AB - Literary genres are commonly viewed as being defined in terms of content and stylistic features. In this paper, we focus on one particular class of lexical features, namely emotion information, and investigate the hypothesis that emotion-related information correlates with particular genres. Using genre classification as a testbed, we compare a model that computes lexicon-based emotion scores globally for complete stories with a model that tracks emotion arcs through stories on a subset of Project Gutenberg with five genres. Our main findings are: (a), the global emotion model is competitive with a large-vocabulary bag-of-words genre classifier (80%F1); (b), the emotion arc model shows a lower performance (59 % F1) but shows complementary behavior to the global model, as indicated by a very good performance of an oracle model (94 % F1) and an improved performance of an ensemble model (84 % F1); (c), genres differ in the extent to which stories follow the same emotional arcs, with particularly uniform behavior for anger (mystery) and fear (adventures, romance, humor, science fiction). KW - Literary Genres KW - Emotional Plot Development ER - TY - CPAPER TI - Identifying Right-Wing Extremism in German Twitter Profiles: A Classification Approach AU - Hartung, Matthias AU - Klinger, Roman AU - Schmidtke, Franziska AU - Vogel, Lars PY - 2017 T2 - Natural Language Processing and Information Systems : 22nd International Conference on Applications of Natural Language to Information Systems, NLDB 2017, Liège, Belgium, June 21-23, 2017, Proceedings SP - 320-325 PB - Springer International Publishing CY - Cham SN - 978-3-319-59569-6 DO - 10.1007/978-3-319-59569-6_40 UR - https://fis.uni-bamberg.de/handle/uniba/93981 LA - eng ER - TY - CPAPER TI - Fine-Grained Opinion Mining from Mobile App Reviews with Word Embedding Features AU - Sänger, Mario AU - Leser, Ulf AU - Klinger, Roman PY - 2017 T2 - Natural Language Processing and Information Systems : 22nd International Conference on Applications of Natural Language to Information Systems, NLDB 2017, Liège, Belgium, June 21-23, 2017, Proceedings SP - 3-14 PB - Springer International Publishing CY - Cham SN - 978-3-319-59569-6 DO - 10.1007/978-3-319-59569-6_1 UR - https://fis.uni-bamberg.de/handle/uniba/93982 LA - eng ER - TY - CPAPER TI - Does Optical Character Recognition and Caption Generation Improve Emotion Detection in Microblog Posts? AU - Klinger, Roman PY - 2017 T2 - Natural Language Processing and Information Systems : 22nd International Conference on Applications of Natural Language to Information Systems, NLDB 2017, Liège, Belgium, June 21-23, 2017, Proceedings SP - 313-319 PB - Springer International Publishing CY - Cham SN - 978-3-319-59569-6 DO - 10.1007/978-3-319-59569-6_39 UR - https://fis.uni-bamberg.de/handle/uniba/93983 LA - eng ER - TY - CPAPER TI - What You Use, Not What You Do: Automatic Classification of Recipes AU - Kicherer, Hanna AU - Dittrich, Marcel AU - Grebe, Lukas AU - Scheible, Christian AU - Klinger, Roman PY - 2017 T2 - Natural Language Processing and Information Systems : 22nd International Conference on Applications of Natural Language to Information Systems, NLDB 2017, Liège, Belgium, June 21-23, 2017, Proceedings SP - 197-209 PB - Springer International Publishing CY - Cham SN - 978-3-319-59569-6 DO - 10.1007/978-3-319-59569-6_22 UR - https://fis.uni-bamberg.de/handle/uniba/93984 LA - eng ER - TY - CPAPER TI - Automatic Emotion Detection for Quantitative Literary Studies : A case study based on Franz Kafka's “Das Schloss” and “Amerika” AU - Klinger, Roman AU - Suliya, Surayya Samat AU - Reiter, Nils PY - 2016 T2 - Digital Humanities 2016 : Conference Abstracts SP - 826-828 PB - Jagiellonian University and Pedagogical University UR - https://fis.uni-bamberg.de/handle/uniba/94011 LA - eng ER - TY - CPAPER TI - Model Architectures for Quotation Detection AU - Scheible, Christian AU - Klinger, Roman AU - Padó, Sebastian PY - 2016 T2 - Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics SP - 1736-1745 PB - Association for Computational Linguistics DO - 10.18653/v1/P16-1164 UR - https://fis.uni-bamberg.de/handle/uniba/93925 LA - eng AB - Quotation detection is the task of locating spans of quoted speech in text. The state of the art treats this problem as a sequence labeling task and employs linear-chain conditional random fields. We question the efficacy of this choice: The Markov assumption in the model prohibits it from making joint decisions about the begin, end, and internal context of a quotation. We perform an extensive analysis with two new model architectures. We find that (a), simple boundary classification combined with a greedy prediction strategy is competitive with the state of the art; (b), a semi-Markov model significantly outperforms all others, by relaxing the Markov assumption. KW - Model Architectures ER - TY - CPAPER TI - SCARE : the Sentiment Corpus of App Reviews with Fine-grained Annotations in German AU - Sänger, Mario AU - Kemmerer, Steffen AU - Adolphs, Peter AU - Klinger, Roman AU - Leser, Ulf PY - 2016 T2 - Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16) SP - 1114–1121 PB - European Language Resources Association (ELRA) SN - 978-2-9517408-9-1 UR - https://fis.uni-bamberg.de/handle/uniba/94064 LA - eng AB - The automatic analysis of texts containing opinions of users about, e.g., products or political views has gained attention within the last decades. However, previous work on the task of analyzing user reviews about mobile applications in app stores is limited. Publicly available corpora do not exist, such that a comparison of different methods and models is difficult. We fill this gap by contributing the Sentiment Corpus of App Reviews (SCARE), which contains fine-grained annotations of application aspects, subjective (evaluative) phrases and relations between both. This corpus consists of 1,760 annotated application reviews from the Google Play Store with 2,487 aspects and 3,959 subjective phrases. We describe the process and methodology how the corpus was created. The Fleiss Kappa between four annotators reveals an agreement of 0.72. We provide a strong baseline with a linear-chain conditional random field and word-embedding features with a performance of 0.62 for aspect detection and 0.63 for the extraction of subjective phrases. The corpus is available to the research community to support the development of sentiment analysis methods on mobile application reviews. KW - Sentiment Corpus of App Reviews ER - TY - CPAPER TI - An Empirical, Quantitative Analysis of the Differences Between Sarcasm and Irony AU - Ling, Jennifer AU - Klinger, Roman PY - 2016 T2 - The Semantic Web : ESWC 2016 Satellite Events, Heraklion, Crete, Greece, May 29 – June 2, 2016. Revised Selected Papers SP - 203-216 PB - Springer International Publishing CY - Cham SN - 978-3-319-47602-5 DO - 10.1007/978-3-319-47602-5_39 UR - https://fis.uni-bamberg.de/handle/uniba/94017 LA - eng ER - TY - CPAPER TI - Towards Opinion Mining from Reviews for the Prediction of Product Rankings AU - Kessler, Wiltrud AU - Klinger, Roman AU - Kuhn, Jonas PY - 2015 T2 - Proceedings of the 6th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis SP - 51-57 PB - Association for Computational Linguistics CY - Lisboa, Portugal DO - 10.18653/v1/W15-2908 UR - https://fis.uni-bamberg.de/handle/uniba/93997 LA - eng AB - Opinion mining aims at summarizing the content of reviews for a specific brand, product, or manufacturer. However, the actual desire of a user is often one step further: Produce a ranking corresponding to specific needs such that a selection process is supported. In this work, we aim towards closing this gap. We present the task to rank products based on sentiment information and discuss necessary steps towards addressing this task. This includes, on the one hand, the identification of gold rankings as a fundament for an objective function and evaluation and, on the other hand, methods to rank products based on review information. To demonstrate early results on that task, we employ real world examples of rankings as gold standard that are of interest to potential customers as well as product managers, in our case the sales ranking provided by Amazon.com and the quality ranking by Snapsort.com. As baseline methods, we use the average star ratings and review frequencies. Our best text-based approximation of the sales ranking achieves a Spearman’s correlation coefficient of ρ = 0.23. On the Snapsort data, a ranking based on extracting comparisons leads to ρ = 0.51. In addition, we show that aspect-specific rankings can be used to measure the impact of specific aspects on the ranking. KW - Opinion Mining ER - TY - CPAPER TI - Instance Selection Improves Cross-Lingual Model Training for Fine-Grained Sentiment Analysis AU - Klinger, Roman AU - Cimiano, Philipp PY - 2015 T2 - Proceedings of the Nineteenth Conference on Computational Natural Language Learning SP - 153-163 PB - Association for Computational Linguistics SN - 978-1-941643-77-8 DO - 10.18653/v1/K15-1016 UR - https://fis.uni-bamberg.de/handle/uniba/94001 LA - eng AB - Scarcity of annotated corpora for many languages is a bottleneck for training finegrained sentiment analysis models that can tag aspects and subjective phrases. We propose to exploit statistical machine translation to alleviate the need for training data by projecting annotated data in a source language to a target language such that a supervised fine-grained sentiment analysis system can be trained. To avoid a negative influence of poor-quality translations, we propose a filtering approach based on machine translation quality estimation measures to select only high-quality sentence pairs for projection. We evaluate on the language pair German/English on a corpus of product reviews annotated for both languages and compare to in-target-language training. Projection without any filtering leads to 23 % F1 in the task of detecting aspect phrases, compared to 41 % F1 for in-target-language training. Our approach obtains up to 47 % F1. Further, we show that the detection of subjective phrases is competitive to in-target-language training without filtering. KW - Instance Selection ER - TY - JOUR TI - Detecting miRNA Mentions and Relations in Biomedical Literature [version 3; peer review: 2 approved, 1 approved with reservations] AU - Bagewadi, Shweta AU - Bobić, Tamara AU - Hofmann-Apitius, Martin AU - Fluck, Juliane AU - Klinger, Roman PY - 2015 JO - F1000Research VL - 3 IS - 205 SP - 1-36 PB - F1000 Research Ltd CY - London SN - 2046-1402 DO - 10.12688/f1000research.4591.3 UR - https://fis.uni-bamberg.de/handle/uniba/93988 LA - eng AB - Introduction: MicroRNAs (miRNAs) have demonstrated their potential as post-transcriptional gene expression regulators, participating in a wide spectrum of regulatory events such as apoptosis, differentiation, and stress response. Apart from the role of miRNAs in normal physiology, their dysregulation is implicated in a vast array of diseases. Dissection of miRNA-related associations are valuable for contemplating their mechanism in diseases, leading to the discovery of novel miRNAs for disease prognosis, diagnosis, and therapy. Motivation: Apart from databases and prediction tools, miRNA-related information is largely available as unstructured text. Manual retrieval of these associations can be labor-intensive due to steadily growing number of publications. Additionally, most of the published miRNA entity recognition methods are keyword based, further subjected to manual inspection for retrieval of relations. Despite the fact that several databases host miRNA-associations derived from text, lower sensitivity and lack of published details for miRNA entity recognition and associated relations identification has motivated the need for developing comprehensive methods that are freely available for the scientific community. Additionally, the lack of a standard corpus for miRNA-relations has caused difficulty in evaluating the available systems. We propose methods to automatically extract mentions of miRNAs, species, genes/proteins, disease, and relations from scientific literature. Our generated corpora, along with dictionaries, and miRNA regular expression are freely available for academic purposes. To our knowledge, these resources are the most comprehensive developed so far. Results: The identification of specific miRNA mentions reaches a recall of 0.94 and precision of 0.93. Extraction of miRNA-disease and miRNA-gene relations lead to an F1 score of up to 0.76. A comparison of the information extracted by our approach to the databases miR2Disease and miRSel for the extraction of Alzheimer's disease related relations shows the capability of our proposed methods in identifying correct relations with improved sensitivity. The published resources and described methods can help the researchers for maximal retrieval of miRNA-relations and generation of miRNA-regulatory networks. Availability: The training and test corpora, annotation guidelines, developed dictionaries, and supplementary files are available at http://www.scai.fraunhofer.de/mirna-corpora.html KW - MicroRNAs KW - prediction algorithms KW - corpus ER - TY - CPAPER TI - An Impact Analysis of Features in a Classification Approach to Irony Detection in Product Reviews AU - Buschmeier, Konstantin AU - Klinger, Roman AU - Cimiano, Philipp PY - 2014 T2 - Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis SP - 42-49 PB - Association for Computational Linguistics CY - Baltimore, Maryland DO - 10.3115/v1/W14-2608 UR - https://fis.uni-bamberg.de/handle/uniba/93991 LA - eng AB - Irony is an important device in human communication, both in everyday spoken conversations as well as in written texts including books, websites, chats, reviews, and Twitter messages among others. Specific cases of irony and sarcasm have been studied in different contexts but, to the best of our knowledge, only recently the first publicly available corpus including annotations about whether a text is ironic or not has been published by Filatova (2012). How- ever, no baseline for classification of ironic or sarcastic reviews has been provided. With this paper, we aim at closing this gap. We formulate the problem as a supervised classification task and evaluate different classifiers, reaching an F1-measure of up to 74 % using logistic regression. We analyze the impact of a number of features which have been proposed in previous research as well as combinations of them. KW - Irony Detection ER - TY - CPAPER TI - Towards Gene Recognition from Rare and Ambiguous Abbreviations using a Filtering Approach AU - Hartung, Matthias AU - Klinger, Roman AU - Zwick, Matthias AU - Cimiano, Philipp PY - 2014 T2 - Proceedings of BioNLP 2014 SP - 118-127 PB - Association for Computational Linguistics CY - Baltimore, Maryland DO - 10.3115/v1/W14-3418 UR - https://fis.uni-bamberg.de/handle/uniba/93994 LA - eng AB - Retrieving information about highly ambiguous gene/protein homonyms is a challenge, in particular where their non-protein meanings are more frequent than their protein meaning (e. g., SAH or HF). Due to their limited coverage in common benchmarking data sets, the performance of existing gene/protein recognition tools on these problematic cases is hard to assess. We uniformly sample a corpus of eight ambiguous gene/protein abbreviations from MEDLINEr and provide manual annotations for each mention of these abbreviations.1 Based on this resource, we show that available gene recognition tools such as conditional random fields (CRF) trained on BioCreative 2 NER data or GNAT tend to underperform on this phenomenon. We propose to extend existing gene recognition approaches by combining a CRF and a support vector machine. In a cross- entity evaluation and without taking any entity-specific information into account, our model achieves a gain of 6 points F1-Measure over our best baseline which checks for the occurrence of a long form of the abbreviation and more than 9 points over all existing tools investigated. KW - Gene Recognition ER - TY - CPAPER TI - The USAGE review corpus for fine grained multi lingual opinion analysis AU - Klinger, Roman AU - Cimiano, Philipp PY - 2014 T2 - Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14) SP - 2211-2218 PB - European Language Resources Association (ELRA) CY - Reykjavik, Iceland SN - 978-2-9517408-8-4 UR - https://fis.uni-bamberg.de/handle/uniba/94000 LA - eng AB - Opinion mining has received wide attention in recent years. Models for this task are typically trained or evaluated with a manually annotated dataset. However, fine-grained annotation of sentiments including information about aspects and their evaluation is very labour-intensive. The data available so far is limited. Contributing to this situation, this paper describes the Bielefeld University Sentiment Analysis Corpus for German and English (USAGE), which we offer freely to the community and which contains the annotation of product reviews from Amazon with both aspects and subjective phrases. It provides information on segments in the text which denote an aspect or a subjective evaluative phrase which refers to the aspect. Relations and coreferences are explicitly annotated. This dataset contains 622 English and 611 German reviews, allowing to investigate how to port sentiment analysis systems across languages and domains. We describe the methodology how the corpus was created and provide statistics including inter-annotator agreement. We further provide figures for a baseline system and results for German and English as well as in a cross-domain setting. The results are encouraging in that they show that aspects and phrases can be extracted robustly without the need of tuning to a particular type of products. KW - sentiment analysis KW - corpus KW - product reviews ER - TY - CPAPER TI - Ontology-based Extraction of Structured Information from Publications on Preclinical Experiments for Spinal Cord Injury Treatments AU - Paaßen, Benjamin AU - Stöckel, Andreas AU - Dickfelder, Raphael AU - Göpfert, Jan Philip AU - Brazda, Nicole AU - Kirchhoffer, Tarek AU - Müller, Hans Werner AU - Klinger, Roman AU - Cimiano, Philipp AU - Hartung, Matthias PY - 2014 T2 - Proceedings of the Third Workshop on Semantic Web and Information Extraction SP - 25-32 UR - https://fis.uni-bamberg.de/handle/uniba/94051 LA - eng ER - TY - CPAPER TI - Joint and Pipeline Probabilistic Models for Fine-Grained Sentiment Analysis: Extracting Aspects, Subjective Phrases and their Relations AU - Klinger, Roman AU - Cimiano, Philipp PY - 2013 T2 - ICDMW '13: Proceedings of the 2013 IEEE 13th International Conference on Data Mining Workshops SP - 937-944 PB - IEEE SN - 978-1-4799-3142-2 DO - 10.1109/ICDMW.2013.13 UR - https://fis.uni-bamberg.de/handle/uniba/94003 LA - eng ER - TY - CPAPER TI - Bi-directional Inter-dependencies of Subjective Expressions and Targets and their Value for a Joint Model AU - Klinger, Roman AU - Cimiano, Philipp PY - 2013 T2 - Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics SP - 848-854 PB - Association for Computational Linguistics CY - Sofia, Bulgaria UR - https://fis.uni-bamberg.de/handle/uniba/94004 LA - eng AB - Opinion mining is often regarded as a classification or segmentation task, involving the prediction of i) subjective expressions, ii) their target and iii) their polarity. Intuitively, these three variables are bidirectionally interdependent, but most work has either attempted to predict them in isolation or proposing pipeline-based approaches that cannot model the bidirectional interaction between these variables. Towards better understanding the interaction between these variables, we propose a model that allows for analyzing the relation of target and subjective phrases in both directions, thus providing an upper bound for the impact of a joint model in comparison to a pipeline model. We report results on two public datasets (cameras and cars), showing that our model outperforms state-of-the-art models, as well as on a new dataset consisting of Twitter posts KW - Joint Model ER - TY - JOUR TI - Committee-based Selection of Weakly Labeled Instances for Learning Relation Extraction AU - Bobic, Tamara AU - Klinger, Roman PY - 2013 JO - Research in Computing Science VL - 70 SP - 187-197 UR - https://fis.uni-bamberg.de/handle/uniba/93989 LA - eng ER - TY - CPAPER TI - Orthonormal Explicit Topic Analysis for Cross-Lingual Document Matching AU - McCrae, John Philip AU - Cimiano, Philipp AU - Klinger, Roman PY - 2013 T2 - Proceedings of the 2013 Conference on Empirical Natural Language Processing SP - 1732-1740 PB - Association for Computational Linguistics CY - Seattle, Washington UR - https://fis.uni-bamberg.de/handle/uniba/94019 LA - eng AB - Cross-lingual topic modelling has applications in machine translation, word sense disambiguation and terminology alignment. Multilingual extensions of approaches based on latent (LSI), generative (LDA, PLSI) as well as explicit (ESA) topic modelling can induce an interlingual topic space allowing documents in different languages to be mapped into the same space and thus to be compared across languages. In this paper, we present a novel approach that combines latent and explicit topic modelling approaches in the sense that it builds on a set of explicitly defined topics, but then computes latent relations between these. Thus, the method combines the benefits of both explicit and latent topic modelling approaches. We show that on a cross-lingual mate retrieval task, our model significantly outperforms LDA, LSI, and ESA, as well as a baseline that translates every word in a document into the target language. KW - Cross-Lingual Document Matching ER - TY - CPAPER TI - E-Government and Policy Simulation in Intelligent Virtual Environments AU - Aisopos, Fotis AU - Kardara, Magdalini AU - Klinger, Roman AU - Papaoikonomou, Athanasios AU - Tserpes, Konstantinos AU - Gardner, Michael AU - Varvarigou, Theodora A. AU - Senger, Philipp PY - 2012 T2 - Proceedings of the 8th International Conference on Web Information Systems and Technologies : Porto, Portugal, 18 - 21 April, 2012 SP - 129-135 PB - SciTePress SN - 978-989-8565-08-2 UR - https://fis.uni-bamberg.de/handle/uniba/93986 LA - eng ER - TY - CPAPER TI - Improving Distantly Supervised Extraction of Drug-Drug and Protein-Protein Interactions AU - Bobić, Tamara AU - Klinger, Roman AU - Thomas, Philippe AU - Hofmann-Apitius, Martin PY - 2012 T2 - Proceedings of the Joint Workshop on Unsupervised and Semi-Supervised Learning in NLP SP - 35-43 PB - Association for Computational Linguistics UR - https://fis.uni-bamberg.de/handle/uniba/93990 LA - eng ER - TY - CPAPER TI - Online Communities Support Policy-Making : The Need for Data Analysis AU - Klinger, Roman AU - Senger, Philipp AU - Madan, Sumit AU - Jacovi, Michal PY - 2012 T2 - Electronic participation : 4th IFIP WG 8.5 international conference ; proceedings / EPart 2012, Kristiansand, Norway, September 3 - 5, 2012 SP - 132-143 PB - Springer CY - Berlin ; Heidelberg T3 - Lecture Notes in Computer Science SN - 978-3-642-33249-4 DO - 10.1007/978-3-642-33250-0_12 UR - https://fis.uni-bamberg.de/handle/uniba/94010 LA - eng ER - TY - CPAPER TI - Learning to Extract Protein-Protein Interactions using Distant Supervision AU - Thomas, Philippe AU - Solt, Illés AU - Klinger, Roman AU - Leser, Ulf PY - 2011 SP - 25-32 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/96475 LA - eng AB - Most relation extraction methods, especially in the domain of biology, rely on machine learning methods to classify a cooccurring pair of entities in a sentence to be related or not. Such an approach requires a training corpus, which involves expert annotation and is tedious, time-consuming, and expensive. We overcome this problem by the use of existing knowledge in structured databases to automatically generate a training corpus for protein-protein interactions. An extensive evaluation of different instance selection strategies is performed to maximize robustness on this presumably noisy resource. Successful strategies to consistently improve performance include a majority voting ensemble of classifiers trained on subsets of the training corpus and the use of knowledge bases consisting of proven non-interactions. Our best configured model built without manually annotated data shows very competitive results on several publicly available benchmark corpora KW - Protein-Protein Interactions ER - TY - THES TI - Conditional Random Fields for Named Entity Recognition - Feature Selection and Optimization in Biology and Chemistry AU - Klinger, Roman PY - 2011 PB - Shaker CY - Aachen, Germany UR - https://fis.uni-bamberg.de/handle/uniba/93999 LA - eng ER - TY - CPAPER TI - Inter-Event Dependencies support Event Extraction from Biomedical Literature AU - Klinger, Roman AU - Riedel, Sebastian AU - McCallum, Andrew PY - 2011 DO - 10.24406/publica-fhg-374306 UR - https://fis.uni-bamberg.de/handle/uniba/94009 LA - eng ER - TY - JOUR TI - Challenges in the association of human single nucleotide polymorphism mentions with unique database identifiers AU - Thomas, Philippe E. AU - Klinger, Roman AU - Furlong, Laura I. AU - Hofmann-Apitius, Martin AU - Friedrich, Christoph M. PY - 2011 JO - BMC Bioinformatics VL - 12 IS - S4, Proceedings of the European Conference on Computational Biology (ECCB) 2010 Workshop: Annotation, interpretation and management of mutations (AIMM) SP - 1-18 PB - Springer CY - Berlin ; Heidelberg SN - 1471-2105 DO - 10.1186/1471-2105-12-S4-S4 UR - https://fis.uni-bamberg.de/handle/uniba/93923 LA - eng ER - TY - CPAPER TI - Learning to Extract Protein-Protein Interactions using Distant Supervision AU - Thomas, Philippe AU - Solt, Illés AU - Klinger, Roman AU - Leser, Ulf PY - 2011 T2 - Proceedings of Workshop on Robust Unsupervised and Semisupervised Methods in Natural Language Processing SP - 25-32 PB - Association for Computational Linguistics CY - Hissar, Bulgaria UR - https://fis.uni-bamberg.de/handle/uniba/94069 LA - eng AB - Most relation extraction methods, especially in the domain of biology, rely on machine learning methods to classify a cooccurring pair of entities in a sentence to be related or not. Such an approach requires a training corpus, which involves expert annotation and is tedious, time-consuming, and expensive. We overcome this problem by the use of existing knowledge in structured databases to automatically generate a training corpus for protein-protein interactions. An extensive evaluation of different instance selection strategies is performed to maximize robustness on this presumably noisy resource. Successful strategies to consistently improve performance include a majority voting ensemble of classifiers trained on subsets of the training corpus and the use of knowledge bases consisting of proven non-interactions. Our best configured model built without manually annotated data shows very competitive results on several publicly available benchmark corpora KW - Protein-Protein Interactions ER - TY - CPAPER TI - Automatically Selected Skip Edges in Conditional Random Fields for Named Entity Recognition AU - Klinger, Roman PY - 2011 T2 - Proceedings of the International Conference Recent Advances in Natural Language Processing 2011 SP - 580-585 PB - Association for Computational Linguistics CY - Hissar UR - https://fis.uni-bamberg.de/handle/uniba/93998 LA - eng AB - Incorporating distant information via manually selected skip chain templates has been shown to be beneficial for the performance of conditional random field models in contrast to a simple linear chain based structure (Sutton and McCallum, 2007; Galley, 2006; Liu et al., 2010). The set of properties to be captured by a template is typically manually chosen with respect to the application domain. In this paper, a search strategy to find meaningful skip chains independent from the application domain is proposed. From a huge set of potentially beneficial templates, some can be shown to have a positive im- pact on the performance. The search for a meaningful graphical structure demonstrates the usefulness of the approach with an increase of nearly 2 % F1 measure on a publicly available data set (Klinger et al., 2008). KW - Automatically Selected Skip Edges ER - TY - CPAPER TI - An Empirical Evaluation of Resources for the Identification of Diseases and Adverse Effects in Biomedical Literature AU - Klinger, Roman AU - Hofmann-Apitius, Martin AU - Fluck, Juliane AU - Gurulingappa, Harsha PY - 2010 T2 - 2nd Workshop on Building and evaluating resources for biomedical text mining (7th edition of the Language Resources and Evaluation Conference) UR - https://fis.uni-bamberg.de/handle/uniba/93992 LA - eng ER - TY - CPAPER TI - Abstracts versus Full Texts and Patents : A Quantitative Analysis of Biomedical Entities AU - Müller, Bernd AU - Klinger, Roman AU - Gurulingappa, Harsha AU - Mevissen, Heinz-Theodor AU - Fluck, Juliane AU - Friedrich, Christoph M. AU - Hofmann-Apitius, Martin PY - 2010 T2 - Advances in Multidisciplinary Retrieval : First Information Retrieval Facility Conference, IRFC 2010, Vienna, Austria, May 31, 2010, Proceedings SP - 152-165 PB - Springer CY - Berlin ; Heidelberg ; New York, NY T3 - Lecture Notes in Computer Science SN - 978-3-642-13083-0 DO - 10.1007/978-3-642-13084-7_12 UR - https://fis.uni-bamberg.de/handle/uniba/94018 LA - eng ER - TY - CPAPER TI - Patent Retrieval in Chemistry based on semantically tagged Named Entities AU - Gurulingappa, Harsha AU - Müller, Bernd Hans AU - Klinger, Roman AU - Mevissen, Heinz-Theo AU - Hofmann-Apitius, Martin AU - Friedrich, Christoph M. AU - Fluck, Juliane PY - 2009 T2 - Eighteenth Text REtrieval Conference, TREC 2009. Proceedings DO - 10.24406/publica-fhg-363923 UR - https://fis.uni-bamberg.de/handle/uniba/93993 LA - eng ER - TY - CPAPER TI - User's Choice of Precision and Recall in Named Entity Recognition AU - Klinger, Roman AU - Friedrich, Christoph M. PY - 2009 T2 - Proceedings of the International Conference RANLP-2009 SP - 192-196 PB - Association for Computational Linguistics CY - Borovets, Bulgaria UR - https://fis.uni-bamberg.de/handle/uniba/94005 LA - eng AB - Conditional Random Fields are commonly trained to maximize likelihood. The corresponding Fβ measure, the weighted harmonic mean of preci-sion and recall, which is established for evaluation in information retrieval and text mining, is not necessarily the optimal result for the user’s choice of β. Some approaches have been published to optimize multivariate measures like Fβ to overcome this inconsistency. The limitation is that constraints like the value of β have to be known at training time. This publication proposes a method of multiobjective optimization of both precision and recall based on a preceding likelihood training. The output is an estimation of pareto-optimal solutions from which the user can select the best for the actual application. Evaluated on two publicly available data sets in the field of named entity recognition, nearly all Fβ values are superior to those resulting from log-likelihood training KW - Named Entity Recognition KW - Conditional Random Fields KW - Multi-Objective Optimization KW - NSGA-II KW - Fβ measure KW - Recall KW - Precision ER - TY - CPAPER TI - Identification of Histone Modifications in Biomedical Text for Supporting Epigenomic Research AU - Kolarik, Corinna AU - Klinger, Roman AU - Hofmann-Apitius, Martin PY - 2009 VL - 10 IS - S28 SP - 1-11 PB - BioMed Central CY - London SN - 1471-2105 DO - 10.1186/1471-2105-10-S1-S28 UR - https://fis.uni-bamberg.de/handle/uniba/94016 LA - eng ER - TY - JOUR TI - Detection of IUPAC and IUPAC-like Chemical Names AU - Klinger, Roman AU - Kolarik, Corinna AU - Fluck, Juliane AU - Hofmann-Apitius, Martin AU - Friedrich, Christoph M. PY - 2008 JO - Bioinformatics VL - 24 IS - 13 SP - i268-i276 PB - Oxford Univ. Press CY - Oxford SN - 1367-4803 DO - 10.1093/bioinformatics/btn181 UR - https://fis.uni-bamberg.de/handle/uniba/93920 LA - eng ER - TY - JOUR TI - Overview of BioCreative II gene mention recognition AU - Smith, Larry AU - Tanabe, Lorraine K. AU - Ando, Rie Johnson nee AU - Juo, Cheng-Ju AU - Chung, I.-Fang AU - Hsu, Chun-Nan AU - Lin, Yu-Shi AU - Klinger, Roman AU - Ganchev, Kuzman AU - Torii, Manabu AU - Liu, Hongfang AU - Haddow, Barry AU - Struble, Craig A. AU - Povinelli, Richard J. AU - Vlachos, Andreas AU - Jr, William A. Baumgartner AU - Hunter, Lawrence AU - Carpenter, Bob AU - Tsai, Richard Tzong-Han AU - Dai, Hong-jie AU - Liu, Feng AU - Chen, Yifei AU - Sun, Chengjie AU - Katrenko, Sophia AU - Adriaans, Pieter AU - Blaschke, Christian AU - Perez, Rafel Torres AU - Neves, Mariana AU - Nakov, Preslav AU - Divoli, Anna AU - Mana, Manuel AU - Mata-Vazquez, Jacinto AU - Wilbur, W. John AU - Friedrich, Christoph M. PY - 2008 JO - Genome biology : biology for the post-genomic era VL - 9 IS - Suppl 2 SP - S2.2-S2.18 PB - BioMed Central CY - London SN - 1465-6906 DO - 10.1186/gb-2008-9-s2-s2 UR - https://fis.uni-bamberg.de/handle/uniba/93924 LA - eng ER - TY - JOUR TI - Knowledge Environments Representing Molecular Entities for the Virtual Physiological Human AU - Hofmann-Apitius, Martin AU - Fluck, Juliane AU - Furlong, Laura AU - Fornes, Oriol AU - Kolarik, Corinna AU - Hanser, Susanne AU - Boecker, Martin AU - Schultz, Stefan AU - Sanz, Ferran AU - Klinger, Roman AU - Mevissen, Theo AU - Gatterneyer, Tobias AU - Oliva, Baldo AU - Friedrich, Christoph PY - 2008 JO - Philosophical transactions of the Royal Society : A, Mathematical, physical and engineering sciences VL - 366 IS - 1878 PB - Royal Society CY - London SN - 1471-2962 DO - 10.1098/rsta.2008.0099 UR - https://fis.uni-bamberg.de/handle/uniba/93995 LA - eng ER - TY - CPAPER TI - Chemical Names : Terminological Resources and Corpora Annotation AU - Kolarik, Corinna AU - Klinger, Roman AU - Friedrich, Christoph M. AU - Fluck, Juliane AU - Hofmann-Apitius, Martin PY - 2008 SP - 51-58 UR - https://fis.uni-bamberg.de/handle/uniba/94015 LA - eng ER - TY - CPAPER TI - Named Entity Recognition with Combinations of Conditional Random Fields AU - Klinger, Roman AU - Friedrich, Christoph M. AU - Hofmann-Apitius, Martin AU - Fluck, Juliane PY - 2007 T2 - Proceedings of the Second BioCreative Challenge Evaluation Workshop SP - 89-91 CY - Madrid, Spain DO - 10.5281/zenodo.4274543 UR - https://fis.uni-bamberg.de/handle/uniba/94007 LA - eng AB - The Gene Mention task is a Named Entity Recognition (NER) task for labeling gene and gene product names in biomedical text. To deal with acceptable alternatives additionally to the gold standard, we use combinations of Conditional Random Fields (CRF) together with a normalizing tagger. This process isfollowed by a postprocessing step including an acronym disambiguation based on Latent Semantic Analysis(LSA). For robust model selection we apply 50-fold Bootstrapping to obtain an average F-Score of 84.58 % on the trainingset and 86.33 % on the test set. KW - named entity recognition KW - text mining KW - data mining KW - conditional random fields KW - multi model approach ER - TY - RPRT TI - Classical Probabilistic Models and Conditional Random Fields AU - Klinger, Roman AU - Tomanek, Katrin PY - 2007 SP - 1-31 PB - Technische Universität Dortmund. Chair of Algorithm Engineering (LS11) CY - Dortmund SN - TR07-2-013 UR - https://fis.uni-bamberg.de/handle/uniba/94013 LA - eng ER - TY - THES TI - Automatische Komposition von Musik mit Methoden der Computational Intelligence AU - Klinger, Roman PY - 2006 PB - Department of Computer Science, University of Dortmund CY - Dortmund UR - https://fis.uni-bamberg.de/handle/uniba/94012 LA - deu ER -