TY - CPAPER TI - Actor Identification in Discourse : A Challenge for LLMs? AU - Barić, Ana AU - Padó, Sebastian AU - Papay, Sean PY - 2026 SP - 64–70 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/115586 LA - eng AB - The identification of political actors who put forward claims in public debate is a crucial step in the construction of discourse networks, which are helpful to analyze societal debates. Actor identification is, however, rather challenging: Often, the locally mentioned speaker of a claim is only a pronoun (“He proposed that [claim]”), so recovering the canonical actor name requires discourse understanding. We compare a traditional pipeline of dedicated NLP components (similar to those applied to the related task of coreference) with a LLM, which appears a good match for this generation task. Evaluating on a corpus of German actors in newspaper reports, we find surprisingly that the LLM performs worse. Further analysis reveals that the LLM is very good at identifying the right reference, but struggles to generate the correct canonical form. This points to an underlying issue in LLMs with controlling generated output. Indeed, a hybrid model combining the LLM with a classifier to normalize its output substantially outperforms both initial models. KW - Actor Identification ER - TY - CPAPER TI - Automatic Analysis of Political Debates and Manifestos : Successes and Challenges AU - Ceron, Tanise AU - Barić, Ana AU - Blessing, André AU - Haunss, Sebastian AU - Kuhn, Jonas AU - Lapesa, Gabriella AU - Padó, Sebastian AU - Papay, Sean AU - Zauchner, Patricia F. PY - 2026 SP - 71–88 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/115585 LA - eng AB - The opinions of political actors (e.g., politicians, parties, organizations) expressed through claims are the core elements of political debates and decision-making. Political actors communicate through different channels: parties publish manifestos for major elections, while individual actors make statements on a day-to-day basis as reflected in the media. These two channels offer different approaches for analysis: Manifestos, on the one hand, are useful to characterize the parties’ positions at a global ideological level over time. In contrast, individual statements can be collected to analyze debates in particular policy domains on a fine-grained level, in terms of individual actors and claims. In this article, we summarize a series of studies we have carried out. We apply NLP-driven (semi-)automatic analyses on these two channels and compare their potentials and challenges. The fine-grained analysis yields rich insights into the communication but comes at the cost of three challenges: (a) a substantial hunger for manual annotation, introducing practical hurdles for analysis both within and across languages; (b) difficulties in claim classification arising from the uneven frequency distribution over the theory-based annotation schemas; (c) the need to map actor mentions onto canonical versions. Manifesto-based analysis avoids these challenges to a substantial extent when a more coarse-grained analysis of party positions is sufficient. We highlight the benefits and challenges of both approaches, and conclude by outlining perspectives for addressing the challenges in future research. KW - Claim identification KW - discourse network analysis KW - party positioning KW - argument mining ER - TY - CPAPER TI - Medical Summarization in Practice : Design, Deployment, and Analysis of a Clinical Summarization System for a German Hospital AU - Rauf, Moiz AU - Papay, Sean PY - 2026 SP - 455-466 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/115584 LA - eng AB - Through the course of hospital treatment, a large number of electronic health records (EHRs) are created for a patient, detailingaspects of care history such as lab results, physician notes, and treatments administered.At the conclusion of treatment, this collection of EHRs must be summarized into a discharge summary,describing the course or care clearly and cohesively.In this paper, we present the design and development of a clinical summarization system integrated into a live German hospital workflowto help with the generation of discharge summaries.We first describe the system, its components, and its context of use within a hospital,before performing a number of experiments to gain insights into how best to use and evaluate our system.We investigate summarization performance across multiple input encoding strategies, compare expert judgments against automatic evaluation of summaries,and analyze the consistency of model summaries across multiple text generations.This work not only acts as a case study to demonstrate the feasibility of LLM integration into healthcare infrastructure,but also provides actionable insights into the use and evaluation of such systems. KW - Clinical Summarization System ER - TY - CPAPER TI - Medical Summarization in Practice : Design, Deployment, and Analysis of a Clinical Summarization System for a German Hospital AU - Rauf, Moiz AU - Papay, Sean PY - 2026 T2 - Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics SP - 455–466 PB - Association for Computational Linguistics SN - 979-8-89176-384-5 DO - 10.18653/v1/2026.eacl-industry.34 UR - https://fis.uni-bamberg.de/handle/uniba/115295 LA - eng AB - Through the course of hospital treatment, a large number of electronic health records (EHRs) are created for a patient, detailingaspects of care history such as lab results, physician notes, and treatments administered.At the conclusion of treatment, this collection of EHRs must be summarized into a discharge summary,describing the course or care clearly and cohesively.In this paper, we present the design and development of a clinical summarization system integrated into a live German hospital workflowto help with the generation of discharge summaries.We first describe the system, its components, and its context of use within a hospital,before performing a number of experiments to gain insights into how best to use and evaluate our system.We investigate summarization performance across multiple input encoding strategies, compare expert judgments against automatic evaluation of summaries,and analyze the consistency of model summaries across multiple text generations.This work not only acts as a case study to demonstrate the feasibility of LLM integration into healthcare infrastructure,but also provides actionable insights into the use and evaluation of such systems. KW - Clinical Summarization System ER - TY - CPAPER TI - Strategies for political-statement segmentation and labelling in unstructured text AU - Nikolaev, Dmitry AU - Papay, Sean PY - 2026 T2 - Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities SP - 437–451 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/115245 LA - eng AB - Analysis of parliamentary speeches and political-party manifestos has become an integral area of computational study of political texts. While speeches have been overwhelmingly analysed using unsupervised methods, a large corpus of manifestos with by-statement political-stance labels has been created by the participants of the MARPOR project. It has been recently shown that these labels can be predicted by a neural model; however, the current approach relies on provided statement boundaries, limiting out-of-domain applicability. In this work, we propose and test a range of unified split-and-label frameworks—based on linear-chain CRFs, fine-tuned text-to-text models, and the combination of in-context learning with constrained decoding—that can be used to jointly segment and classify statements from raw textual data. We show that our approaches achieve competitive accuracy when applied to raw text of political manifestos, and then demonstrate the research potential of our method by applying it to the records of the UK House of Commons and tracing the political trajectories of four major parties in the last three decades. KW - manifesto KW - corpus KW - annotation ER - TY - CPAPER TI - Which Demographics do LLMs Default to During Annotation? AU - Schäfer, Johannes AU - Combs, Aidan AU - Bagdon, Christopher Doyle AU - Li, Jiahui AU - Probol, Nadine AU - Greschner, Lynn AU - Papay, Sean AU - Menchaca Resendiz, Yarik AU - Velutharambath, Aswathy AU - Wuehrl, Amelie AU - Weber, Sabine AU - Klinger, Roman PY - 2026 SP - 17331–17348 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/114413 LA - eng AB - Demographics and cultural background of annotators influence the labels they assign in text annotation – for instance, an elderly woman might find it offensive to read a message addressed to a “bro”, but a male teenager might find it appropriate. It is therefore important to acknowledge label variations to not underrepresent members of a society. Two research directions developed out of this observation in the context of using large language models (LLM) for data annotations, namely (1) studying biases and inherent knowledge of LLMs and (2) injecting diversity in the output by manipulating the prompt with demographic information. We combine these two strands of research and ask the question to which demographics an LLM resorts when no demographics is given. To answer this question, we evaluate which attributes of human annotators LLMs inherently mimic. Furthermore, we compare non-demographic conditioned prompts and placebo-conditioned prompts (e.g., “you are an annotator who lives in house number 5”) to demographics-conditioned prompts (“You are a 45 year old man and an expert on politeness annotation. How do you rate {instance}”). We study these questions for politeness and offensiveness annotations on the POPQUORN data set, a corpus created in a controlled manner to investigate human label variations based on demographics which has not been used for LLM-based analyses so far. We observe notable influences related to gender, race, and age in demographic prompting, which contrasts with previous studies that found no such effects KW - - ER - TY - CPAPER TI - Regular-pattern-sensitive CRFs for Distant Label Interactions AU - Papay, Sean AU - Klinger, Roman AU - Padó, Sebastian PY - 2026 SP - 26–35 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/114408 LA - eng AB - While LLMs have grown popular in sequence labeling, linear-chain conditional random fields (CRFs) remain a popular alternative with the ability to directly model interactions between labels. However, the Markov assumption limits them to interactions between adjacent labels. Weighted finite state transducers (FSTs), in contrast, can model distant label–label interactions, but exact label inference is intractable in general. In this work, we present regular-pattern-sensitive CRFs (RPCRFs), a method of enriching standard linear-chain CRFs with the ability to learn long-distance label interactions through userspecified patterns. This approach allows users to write regular-expression label patterns concisely specifying which types of interactions the model should take into account, allowing the model to learn from data whether and in which contexts these patterns occur. The result can be interpreted alternatively as a CRF augmented with additional, non-local potentials, or as a finite-state transducer whose structure is defined by a set of easily-interpretable patterns. Critically, exact training and inference are tractable for many pattern sets. We detail how an RPCRF can be automatically constructed from a set of user-specified patterns, and demonstrate the model’s effectiveness on a sequence of three synthetic sequence modeling datasets. KW - - ER - TY - CPAPER TI - Are Humans as Brittle as Large Language Models? AU - Li, Jiahui AU - Papay, Sean AU - Klinger, Roman PY - 2026 SP - 2130–2155 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/113162 LA - eng AB - The output of large language models (LLMs) is unstable, due both to non-determinism of the decoding process as well as to prompt brittleness. While the intrinsic non-determinism of LLM generation may mimic existing uncertainty in human annotations through distributional shifts in outputs, it is largely assumed, yet unexplored, that the prompt brittleness effect is unique to LLMs. This raises the question: do human annotators show similar sensitivity to prompt changes? If so, should prompt brittleness in LLMs be considered problematic? One may alternatively hypothesize that prompt brittleness correctly reflects human annotation variances. To fill this research gap, we systematically compare the effects of prompt modifications on LLMs and identical instruction modifications for human annotators, focusing on the question of whether humans are similarly sensitive to prompt perturbations. To study this, we prompt both humans and LLMs for a set of text classification tasks conditioned on prompt variations. Our findings indicate that both humans and LLMs exhibit increased brittleness in response to specific types of prompt modifications, particularly those involving the substitution of alternative label sets or label formats. However, the distribution of human judgments is less affected by typographical errors and reversed label order than that of LLMs. KW - Large Language Model ER - TY - CPAPER TI - Are Humans as Brittle as Large Language Models? AU - Li, Jiahui AU - Papay, Sean AU - Klinger, Roman PY - 2025 T2 - Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics SP - 2130–2155 PB - The Asian Federation of Natural Language Processing and The Association for Computational Linguistics CY - Mumbai, India SN - 979-8-89176-298-5 UR - https://fis.uni-bamberg.de/handle/uniba/112676 LA - eng AB - The output of large language models (LLMs) is unstable, due both to non-determinism of the decoding process as well as to prompt brittleness. While the intrinsic non-determinism of LLM generation may mimic existing uncertainty in human annotations through distributional shifts in outputs, it is largely assumed, yet unexplored, that the prompt brittleness effect is unique to LLMs. This raises the question: do human annotators show similar sensitivity to prompt changes? If so, should prompt brittleness in LLMs be considered problematic? One may alternatively hypothesize that prompt brittleness correctly reflects human annotation variances. To fill this research gap, we systematically compare the effects of prompt modifications on LLMs and identical instruction modifications for human annotators, focusing on the question of whether humans are similarly sensitive to prompt perturbations. To study this, we prompt both humans and LLMs for a set of text classification tasks conditioned on prompt variations. Our findings indicate that both humans and LLMs exhibit increased brittleness in response to specific types of prompt modifications, particularly those involving the substitution of alternative label sets or label formats. However, the distribution of human judgments is less affected by typographical errors and reversed label order than that of LLMs. KW - Large Language Model ER - TY - CPAPER TI - Dissecting Span Identification Tasks with Performance Prediction AU - Papay, Sean AU - Klinger, Roman AU - Padó, Sebastian PY - 2025 SP - 4881-4895 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108319 LA - eng AB - Span identification (in short, span ID) tasks such as chunking, NER, or code-switching detection, ask models to identify and classify relevant spans in a text. Despite being a staple of NLP, and sharing a common structure, there is little insight on how these tasks’ properties influence their difficulty, and thus little guidance on what model families work well on span ID tasks, and why. We analyze span ID tasks via performance prediction, estimating how well neural architectures do on different tasks. Our contributions are: (a) we identify key properties of span ID tasks that can inform performance prediction; (b) we carry out a large-scale experiment on English data, building a model to predict performance for unseen span ID tasks that can support architecture choices; (c), we investigate the parameters of the meta model, yielding new insights on how model and task properties interact to affect span ID performance. We find, e.g., that span frequency is especially important for LSTMs, and that CRFs help when spans are infrequent and boundaries non-distinctive. KW - Span Identification Tasks ER - TY - CPAPER TI - Which Demographics do LLMs Default to During Annotation? AU - Schäfer, Johannes AU - Combs, Aidan AU - Bagdon, Christopher Doyle AU - Li, Jiahui AU - Probol, Nadine AU - Greschner, Lynn AU - Papay, Sean AU - Menchaca Resendiz, Yarik AU - Velutharambath, Aswathy AU - Wuehrl, Amelie AU - Weber, Sabine AU - Klinger, Roman PY - 2025 T2 - Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics SP - 17331–17348 PB - Association for Computational Linguistics SN - 979-8-89176-251-0 DO - 10.18653/v1/2025.acl-long.848 UR - https://fis.uni-bamberg.de/handle/uniba/109541 LA - eng AB - Demographics and cultural background of annotators influence the labels they assign in text annotation – for instance, an elderly woman might find it offensive to read a message addressed to a “bro”, but a male teenager might find it appropriate. It is therefore important to acknowledge label variations to not underrepresent members of a society. Two research directions developed out of this observation in the context of using large language models (LLM) for data annotations, namely (1) studying biases and inherent knowledge of LLMs and (2) injecting diversity in the output by manipulating the prompt with demographic information. We combine these two strands of research and ask the question to which demographics an LLM resorts when no demographics is given. To answer this question, we evaluate which attributes of human annotators LLMs inherently mimic. Furthermore, we compare non-demographic conditioned prompts and placebo-conditioned prompts (e.g., “you are an annotator who lives in house number 5”) to demographics-conditioned prompts (“You are a 45 year old man and an expert on politeness annotation. How do you rate {instance}”). We study these questions for politeness and offensiveness annotations on the POPQUORN data set, a corpus created in a controlled manner to investigate human label variations based on demographics which has not been used for LLM-based analyses so far. We observe notable influences related to gender, race, and age in demographic prompting, which contrasts with previous studies that found no such effects KW - - ER - TY - CPAPER TI - Regular-pattern-sensitive CRFs for Distant Label Interactions AU - Papay, Sean AU - Klinger, Roman AU - Padó, Sebastian PY - 2025 T2 - Proceedings of the 1st Joint Workshop on Large Language Models and Structure Modeling (XLLM 2025) SP - 26–35 PB - Association for Computational Linguistics SN - 979-8-89176-286-2 DO - 10.18653/v1/2025.xllm-1.4 UR - https://fis.uni-bamberg.de/handle/uniba/109547 LA - eng AB - While LLMs have grown popular in sequence labeling, linear-chain conditional random fields (CRFs) remain a popular alternative with the ability to directly model interactions between labels. However, the Markov assumption limits them to interactions between adjacent labels. Weighted finite state transducers (FSTs), in contrast, can model distant label–label interactions, but exact label inference is intractable in general. In this work, we present regular-pattern-sensitive CRFs (RPCRFs), a method of enriching standard linear-chain CRFs with the ability to learn long-distance label interactions through userspecified patterns. This approach allows users to write regular-expression label patterns concisely specifying which types of interactions the model should take into account, allowing the model to learn from data whether and in which contexts these patterns occur. The result can be interpreted alternatively as a CRF augmented with additional, non-local potentials, or as a finite-state transducer whose structure is defined by a set of easily-interpretable patterns. Critically, exact training and inference are tractable for many pattern sets. We detail how an RPCRF can be automatically constructed from a set of user-specified patterns, and demonstrate the model’s effectiveness on a sequence of three synthetic sequence modeling datasets. KW - - ER - TY - CPAPER TI - DERE : A Task and Domain-Independent Slot Filling Framework for Declarative Relation Extraction AU - Adel, Heike AU - Bostan, Laura Ana Maria AU - Papay, Sean AU - Padó, Sebastian AU - Klinger, Roman PY - 2025 SP - 42–47 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/108297 LA - eng AB - Most machine learning systems for natural language processing are tailored to specific tasks. As a result, comparability of models across tasks is missing and their applicability to new tasks is limited. This affects end users without machine learning experience as well as model developers. To address these limitations, we present DERE, a novel framework for declarative specification and compilation of template-based information extraction. It uses a generic specification language for the task and for data annotations in terms of spans and frames. This formalism enables the representation of a large variety of natural language processing challenges. The backend can be instantiated by different models, following different paradigms. The clear separation of frame specification and model backend will ease the implementation of new models and the evaluation of different models across different tasks. Furthermore, it simplifies transfer learning, joint learning across tasks and/or domains as well as the assessment of model generalizability. DERE is available as open-source software. KW - DERE ER - TY - CPAPER TI - Strategies for political-statement segmentation and labelling in unstructured text AU - Nikolaev, Dmitry AU - Papay, Sean PY - 2025 T2 - Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities SP - 437–451 PB - Association for Computational Linguistics CY - Stroudsburg, PA DO - 10.18653/v1/2025.nlp4dh-1.38 UR - https://fis.uni-bamberg.de/handle/uniba/108001 LA - eng AB - Analysis of parliamentary speeches and political-party manifestos has become an integral area of computational study of political texts. While speeches have been overwhelmingly analysed using unsupervised methods, a large corpus of manifestos with by-statement political-stance labels has been created by the participants of the MARPOR project. It has been recently shown that these labels can be predicted by a neural model; however, the current approach relies on provided statement boundaries, limiting out-of-domain applicability. In this work, we propose and test a range of unified split-and-label frameworks—based on linear-chain CRFs, fine-tuned text-to-text models, and the combination of in-context learning with constrained decoding—that can be used to jointly segment and classify statements from raw textual data. We show that our approaches achieve competitive accuracy when applied to raw text of political manifestos, and then demonstrate the research potential of our method by applying it to the records of the UK House of Commons and tracing the political trajectories of four major parties in the last three decades. KW - manifesto KW - corpus KW - annotation ER - TY - THES TI - Task generality in relation extraction AU - Papay, Sean PY - 2024 PB - Universitätsbibliothek der Universität Stuttgart CY - Stuttgart DO - https://doi.org/10.18419/opus-14421 UR - https://fis.uni-bamberg.de/handle/uniba/115309 LA - eng ER - TY - CPAPER TI - Actor Identification in Discourse : A Challenge for LLMs? AU - Barić, Ana AU - Padó, Sebastian AU - Papay, Sean PY - 2024 T2 - Proceedings of the 5th Workshop on Computational Approaches to Discourse (CODI 2024) SP - 64–70 PB - Association for Computational Linguistics DO - 10.18653/v1/2024.codi-1.6 UR - https://fis.uni-bamberg.de/handle/uniba/115293 LA - eng AB - The identification of political actors who put forward claims in public debate is a crucial step in the construction of discourse networks, which are helpful to analyze societal debates. Actor identification is, however, rather challenging: Often, the locally mentioned speaker of a claim is only a pronoun (“He proposed that [claim]”), so recovering the canonical actor name requires discourse understanding. We compare a traditional pipeline of dedicated NLP components (similar to those applied to the related task of coreference) with a LLM, which appears a good match for this generation task. Evaluating on a corpus of German actors in newspaper reports, we find surprisingly that the LLM performs worse. Further analysis reveals that the LLM is very good at identifying the right reference, but struggles to generate the correct canonical form. This points to an underlying issue in LLMs with controlling generated output. Indeed, a hybrid model combining the LLM with a classifier to normalize its output substantially outperforms both initial models. KW - Actor Identification ER - TY - CPAPER TI - Automatic Analysis of Political Debates and Manifestos : Successes and Challenges AU - Ceron, Tanise AU - Barić, Ana AU - Blessing, André AU - Haunss, Sebastian AU - Kuhn, Jonas AU - Lapesa, Gabriella AU - Padó, Sebastian AU - Papay, Sean AU - Zauchner, Patricia F. PY - 2024 T2 - Robust argumentation machines : first international conference, RATIO 2024, Bielefeld, Germany, June 5-7, 2024 : proceedings SP - 71–88 PB - Springer Nature CY - Cham, Switzerland SN - 978-3-031-63536-6 DO - 10.1007/978-3-031-63536-6_5 UR - https://fis.uni-bamberg.de/handle/uniba/115294 LA - eng AB - The opinions of political actors (e.g., politicians, parties, organizations) expressed through claims are the core elements of political debates and decision-making. Political actors communicate through different channels: parties publish manifestos for major elections, while individual actors make statements on a day-to-day basis as reflected in the media. These two channels offer different approaches for analysis: Manifestos, on the one hand, are useful to characterize the parties’ positions at a global ideological level over time. In contrast, individual statements can be collected to analyze debates in particular policy domains on a fine-grained level, in terms of individual actors and claims. In this article, we summarize a series of studies we have carried out. We apply NLP-driven (semi-)automatic analyses on these two channels and compare their potentials and challenges. The fine-grained analysis yields rich insights into the communication but comes at the cost of three challenges: (a) a substantial hunger for manual annotation, introducing practical hurdles for analysis both within and across languages; (b) difficulties in claim classification arising from the uneven frequency distribution over the theory-based annotation schemas; (c) the need to map actor mentions onto canonical versions. Manifesto-based analysis avoids these challenges to a substantial extent when a more coarse-grained analysis of party positions is sufficient. We highlight the benefits and challenges of both approaches, and conclude by outlining perspectives for addressing the challenges in future research. KW - Claim identification KW - discourse network analysis KW - party positioning KW - argument mining ER - TY - CPAPER TI - Sentence‑Final Particle de in Mandarin as an Informativity Maximizer AU - Chen, Jun AU - Papay, Sean PY - 2024 T2 - Selected Reflections in Language, Logic, and Information : ESSLLI 2019, ESSLLI 2020 and ESSLLI 2021 Student Sessions, Selected Papers SP - 24–43 PB - Springer Nature Switzerland CY - Cham SN - 978-3-031-50627-7 DO - 10.1007/978-3-031-50628-4_3 UR - https://fis.uni-bamberg.de/handle/uniba/115198 LA - eng ER - TY - CPAPER TI - Constraining Linear-chain CRFs to Regular Languages AU - Papay, Sean AU - Klinger, Roman AU - Padó, Sebastian PY - 2022 PB - arxiv UR - https://fis.uni-bamberg.de/handle/uniba/93893 LA - eng ER - TY - CPAPER TI - Dissecting Span Identification Tasks with Performance Prediction AU - Papay, Sean AU - Klinger, Roman AU - Padó, Sebastian PY - 2020 T2 - Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) SP - 4881-4895 PB - Association for Computational Linguistics DO - 10.18653/v1/2020.emnlp-main.396 UR - https://fis.uni-bamberg.de/handle/uniba/93908 LA - eng AB - Span identification (in short, span ID) tasks such as chunking, NER, or code-switching detection, ask models to identify and classify relevant spans in a text. Despite being a staple of NLP, and sharing a common structure, there is little insight on how these tasks’ properties influence their difficulty, and thus little guidance on what model families work well on span ID tasks, and why. We analyze span ID tasks via performance prediction, estimating how well neural architectures do on different tasks. Our contributions are: (a) we identify key properties of span ID tasks that can inform performance prediction; (b) we carry out a large-scale experiment on English data, building a model to predict performance for unseen span ID tasks that can support architecture choices; (c), we investigate the parameters of the meta model, yielding new insights on how model and task properties interact to affect span ID performance. We find, e.g., that span frequency is especially important for LSTMs, and that CRFs help when spans are infrequent and boundaries non-distinctive. KW - Span Identification Tasks ER - TY - CPAPER TI - DERE : A Task and Domain-Independent Slot Filling Framework for Declarative Relation Extraction AU - Adel, Heike AU - Papay, Sean AU - Klinger, Roman AU - Padó, Sebastian AU - Bostan, Laura Ana Maria PY - 2018 T2 - Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing : System Demonstrations SP - 42–47 PB - Association for Computational Linguistics DO - 10.18653/v1/D18-2008 UR - https://fis.uni-bamberg.de/handle/uniba/93960 LA - eng AB - Most machine learning systems for natural language processing are tailored to specific tasks. As a result, comparability of models across tasks is missing and their applicability to new tasks is limited. This affects end users without machine learning experience as well as model developers. To address these limitations, we present DERE, a novel framework for declarative specification and compilation of template-based information extraction. It uses a generic specification language for the task and for data annotations in terms of spans and frames. This formalism enables the representation of a large variety of natural language processing challenges. The backend can be instantiated by different models, following different paradigms. The clear separation of frame specification and model backend will ease the implementation of new models and the evaluation of different models across different tasks. Furthermore, it simplifies transfer learning, joint learning across tasks and/or domains as well as the assessment of model generalizability. DERE is available as open-source software. KW - DERE ER -