TY - THES TI - Disengaged Responding in Educational Large-Scale Assessments : New Insights into its Identification, Modeling, and Consequences AU - Welling, Jana PY - 2026 PB - Otto-Friedrich-Universität CY - Bamberg DO - 10.20378/irb-116761 UR - https://fis.uni-bamberg.de/handle/uniba/116761 LA - eng AB - Educational large-scale assessments (LSAs) provide information on competencies across cohorts, populations, and test domains. However, when examinees lack test-taking motivation, they can respond to test items without investing their best effort, knowledge, and abilities, a behavior called disengaged responding. Because disengaged responses do not reflect the proficiency of the examinees, these responses can pose a threat to the validity of assessment outcomes. To ensure the interpretability of findings from LSAs, disengaged responding needs to be accurately identified and accounted for in the measurement of competencies. Moreover, it is important to understand in which contexts disengaged responding arises and which consequences it entails. Therefore, this dissertation includes three articles that addressed major, yet unresolved issues in the context of disengaged responding concerning its identification (Article 1), accounting for it in the measurement of competencies (Article 2), and the generalizability of established findings to child samples (Article 3). Most approaches to identifying disengaged responding rely solely on item responses and response times, thereby risking the misclassification of fast engaged or slow disengaged responses. Article 1 aimed to improve the classification accuracy of response-time based identification methods by introducing and evaluating three novel process data indicators of disengaged responding in reading comprehension tasks: text reread, item revisit, and answer change. Findings from an empirical study suggest that the improvement in classification accuracy was marginal to non-existent, thereby confirming response times as an important indicator of disengagement. Although many LSAs provide proficiency estimates as plausible values, no study has yet investigated how disengagement should be accounted for in their estimation. Article 2 addressed this gap by proposing and evaluating different approaches to (not) accounting for disengaged responding in the estimation of plausible values. A simulation study demonstrated that achievement gaps estimated with plausible values can be biased by disengaged responding. While models that accounted for disengagement in the measurement model effectively reduced these biases, models accounting for it solely in the population model did not. An additional empirical study found no difference between the models, supporting recent studies which claim that the impact of disengaged responding on achievement gaps may be negligible in practice. Most empirical findings on disengaged responding are based on adolescent or adult samples, ignoring the fact that many LSAs also include child samples. Article 3 challenged the generalizability of established findings on disengaged responding to child samples by providing theoretical considerations on why the test-taking behavior of children may differ from that of older examinees. Additionally, the findings from an empirical study supported recent research indicating that disengaged responding may be less common among children. Only proficiency could be replicated as predictor of disengaged responding, while gender and socio-economic status could not. Further results suggested that the impact of disengaged responding on achievement gaps may be limited in this age group. Taken together, this dissertation combines theoretical, methodological, and empirical approaches to improve our understanding of disengaged responding and the validity of LSAs. The central findings bear important implications for the identification, modeling, and consequences of disengaged responding. First, they support response-time based measures as strong indicators of disengagement while suggesting that process data may not always provide sufficient information on test-taking engagement to inform new identification methods. Second, they highlight the need to accurately account for disengaged responding also in plausible value estimation. Third, this dissertation suggests that and discusses why the threat (rapid) disengagement poses to the validity of group-based assessment outcomes may be less pronounced than often assumed. Fourth, it challenges the generalizability of established findings to child samples. By laying conceptual and methodological groundwork, this dissertation provides a foundation for future research aimed at developing more robust, valid, and context-sensitive approaches to measuring competencies in educational LSAs. KW - disengaged responding KW - rapid guessing KW - test-taking engagement KW - test-taking behavior KW - large-scale assessments KW - process data ER - TY - JOUR TI - Comparing Different Approaches of (Not) Accounting for Rapid Guessing in Plausible Values Estimation AU - Welling, Jana AU - Zink, Eva AU - Gnambs, Timo PY - 2026 SP - 792-821 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/116524 LA - eng AB - Educational large-scale assessments provide information on ability differences between groups, informing policies and shaping educational decisions. However, some of these differences might partly reflect variations in test-taking motivation rather than in actual abilities. Existing approaches for mitigating the distorting effects of rapid guessing focus mainly on point estimates of abilities, although research questions often refer to latent variables. The present study seeks to (a) determine the bias introduced by rapid guessing in group comparisons based on plausible value estimates and (b) introduce and evaluate different approaches of handling rapid guessing in the estimation of plausible values. In a simulation study, four models were compared: (1) a baseline model did not account for rapid guessing, (2) a person-level model incorporated rapid guessing as a respondent characteristic in the background model, (3) a response-level model filtered responses with item response times lower than a predetermined threshold, and (4) a combined model merged the person- and response-level approaches. Results show that the response-level and combined model performed best while accounting for rapid guessing on the person level did not suffice. An empirical example using data from a German large-scale assessment (N = 478) demonstrates the applicability of all approaches in practice. Recommendations for future research are given to improve ability estimation. KW - rapid guessing KW - plausible values KW - large-scale assessments KW - test-taking engagement KW - engagement gap KW - ability comparisons ER - TY - JOUR TI - Comparing Different Approaches of (Not) Accounting for Rapid Guessing in Plausible Values Estimation AU - Welling, Jana AU - Zink, Eva AU - Gnambs, Timo PY - 2026 JO - Educational and psychological measurement : EPM VL - 86 IS - 4 SP - 792-821 PB - Sage Periodicals CY - Thousand Oaks, Calif. [u.a.] SN - 1552-3888 DO - 10.1177/00131644251395590 UR - https://fis.uni-bamberg.de/handle/uniba/116522 LA - eng AB - Educational large-scale assessments provide information on ability differences between groups, informing policies and shaping educational decisions. However, some of these differences might partly reflect variations in test-taking motivation rather than in actual abilities. Existing approaches for mitigating the distorting effects of rapid guessing focus mainly on point estimates of abilities, although research questions often refer to latent variables. The present study seeks to (a) determine the bias introduced by rapid guessing in group comparisons based on plausible value estimates and (b) introduce and evaluate different approaches of handling rapid guessing in the estimation of plausible values. In a simulation study, four models were compared: (1) a baseline model did not account for rapid guessing, (2) a person-level model incorporated rapid guessing as a respondent characteristic in the background model, (3) a response-level model filtered responses with item response times lower than a predetermined threshold, and (4) a combined model merged the person- and response-level approaches. Results show that the response-level and combined model performed best while accounting for rapid guessing on the person level did not suffice. An empirical example using data from a German large-scale assessment (N = 478) demonstrates the applicability of all approaches in practice. Recommendations for future research are given to improve ability estimation. KW - rapid guessing KW - plausible values KW - large-scale assessments KW - test-taking engagement KW - engagement gap KW - ability comparisons ER - TY - JOUR TI - Identifying Disengaged Responding in Multiple-Choice Items : Extending a Latent Class Item Response Model With Novel Process Data Indicators AU - Welling, Jana AU - Gnambs, Timo AU - Carstensen, Claus H. PY - 2024 SP - 314-339 PB - Otto-Friedrich-Universität CY - Bamberg UR - https://fis.uni-bamberg.de/handle/uniba/95700 LA - eng AB - Disengaged responding poses a severe threat to the validity of educational large-scale assessments, because item responses from unmotivated test-takers do not reflect their actual ability. Existing identification approaches rely primarily on item response times, which bears the risk of misclassifying fast engaged or slow disengaged responses. Process data with its rich pool of additional information on the test-taking process could thus be used to improve existing identification approaches. In this study, three process data variables—text reread, item revisit, and answer change—were introduced as potential indicators of response engagement for multiple-choice items in a reading comprehension test. An extended latent class item response model for disengaged responding was developed by including the three new indicators as additional predictors of response engagement. In a sample of 1,932 German university students, the extended model indicated a better model fit than the baseline model, which included item response time as only indicator of response engagement. In the extended model, both item response time and text reread were significant predictors of response engagement. However, graphical analyses revealed no systematic differences in the item and person parameter estimation or item response classification between the models. These results suggest only a marginal improvement of the identification of disengaged responding by the new indicators. Implications of these results for future research on disengaged responding with process data are discussed. KW - disengaged responding KW - rapid guessing KW - process data KW - item response theory KW - computer-based assessments KW - multiple-choice items ER - TY - JOUR TI - Identifying Disengaged Responding in Multiple-Choice Items : Extending a Latent Class Item Response Model With Novel Process Data Indicators AU - Welling, Jana AU - Gnambs, Timo AU - Carstensen, Claus H. PY - 2024 JO - Educational and psychological measurement : EPM VL - 84 IS - 2 SP - 314-339 PB - Sage Periodicals CY - Thousand Oaks, Calif. [u.a.] SN - 0013-1644 DO - 10.1177/00131644231169211 UR - https://fis.uni-bamberg.de/handle/uniba/95699 LA - eng AB - Disengaged responding poses a severe threat to the validity of educational large-scale assessments, because item responses from unmotivated test-takers do not reflect their actual ability. Existing identification approaches rely primarily on item response times, which bears the risk of misclassifying fast engaged or slow disengaged responses. Process data with its rich pool of additional information on the test-taking process could thus be used to improve existing identification approaches. In this study, three process data variables—text reread, item revisit, and answer change—were introduced as potential indicators of response engagement for multiple-choice items in a reading comprehension test. An extended latent class item response model for disengaged responding was developed by including the three new indicators as additional predictors of response engagement. In a sample of 1,932 German university students, the extended model indicated a better model fit than the baseline model, which included item response time as only indicator of response engagement. In the extended model, both item response time and text reread were significant predictors of response engagement. However, graphical analyses revealed no systematic differences in the item and person parameter estimation or item response classification between the models. These results suggest only a marginal improvement of the identification of disengaged responding by the new indicators. Implications of these results for future research on disengaged responding with process data are discussed. KW - disengaged responding KW - rapid guessing KW - process data KW - item response theory KW - computer-based assessments KW - multiple-choice items ER -