Presentation
Do students rapidly guess repeatedly over time? A longitudinal analysis of student test disengagement, background, and attitudes
September 2020
Description
This session from the 2020 National Council on Measurement in Education virtual conference presents new research findings on understanding and managing test-taking disengagement. (Presentation begins at 22:55).
Considerable research has examined the use of rapid guessing measures to identify disengaged item responses. However, little is known about students who rapidly guess over the course of several tests. In this study, we use achievement test data from six administrations over three years to investigate whether rapid guessing is a stable trait-like behavior or if rapid guessing is determined mostly by situational variables. We do so using a Latent State Trait (LST) model (Prenoveau et al., 2011) displayed in Figure 1. In these models, the relative standing of an individual relative to her peers at each time point is decomposed into a trait factor that is free of situational or interactional effects and an occasion factor capturing the situational or interactional component. If an individual’s rank ordering is entirely consistent over time, then the variance of the observed variables is completely attributable to the trait factor. Within such models, we also examine whether rapid guessing over the course of several tests is associated with certain psychological measures, such as academic self-efficacy survey scores, and background variables like English learner status.
We find that rapid guessing tends to be fairly state-like compared to academic achievement scores, which are fairly stable. Further, we show that repeated rapid guessing is strongly associated with students’ academic self-efficacy and self-management scores. These findings have implications for detecting rapid guessing and intervening to reduce its effect on observed achievement test scores.
See MoreTopics: School & test engagement
Related Topics
This study compared the test taking disengagement of students taking a remotely administered an adaptive interim assessment in spring 2020 with their disengagement on the assessment administered in-school during fall 2019.
Topics: Equity, Innovations in reporting & assessment, School & test engagement
Six insights regarding test-taking disengagement
There has been increasing concern about the presence of disengaged test taking in international assessment programs and its implications for the validity of inferences made regarding a country’s level of educational attainment. In this paper, the author discusses six important insights yielded by 20 years of research on this and implications for assessment programs.
The impact of disengaged test taking on a state’s accountability test results
This study investigated test-taking engagement on a large-scale state summative assessment. Overall, results of this study indicate that disengagement has a material impact on individual state summative test scores, though its impact on score aggregations may be relatively minor.
Topics: Equity, Measurement & scaling, School & test engagement
Variation in respondent speed and its implications: Evidence from an adaptive testing scenario
The more frequent collection of response time data is leading to an increased need for an understanding of how such data can be included in measurement models. Models for response time have been advanced, but relatively limited large-scale empirical investigations have been conducted. We take advantage of a large data set from the adaptive NWEA MAP Growth Reading Assessment to shed light on emergent features of response time behavior.
Topics: School & test engagement, Innovations in reporting & assessment
A method for identifying partial test-taking engagement
This paper describes a method for identifying partial engagement and provides validation evidence to support its use and interpretation. When test events indicate the presence of partial engagement, effort-moderated scores should be interpreted cautiously.
Topics: Measurement & scaling, Innovations in reporting & assessment, School & test engagement
This study uses reading test scores from over 728,923 3rd–8th-grade students in 2,056 schools across the US to compare threshold-setting methods to detect noneffortful item responses. and so helps provide guidance on the tradeoffs involved in using a given method to identify noneffortful responses.
Topics: School & test engagement
Comparability analysis of remote and in-person MAP Growth testing in fall 2020
How have COVID-19 school closures impacted student academic growth and achievement? New research using fall 2020 MAP Growth assessment data for 4.4 million students provides new insights, key findings, and actionable recommendations.
Topics: COVID-19 & schools, Measurement & scaling, School & test engagement
