Presentation
Using retest data to evaluate and improve effort-moderated scoring
September 2020
Description
This session from the National Council on Measurement in Education 2020 virtual conference presents new research findings on understanding and managing test-taking disengagement (presentation begins at 22:55).
Wise, S., & Kuhfeld, M. (2020, September). Using retest data to evaluate and improve effort-moderated scoring. National Council on Measurement in Education virtual conference.
There has been increased research into managing test-taking disengagement. Some research focused on methods for adjusting scores for the effects of rapid guessing, which is a validated indicator of disengaged item responding. One example is the effort-moderated (E-M) model (Wise & DeMars, 2006), in which rapid guesses are identified and excluded during scoring. E-M scoring is intended to estimate the score a disengaged test taker would have received had they been fully engaged. Although it is challenging to evaluate the accuracy of E-M score adjustments, we developed a unique dataset that permits such an evaluation.
Study 1 analyzed a dataset based on archived RIT-score data from MAP Growth assessments in math and reading from students who had been retested within a short period of time. This allowed us to study the accuracy of E-M scoring. Across two testing terms, instances of students being retested within one day were identified. Within this group, over 5,000 cases were found in which students exhibited disengagement (10%+ rapid guesses) on the first testing and zero rapid guesses on the retest. Table 1 shows that E-M scores partially accounted for the observed RIT score differences.
Study 2 will investigate, using the same dataset, a purification method for improving E-M score accuracy. Specifically, the use of various secondary time thresholds for identifying responses excluded during E-M scoring will be investigated. The new method is expected to improve our ability to estimate the score distortion caused by test-taking disengagement, thus enhancing the value of E-M scoring.
See MoreAssociated Research
Related Topics
Predicting Amira Reading Mastery Based on NWEA MAP Reading Fluency Benchmark Assessment Scores
This document presents results from a linking study conducted by NWEA in May 2024 to statistically connect the grades 1–5 English Amira Reading Mastery (ARM) scores with the Scaled-Words-Correct-Per-Minute (SWCPM) scores from the MAP Reading Fluency benchmark assessment taken during Fall and Winter 2023–2024.
Products: MAP Reading Fluency
Topics: Computer adaptive testing, Early learning, Measurement & scaling, Reading & language arts
Achievement and Growth Norms for Course-Specific MAP Growth Tests
This report documents the procedure used to produce the achievement and growth user norms for a series of the course-specific MAP® Growth™ subject tests, including Algebra 1, Geometry, Algebra 2, Integrated Math I, Integrated Math II, Integrated Math III, and Biology/Life Science. Among these tests, Integrated Math I, Integrated Math II, Integrated Math III, and Biology/Life Science were the first time to have their norms available. The remaining tests, i.e., Algebra 1, Geometry, and Algebra 2, had their norms updated including receiving more between-term growth norms by using more recent test events. Procedure for norm sample selection and a model-based approach using the multivariate true score model (Thum & He, 2019) that factors out known imprecision of scores to generate the norms are also provided in detail, along with the snapshots of the achievement and growth norms for each test.
Products: MAP Growth
Topics: Measurement & scaling
Longitudinal models of reading and mathematics achievement in deaf and hard of hearing students
New research using longitudinal data provides evidence that deaf and hard of hearing (DHH) students continue to build skills in math and reading throughout grades 2 to 8, challenging assumptions that DHH students’ skills plataeu in elementary grades.
Topics: Equity, Measurement & scaling
Achievement and growth norms for English MAP Reading Fluency Foundational Skills
This report documents the norming study procedure used to produce the achievement and growth user norms for English MAP Reading Fluency Foundational Skills.
Products: MAP Reading Fluency
Topics: Measurement & scaling
This study compared the test taking disengagement of students taking a remotely administered an adaptive interim assessment in spring 2020 with their disengagement on the assessment administered in-school during fall 2019.
Topics: Equity, Innovations in reporting & assessment, School & test engagement
The MAP Growth theory of action describes key features of MAP Growth and its position in a comprehensive assessment system.
Products: MAP Growth
Topics: Equity, Measurement & scaling, Test design
Bayesian uncertainty estimation for Gaussian graphical models and centrality indices
This study compares estimation of symptom networks with Bayesian GLASSO- and Horseshoe priors to estimation using the frequentist GLASSO using extensive simulations.
Topics: Measurement & scaling
