In my last blog, I discussed the problems caused by disengaged test taking, and how we detect its occurrence when students take MAP Growth. Today, I want to share what NWEA is doing to address the problem – now and in the future.
As I noted last time, in the vast majority of instances, students appear to show good effort throughout their MAP Growth test events. Sometimes, however, students become disengaged when taking MAP Growth, and disengagement usually occurs during only part of the test event. Such disengagement can negatively distort RIT scores, and the more that disengagement occurs, the larger the amount of distortion.
We can spot disengaged test taking by examining the time it takes students to answer individual items. Disengaged students taking a multiple-choice assessment like MAP Growth will show a characteristic behavior—called rapid guessing—of quickly answering an item faster than they could have had time to fully read and understand its challenge. This implies that rapid-guessing behavior is an indicator of disengaged test taking. Using response time, we can evaluate every response a student gives during a MAP Growth test event and classify it as either a rapid guess or a solution behavior (i.e., an effortful response). By aggregating these classifications across all of a student’s item responses, we can assess their overall test engagement.
Using this method of detecting disengaged test taking, NWEA has created new features in MAP Growth that we believe will be very helpful to our partners. Because rapid guessing can be detected in real time during a test event, two key features became possible.
The first one is, in many ways, the most exciting because it promises to actually reduce the amount of disengaged test taking that occurs. During a MAP Growth test event, if a student exhibits a specified pattern of disengagement (currently defined as three rapid guesses in a row), a notification will appear on the test proctor’s computer console indicating which student has become disengaged. When proctors receive such a notification, they are asked to quietly approach the student and encourage them to re-engage and to give their best effort to their MAP Growth test event. This type of proctor intervention should eliminate much of the subsequent disengagement that would have occurred had the proctor not been notified. (You can find more detailed instructions for proctors here in our community.)
The second feature is a change in the way that MAP Growth selects the items it gives students. MAP Growth is a computerized adaptive test. It adapts the difficulty of the items to match the student’s achievement level, based on the student’s performance on items given earlier in the test. Rapid guessing, however, confuses item selection, because rapid guesses are correct only about as often as random guesses, and make it appear that the student is of a lower achievement level than they really are. The item selection method has now been improved so that rapid guesses no longer affect item selection, which preserves MAP Growth’s efficiency and the precision of its RIT scores.
These new features represent NWEA’s best efforts at successfully dealing with the validity threat posed by disengaged test taking, as we strive to provide the most valid information we can to our partners. Our partners, however, also play a key role in this pursuit of valid scores. They choose when and where MAP Growth test events occur, how they are framed for the students, and how they are proctored. All of these factors can affect student engagement (and thereby affect score validity), and we provide guidance and other resources intended for partners to do what they can to maximize the test taking engagement of their students.
The NWEA research that drives innovation in our products doesn’t end there – stay tuned for future updates to MAP Growth based on our work in the area of student engagement. You can read more about what NWEA research is doing in this EdWeek piece, New Tool Alerts Teachers When Students Give Up on Tests.
