Our MAP® Growth™ assessment is designed to provide educators with useful information about what students know and can do, and what they are ready to learn next. But having a well-designed test simply isn’t enough to guarantee valid scores. It also requires students who are engaged during the test and try to do their best. Few people would argue with the idea that, if you don’t try your best on a task, you probably won’t perform as well as you are capable of. This idea is certainly true in test taking.
The good news is that the vast majority of students taking MAP Growth show good engagement. It is also true, however, that students sometimes become disengaged when taking MAP Growth. This problem is by no means unique to MAP Growth; all educational assessments are vulnerable to disengagement.
Disengaged test taking usually leads to scores that underestimate what the student actually knows and can do. The degree of underestimation depends how disengaged the student was, but in extreme cases RIT scores can be distorted by 50 points or more. This can have serious implications. If enough distortion occurs, a student might be put into a less-advanced instructional program that they don’t really need. This also has opportunity costs because time spent in an unneeded instructional program is time wasted — in that the student is not receiving instruction on material they are ready to learn. In this way, disengaged test taking leads to wasted educational time and resources.
NWEA researchers have done extensive studies of the prevalence of disengaged test taking and its impact on RIT scores. This research, much of which has been published in scholarly research journals, has produced several general findings about MAP Growth:
- Disengagement is low in early grades, but tends to increase in frequency across grades
- Disengagement occurs more often in Reading than in Math
- At any grade/subject combination, boys show disengagement about twice as often as girls
- The later in the day a test event occurs, the more likely disengagement is to occur
- Students often move between engagement and disengagement during a test event
One interesting outcome of disengaged test taking on MAP Growth is that a growth score across two occasions can be affected in two different ways. If a student was disengaged during the first occasion, the growth score will be exaggerated—often to an unrealistic degree. We have observed fall-to-spring growth scores exceeding 40 points which—while appearing impressive—are simply not credible because they reflect an amount of growth that is extremely unlikely to have actually occurred. In contrast, if disengagement occurred during the second occasion, the growth score will be much lower. In fact, it will often show a large negative value, falsely suggesting that the student knew much less in the spring than they did in the previous fall! Thus, disengaged test taking can distorts growth scores in different ways, and can provide confusing information to educators about their student progress.
But how do we know when (and how much) disengagement occurs? The answer lies in the fact that MAP Growth presents multiple-choice items that must be answered for a student to move on to the next item. Because they cannot simply omit answering items to move through the test, disengaged students tend to show a characteristic behavior of quickly choosing an answer faster than they could have had time to fully read and understand the challenge posed by the item. This behavior, called rapid guessing, indicates a disengaged response from the student.
MAP Growth records the amount of time a student takes to answer each item, which allows every item response a student gives to be classified as either a rapid guess or its engaged counterpart, solution behavior. Most students are fully engaged, exhibiting all solution behaviors during their MAP Growth test events, while some students exhibit one or more rapid guesses. The more rapid guesses that occur, the more disengaged the student was. In this way, the prevalence of rapid guessing allows us to measure how engaged a student was when they took MAP Growth.
At NWEA, our ability to detect rapid guessing in MAP Growth has led to several new and emerging features for effectively managing the problems posed by disengaged test taking. We are excited about these groundbreaking innovative features, as we are the first assessment organization to address this problem. In my next blog, I will overview the new MAP Growth features and discuss their use.
