We’ve been encouraged by the attention received thus far from journalists, bloggers, and the general public about our study, “Do High Flyers Maintain Their Altitude? Performance Trends of Top Students,” published by the Thomas B. Fordham Institute. We’ve received some positive feedback, some constructive criticism, and some not-so-constructive criticism, too (Why yes, SamsDad67, all our parents were married, though in my own case it was a close call).
One criticism we have received is that our study failed to differentiate between students whose loss of high achievement status was due to school, home, or personal factors, vs. those students whose loss of status was attributable to normal “regression to the mean”. Such critics assert that it is reasonable, or even expected that high achieving young students will tend to grow less exceptional as they age, and that such tendencies are an incontrovertible law of nature.
Regression to the Mean is a real phenomenon. So are the Aurora Borealis and gravity. But none of these had any major impact on our study’s results. Regression to the mean can be thought of as an application of a statistical concept called the Central Limit Theorem, which mathematically describes the average tendency for any measurable process controlled by random forces to orbit a central (mean) value. In other words, if one tries to measure some attribute on multiple occasions, and that attribute is governed by random forces, then the measurements will tend to hover around an average value. When one tosses a pair of dice, for example, the sum of the two dice tends to be seven. Regression to the mean simply means that a roll of two or twelve (extreme departures from the mean) will tend to be followed by a roll that falls closer to seven, or the mean value.
What has all this to do with the high achievers in our study? Not that much, actually. Regression to the mean is a phenomenon seen in processes controlled by random chance…things like rolls of the dice, weather patterns, and my wife’s mood. To be sure, there is an element of random chance in the measurement of student academic achievement, which we call measurement error, and that measurement error will show regression to the mean. But the relevant mean is not the population average value, but the student’s “true score” on the achievement test (if it could be measured with fidelity). And that measurement error is quite small in the measures used in this study.
Think about it this way: If I’m a bowler who bowls an average of 180, any given game tends to hover somewhere in the vicinity of my average. I might bowl 170, 185, 160, then 190 but the scores tend to hover around my average, which is the best estimate of my true bowling ability. I may occasionally roll a 220, but when I do, the next score is very likely to be closer to my personal mean. My personal mean, however, does not have any connection to the population mean. That is, if my average is 180 and the typical bowler averages around 150, then I shouldn’t expect to see regression to the population mean (150) when I roll a 170, because the typical bowler’s average isn’t my own.
Our study examined the growth of high achievers over time. These were not average students. For critics to posit that one should expect high achievers to become less exceptional with age because of “regression to the mean” is to profoundly misunderstand what regression to the mean really means.
