Education Sector’s Mr. Aldeman, one of the prolific writers on The Quick and the Ed, has declared dead the usefulness of the SAT/ACT. It wasn’t his idea; he read it in Crossing the Finish Line:
Crossing the Finish Line has things to say about virtually every important factor in college life, but by far the most important thing is this:
The SAT and ACT do not matter in predicting college success.
I have been an unequivocal supporter of using the SAT/ACT* in making college admissions decisions (see here and here), but this sample of students and the rigor of this study are impossible to ignore.
No one should ignore what’s in Crossing, but I’m not about to gobble it hook, line and sinker.
The conclusions are based on a ton of data:
“Crossing the Finish Line, an important new book by former Princeton president William Bowen, former Macalaster College president Michael McPherson, and Matthew Chingos, relied on two massive databases on the entering class of 1999–one on 96,000 first-time freshmen and 30,000 entering transfer students at 21 flagship universities and the other on 108,000 freshmen and 42,000 transfers at less selective state colleges and universities in four states (Maryland, North Carolina, Ohio, and Virginia)–to compile a wide-ranging book of empirical research on topics impacting American higher education.”
Aldeman details his position-reversal on the value of these tests:
I have been an unequivocal supporter of using the SAT/ACT* in making college admissions decisions (see here and here), but this sample of students and the rigor of this study are impossible to ignore. Here’s what the authors found:
- Taken separately, high school GPA is a better predictor of college graduation rates than SAT/ACT score. This findings holds true across institution type, and gets stronger the less selective an institution is. High school GPA is three to five times more important in predicting college graduation than SAT/ ACT score.
- SAT and ACT scores are proxies for high school quality. When the authors factored in which high schools students attended (i.e. high school quality), the predictive power of high school GPA went up, and the predictive power of SAT/ ACT scores fell below zero.
- High school quality mattered, but not nearly as much as the student’s GPA. Other research, most notably on Texas’ ten percent admission rule, has proven this before. It’s somewhat counter-intuitive, but it shows that a student’s initiative to succeed, complete their work, and jump any hurdles that come up matters more than the quality of their high school.
Then he asks, “What should various actors do with this information?”
Time out.
As I wrote on the Quick’s blog entry, here’s why:
“Keep in mind that Alderman’s entire argument – and the authors’ – rests on the definition of “college success” being “graduation” or “obtaining a degree/certification.”
In theory, that’s sufficient. I’d prefer to talk about reality.
In reality, some degrees are watered-down and border on useless. At some institutions, the majority of programs fall into this category. If we pretend for a second that the degrees they award are little more than certificates of attendance and good standing with the Bursar’s Office, we do higher education reform a disservice.
Bowen, McPherson and Chingos, in a roundabout way, may have just proven not that SAT/ACT scores indicate nothing, but that high school GPA-as-harbinger means higher education is increasingly mimicking the weakness of the average American public high school.”
Mr. Aldeman et al.: Get serious about what a degree means – and what it doesn’t – and then we’ll get to work on the value of the ACT/SAT. Until then, I’m not about to worship at the altar of Crossing along with the EdSectorites.
Come to think of it, Education Sector could profit a bit from ACTA’s What Will They Learn?
{ 2 trackbacks }
{ 8 comments… read them below or add one }
Dan Willingham 11.04.09 at 10:01 pm
I’m no expert on this, but the prospect of using “high school quality” and GPA makes me nervous. My understanding is that the idea of a national test was developed exactly to give kids from lousy high schools a chance to compete against kids from Dalton and Choate. An admissions officer might discount the 3.9 average of the kid from Podunk HS in Virginia and take the kid with the 3.0 from Groton.
Chad Aldeman 11.05.09 at 1:50 am
Matthew, you’re right that we can’t measure “college success” much better than “obtaining a degree.” That’s unfortunate, but you have to remember that almost all previous studies have defined “success” as “first-year college grades” or “one-year retention rate.” Surely you’d agree that success is closer to graduation than it is to those interim measures.
Dan, one of the original purposes of the SAT was to find talented students from unknown high schools to attend Ivy League schools. But now it’s used as a giant sorting mechanism to dole out college placements. That should be enough to make anyone nervous, especially now that we have really good evidence that it’s not predicting what it says it does.
Anon 11.06.09 at 6:10 pm
There’s also the fact that by the time students arrive at college, they have already been partially sorted by means of ACT/SAT scores. The high-scoring ones have tended to be accepted by selective institutions, middle-scoring ones by less-selective ones, and low-scoring ones by open-admission colleges and unversities. So it might be accurate to say that within a specified range of test scores, high school grades are more predictive of college graduation, but I sure hope the author isn’t saying that students who score below 19 on the ACT are likely to graduate from Cal Tech if they have good grades from that Podunk HS.
Barry Garelick 11.11.09 at 8:36 pm
I’ve always been confused about what SAT/ACT is supposed to predict. I had read long ago that it correlated highly with grades achieved in freshman year–not whether the student graduated. I found this study (http://professionals.collegeboard.com/profdownload/pdf/RR%2088-11.PDF)
on the College Board’s website done in 1988 looking at whether high school GPA or SAT scores were better predictors of “college success”. Reading through the report, “college success” appeared to be freshman grades.
Matthew K. Tabor 11.12.09 at 5:10 pm
Dan,
The simplicity with which some are looking at both tests and GPA is appalling. I read today an article pointing to Aldeman’s original piece that was titled, “Idea of the day: ditch standardized tests.”
We need evaluators to commit to intellectual honesty regarding what these tests – and GPA – mean. What we’re seeing is folks sticking to evidence on meaning when it’s convenient to their argument, ignoring meaning when it isn’t, and weaving in context as they see fit.
The debate tells us a lot about how some of the most respect names/think-tanks view education.
Matthew K. Tabor 11.12.09 at 5:11 pm
Chad,
I’ve responded to your comment here:
http://www.matthewktabor.com/2009/11/12/i-dont-want-to-ruin-my-gpa/
Matthew K. Tabor 11.12.09 at 5:11 pm
Anon,
I can’t tell what Aldeman is trying to say. I suspect that we’ll have a better handle on it when he figures it out, too.
Matthew K. Tabor 11.12.09 at 5:13 pm
Barry,
The love affair with what the SAT predicts seems to take a backup role to what the test shows. Common sense dictates that we’d get a handle on meaning before we use it to predict with any value; unfortunately, as we well know, common sense rarely reigns.