This Friday is the deadline to sign up for the next administration of the SAT. According to the test prep industry, it’s best for high school students to start off by taking the test in the spring of their junior year. That way, they’ll have plenty of time to practice and retake the test throughout the summer and the fall, before they have to send in their college applications. Students who sign up for a private study course are even “guaranteed” to see improvement, with a boost of 200 points or more.
Critics of standardized testing cite this supposed coaching effect—and the unequal access to its benefits—as a major reason the system tilts in favor of the richest kids and should be reformed. Their claims have been amplified in recent weeks, as the “Varsity Blues” college admissions scandal revealed how well-off families connive to get kids into selective colleges. “Rich Parents Have Plenty of Ways to Game the U.S. Education System,” read a Bloomberg headline, on a piece that noted how even basic test prep courses may cost $1,000 or more. Bob Schaeffer, the public education director of FairTest, has lately made the media rounds calling high-priced tutoring a form of “test-prep steroids,” and the existence of high-priced tutors has been cited in numerous articles on whether it might be time to abolish standardized testing for college admissions.
It would be useful to know, in the midst of this debate, how much of an effect these test prep programs really have.
For many decades, the testing industry denied that it was possible to raise a student’s scores to a significant degree. If coaching worked, wrote Nicholas Lemann in his essential history of testing from 1999, then “it made the SAT look like a series of parlor tricks and word games, rather than a gleaming instrument of scientific measurement.” Yet it’s clear that even the most elementary form of test prep—simply retaking the exam, multiple times—seems to have some benefit. The best independent research suggests that formal coaching can further boost a student’s score, but only by a little bit.
It’s been harder to figure out the exact size of that “bit,” and whether any subset of test-prepping students might be extra likely to improve their results. Large-scale studies have gathered information on students’ self-reported preparation activities for standardized tests and compared them with their final scores. Observations of this kind are prone to bias and confounding, though. For example, kids who enroll in test prep courses and stay enrolled until the end are likely to be very different in a lot of ways from the kids who don’t. They may come from different backgrounds, with different aptitudes, motivations, and tendencies. Once scholars control for all these factors as best they can, they find that coaching has a positive but small effect: Perhaps 10 or 20 points in total on the SAT, mostly on the math section, according to careful work by Derek Briggs of the University of Colorado Boulder and Ben Domingue of Stanford University.
Some estimates run a little higher: A 2010 study led by Ohio State’s Claudia Buchmann and based on the National Education Longitudinal Study found that the use of books, videos, and computer software offers no boost whatsoever, while a private test prep course adds 30 points to your score on average, and the use of a private tutor adds 37.
A few years ago, the College Board itself came out with its own online practice tool for the SAT, created in conjunction with the Khan Academy and free to use for any student. In 2017, the College Board and Khan announced that kids who used this tool for just six to eight hours saw a 90-point improvement in their scores, while those who studied for more than 20 hours saw their scores go up by 115 points. The details of this study, which appears to show a dose-response relationship for test prep, were never published, though, and Briggs refers to it as a “quasi-experiment.” Its main results have been grossly overstated too: Yes, students’ scores went up after studying—but they also went up for the control group. According to the data as they’ve been described, the scores of kids who completed exactly zero hours of official practice still went up by 60 points from one test-taking to the next.
The figures drawn from more credible, independent research suggest a trivial increase—a small fraction of a standard deviation. “From a psychometric standpoint,” wrote Briggs in 2009, “these effects cannot be distinguished from measurement error.” (When the College Board checks the noise in scoring on the SAT, it finds a standard error of 32 points.) At the same time, there’s a lot we still don’t know.
Many of the key studies of coaching effects were based on kids who took the SAT or ACT in the 1990s and 2000s, Briggs told me. The tests have since been changed a fair amount, in an effort to make them more like statewide achievement tests and to minimize their similarity to intelligence tests. There’s also been a blurring of the line between test prep coaching and conventional, cumulative learning of material. We might say that “coaching” is a form of short-term cramming—and that its effects will be specific to a test, and temporary. That’s certainly the case for old-fashioned Princeton Review–type tricks and the like. But how should we categorize the behavior of students in New York City who spend “months or even years,” per the New York Times, preparing for their high school entrance exams? If their years of work result in higher scores, is that a form of a “test-prep steroids,” too?
In any case, even small effects can be unfair. Let’s assume the effects of short-term coaching are really just a 20- or 30-point jump in students’ scores. That means they ought to be irrelevant to college admissions officers. Briggs found otherwise, however. Analyzing a 2008 survey conducted by the National Association for College Admission Counseling, he noted that one-third of respondents described a jump from 750 to 770 on the math portion of the SAT as having a significant effect on a student’s chances of admissions, and this was true among counselors at more and less selective schools alike. Even a minor score improvement for a high-achieving student, then—and one that falls within the standard measurement error for the test—can make a real difference.
Compare that with students from families that earn less than $20,000 per year, who have a mean score on the math portion of the SAT around 450. According to the same admissions counselor survey, a 20-point improvement to a score in this range would have no practical meaning for students who are trying to get into more selective schools. Even when it comes to less selective schools, just 20 percent of the counselors said a boost from 430 to 450 would make a difference. In other words, research has debunked the myth that pricey test prep gives a major bump to students’ scores, but it’s also hinted that whatever modest bumps they do provide are more likely to help the people who are already at the top.
