Problems with Newsweek's ranking of America's best high schools

Via the Volokh Conspiracy, Newsweek has released a ranking of the best high schools in America (Story, Ranking). Orin Kerr notes that they also did a ranking in 2003 and,– if memory serves correctly, they were doing this back in 2000, as well.

Nothing’'s changed since 2000. Newsweek’'s ranking is still absurd. Here’'s their formula for discovering which of America’s high schools provide the best education: take the number of Advanced Placement exams administered each year and divide it by the number of graduating seniors.

That’'s it. According to Newsweek, the only two things we need to know in order to judge a school’'s quality are the number of tests its students took and how many students there are.

There are so many problems with Newsweek’'s methodology that it’s hard to know where to start.

First, the absurdity of Newsweek'’s ranking system should be obvious just by looking at the scores of the top ten schools. Number one, the best school in America according to Newsweek, has a score of 10.755. Number ten, still a rather elite school, one would think, puts up 5.085. Does this mean that Jefferson County is twice as good as Hillsborough? In the absence of any further explanation of what exactly the score means, this would seem to be the conclusion. Given that there'’s no way that, among America’s elite high schools, any given school is twice as good as another, this is a key flaw in Newsweek'’s presentation and methodology.

Using Newsweek’'s metric, schools are rewarded for their students taking lots of tests. How those students do on those tests is irrelevant. A school'’s students could be getting 1s and 2s on all their AP tests (out of a possible score of 5), but as long as they’'re taking a lot of them, the school looks good. If a school wanted to improve its ranking, the strategy'’s painfully clear: encourage their students to take more AP tests. Not, you'’ll note, provide an environment where their students can learn the material and thinking skills necessary to do well on an AP test. Just get them to take the test. Ignoring performance is a huge problem.

Even if Newsweek modified their ranking index to account for students’' performance on AP tests (say, by looking at both the average scores achieved by their and the total number of exams administered), they still have an incredibly simplistic notion of what makes a good school. For one thing, AP tests are not offered at all schools. While I'’m fairly certain most students have the opportunity to take AP tests, they may have to take them at different schools. Newsweek would have you believe that schools that offer AP tests are inherently better than those that do not, but there’'s no obvious reason to believe this. Maybe the school places little emphasis on preparing for exams, preferring to teach their students in a low-stress, non-competitive environment. Or maybe it has a non-traditional curriculum that doesn’'t map well onto the AP tests. In any case, there'’s absolutely no reason why great teaching can'’t take place in a school that doesn’'t offer AP tests. For Newsweek to discount this possibility excludes potentially top schools from their ranking.

And what about all those students who don’t take AP tests at all? At my high school, there were three primary “tracks” in each of the major academic subjects; typically only students in the top track took AP tests, and not even all that many of them did. But what about the education provided to the students in the lower tracks, the average students? Newsweek'’s ranking pays no attention at all to them, focusing instead on the elite students and how good their education is (except not really, for the reasons outlined above). But a school’'s quality is signaled by more than the performance of its top students. To assess how effectively a school educates all its students, we should look at things like graduation rate, what percentage of graduates go on to further education, and the like. If a parent of an median student were to use this ranking as a way of determining where to send their child, they’'d want to know how well the school taught the average students, not its best performers. Again, Newsweek’s ignoring a large set of data that'’s clearly relevant to determining the best schools.

Then there'’s the question of non-educational factors. Surely a school'’s location, facilities, and extracurricular activities must factor into any assessment of its quality. Unless, of course, parents care only about sending their kids to a school where they will take lots of tests. Now, I’'m not a parent, but I'’m guessing that'’s not what most parents want.

You almost have to admire the ambition of Newsweek for presuming that they can objectively quantify the quality of American high schools. But it'’s hard to maintain that admiration when considering how simplistically and narrow-mindedly they try to do it. Looks to me like a ploy to follow the footsteps of U.S. News and World Report and sell magazines.