Further thoughts on Newsweek's high school rankings

After poking around the Newsweek website a bit, I came across this FAQ on their ranking of America's top high schools. Jay Mathews, the creator of the ranking, addresses some of the issues I brought up but I remain unconvinced that the ranking provides a useful way of measuring schools' effectiveness.

When asked about his decision to ignore test performance and examine only how many tests each school administered, Mathews claims that "most American high schools keep their passing rates artificially high by allowing only A students to take the courses. In some cases, they open the courses to all but wrongly encourage only the best students to take the tests." This doesn't fit with my own experience. At my high school, students could take as many Advanced Placement tests as they'd like; in fact, they typically had to take initiative in registering for the tests (caveat: this was back in the late '90s and things may have changed by now). Even in classes that had the AP label, plenty of students (probably a majority) did not take the AP exam. Now as Mathews views it, the fact that most students didn't take the test reflects poorly on my school's educational effectiveness. But the important question is not whether students take a test, but rather whether they actually learn anything.

Mathews appears to be on the same page here, arguing that a rigorous academic environment is crucial to a school's success. He goes so far as to argue that "a student who works hard but struggles in an AP or IB course, and does poorly on the AP or IB test, is still better prepared for college than he would be if he were forced to take an easier course and test." If his argument is that the best high schools for top students are those that challenge them, he has no disagreement from me. I'm all for hard classes.

But given his belief that the mark of a good school is its difficult classes, it's hard to grasp why Mathews places so much (indeed, all his) emphasis on the AP tests that come at the end of the year. The real value of challenging course work lies in its development of critical thinking skills and broadening of knowledge, not in its providing students the skills to take a test at the end of the year. Yet Mathews clearly believes that the test-taking itself is crucial: "Taking the test is good. It gives students a necessary taste of college trauma. It is bad that many students in AP courses avoid taking the tests just because they prefer to spend May of their senior year sunning themselves on the beach or buying their prom garb." A few comments about this. First, the trauma of AP tests is nothing like the trauma of beginning college. If Mathews really believes that taking a three-hour test with lots of multiple choice questions really prepares you well for college, he doesn't have a clear understanding of the actual stresses associated with beginning college. Believe me, exams aren't even near the top of the list. On a somewhat related note, I think Mathews is mistaken in assuming students in AP classes who don't take the exams are just a lazy bunch. To be sure, there's plenty of such people. But for others, there's often no good reason to take the exam - their college might not grant credit or placement, so why shell out $82 for a test that doesn't "get" you anything. Assuming the class is taught well, the benefits have accrued over the course of the school year, not in a few weeks of studying in May.

None of this is meant to suggest that AP is a bad program. I took a bunch of AP classes and tests in high school and learned a great deal in the process. If I had to do it all over again, I'd still probably take lots of AP exams. But there was no major differences between AP classes and non-AP (but still top-level) classes. Indeed, in some of my AP classes, practically no time was spent explicitly preparing for the AP exams. When I think back to the best classes I had in high school, I don't think about it in terms of AP and non-AP classes. The best classes were the ones with good teachers who cared about their students and challenged them to reach their potential. While the AP designation might signal that a particular class is top-notch, its absence in no way suggests that the class is necessarily less rigorous or teaches students less. Thus, while Mathews's "Challenge Index" might successfully identify good schools, it doesn't even come close to determining which schools are best.

And, as this St. Petersburg Times article suggests, Mathews's method doesn't even find good schools all of the time. Hillsborough High School, according to Matthews and Newsweek the tenth best high school in the country, received a D from the state of Florida last year. According to GreatSchools.net, Hillsborough's graduation rate for the 2002-3 school year was 60.8%, almost ten points below the Florida average (which is saying something, if you know about the state of Florida's public school system). It's hard to reconcile these data with Mathews's ranking of Hillsborough. Are we really to believe that a school that receives a D and has a graduation rate of 60% is one of the ten best in the country?

There you have it. Mathews methodology is useless. It doesn't identify the country's best schools and it doesn't even appear to highlight good schools. What it does is exactly what it says it does: it rewards schools whose students take a lot of tests. Is this really an ideal vision of American education?