Inside Higher Ed, Methodology Change for Doctoral Rankings:
The National Research Council — responding to criticism it received in the internal peer review of its forthcoming doctoral program rankings — is changing the methodology in a few key places for the long-awaited project.
The changes — which are not yet final — are likely to divide the main ranking of each program into two separate rankings — one based on explicit faculty determinations of which criteria matter in given disciplines, and one based on implicit criteria. Further, the council is likely to release ranges of ratings for a 90% "confidence level," not the confidence level target of 50% that was in the methodology released last year. …
On the question of the ranges to be reported, [Jeremiah P. Ostriker, chair of the committee overseeing the project and a professor of astronomy at Princeton University] said that the committee has long wanted to avoid the "spurious precision problem" of previous rankings in implying certainty that a given program is a precise number in relation to all others. … He noted that "commercial" ranking efforts tend to give a single number, "but that's no excuse for us making the error." …
Robert Morse, who directs college ratings at U.S. News & World Report (including a graduate program ranking each year), said he also saw serious credibility problems with basing rankings on information that "is getting old and stale."
U.S. News primarily uses "peer evaluations" for graduate school rankings (in essence surveying faculty members on what they think of other programs) although more complicated formulas are used in some fields. Morse acknowledged that critics call his magazine's formula "simplistic," but he questioned whether the NRC was going too far in the opposite direction.
"They seem to be trying to produce something so sophisticated and complicated and nuanced that they think will give it credibility in the marketplace. I just wonder whether anyone's going to understand it," Morse said. "Do you need a Ph.D. to understand it? If you can't understand it, I just wonder whether it's going to be accepted."
