The ratings (framework) are out (more promises actually)! I wrote my semi-formal response over at my work blog. In that post I reference Stephen Porter’s post on why a single institutional performance metric is exactly like the Holy Grail. I’m kind of stuck on this comparison, and not because it arose as a response to Bob Morse of US News & World Report. Really, I am just a big fan of the Arthurian legend.

If one accounts for the general imperfection of law-making, it is not too difficult to believe that Richmond, VA is the real Camelot:

A law was made a distant moon ago here:
July and August cannot be too hot.
And there’s a legal limit to the snow here
In Camelot.
The winter is forbidden till December
And exits March the second on the dot.
By order, summer lingers through September
In Camelot.
Camelot! Camelot!
I know it sounds a bit bizarre,
But in Camelot, Camelot
That’s how conditions are.
The rain may never fall till after sundown.
By eight, the morning fog must disappear.
In short, there’s simply not
A more congenial spot
For happily-ever-aftering than here
In Camelot.

Law-making is imperfect. Often what the General Assembly decrees is not quite what happens, so really, it is just not much of a stretch to imagine the Quest for the Holy Grail occurring in the green hills of Virginia. I’ve walked much of the Appalachian Trail in Virginia, and at night, in the mist, on the trail or on the Blue Ridge Parkway, I have had little difficulty hearing the distant hoofbeats of a quest.

This is perhaps all the more true as I consider the Commonwealth’s endeavors over the last decades in developing and packaging performance indicators. While, I have played a role in those efforts the last 14 years, I have always tried for a package of measures, generally more than fewer. Institutions are simply too complex to be represented by a single aspect, let alone a single measure.  In fact, discussion such measures quickly become a rather intense and political discussion.

But now we have the a framework for the Postsecondary Institution Ratings System and the excitement was just like I suggested a couple weeks ago in tying PIRS to the arrival of the new phone books. We also have some new goal statements. For example, in a blog post, Jamienne Studley (of the It’s Just Like Rating a Blender comment) says:

The development of a college ratings system is an important part of the President’s plan to expand college opportunity by recognizing institutions that excel at enrolling students from all backgrounds; focus on maintaining affordability; and succeed at helping all students graduate with a degree or certificate of value. Our aim is to better understand the extent to which colleges and universities are meeting these goals. As part of this process, we hope to use federal administrative data to develop higher quality and nationally comparable measures of graduation rates and employment outcomes that improve on what is currently available.

So, we have language of equity,  affordability, combined with the new phrase of the realm the last year “certificate of value,” to describe the new goals being assigned to institutions. Some may/will argue this point, but the reality is that not all institutions were founded to be affordable, let alone open to all, or even “helping” students graduate. Some institutions, particularly one small college in the PNW, have been (please note the use of the past tense) famously proud of their low graduation rates. Completion was seen as a mark of distinction among super-smart and well-qualified students. But, these are all worthy goals and those footing the bill (or a large chunk of it through gifts and financing) get to make the rules. That is the Golden Rule: He who has the gold makes the rules. Also, sticking to our Arthurian theme, Might Makes Right.

“we hope to use federal administrative data to develop higher quality and nationally comparable measures of graduation rates and employment outcomes that improve on what is currently available.”

So, they are going to use the National Student Loan Data System (NSLDS) to create measures of graduation rates for Title IV students. This means they will build estimates that assume first appearance as Title IV recipients will be first enrollment in college. For many students this will work, but not all. To incorporate transfers into the mix will be a much greater challenge, particularly those from California community colleges where so very few students use Title IV to attend. There will be some estimation possible using annual loan amounts since maximum subsidized Stafford loans are different for third and fourth year students. However, the great many students transferring in fewer than two years from a community college will be damned hard to identify.

Of course, all this can be fixed going forward by making changes to the NSLDS collection.

Using NSLDS data to match to Social Security earnings is already tested at the program level for Gainful Employment. It should not be a stretch to do that at the institution level. The interesting thing will be to see how this figures compare to what states like Virginia, Texas, and others are reporting using UI Wage data. And Payscale data. I don’t know about my colleagues in the other states, but I am ready to assist.

To do this well though, they are really going to have to do more than ask for comments. They need to bring people together. (About 2 minutes in on the next clip.)

I appreciate what the president is trying to do. I just don’t think ratings are the way to go for a government. Save with this caveat: as long as the ratings are billed and described solely as Title 4 Performance Ratings and not Institutional Ratings, then I am happy and fully supportive. I have said all along it is completely appropriate for the Department to evaluate institutions based on their performance under Title IV. Program evaluation is part and parcel to government programs. Or should be. Let’s just keep the focus where it belongs and not try to be all things to all people, especially when neither the data nor the legitimate bounds of authority warrant more than that.

In any event, the Student Right-to-Know Before You Go Act is a better solution to the goals Studley’s post articulate and the goals presented within the draft framework. Better data,  better information, within an appropriate scope.

We can achieve a version of Camelot in the cult-word of higher ed data.

Just choose wisely.