For one thing, it means low performance on a handful of federal measures.

Today, Washington Monthly released its college rankings and added four different takes on “America’s Worst Colleges.”  Five Virginia colleges were distributed across the four lists. One of these is Ferrum College, which is where my son attended and graduated. While I am not writing so much to defend  Ferrum, I am going to use it as an example as to the limits of metrics since I think I have enough information feel good about the quality of education there.

Washington Monthly does a separate article about Ferrum in “Held Accountable” to describe the type of school that should fear Obama’s college rankings. It is a fair article and it references the data we produce and publish at SCHEV.

But this explanation goes only so far. Nationally, there is a lack of data that can sharply distinguish between schools that are dropout factories and those that largely serve as feeder schools to more elite institutions. But in Virginia, making that distinction is far easier. That’s because it is one of a handful of states that keep track of whether college students who leave one in-state institution ever wind up graduating from another. And it also tracks what their earnings are if they go on to work in the state after graduating. What this information reveals about Ferrum is not pretty.

No, it is not particularly pretty. Washington Monthly used our data well, and used a pretty full spectrum of the data.

For example, only 22.8 percent of seventeen- to nineteen-year-olds who started their college careers at Ferrum in 2003-04 graduated within four years from Ferrum or from any other public or private nonprofit institution in Virginia. Yet 43.3 percent of their counterparts at all other private nonprofit four-year institutions in the state graduated within four years. Even ten years later, the 46 percent of students who started as seventeen- to nineteen-year-old freshmen at Ferrum had still not graduated from any college in Virginia, compared to a ten-year in-state graduation rate of 62.3 for students who previously attended Virginia’s other private four-year colleges.

Are there schools that do worse on this one metric than Ferrum? Yes, there are a handful. Historically black Norfolk State University, for example. But what makes Ferrum really stand out is the way that it scores consistently low across a broad range of performance measures. For example, Ferrum is a pricy place for what it offers. Even with grants and scholarships, the average net price to attend the college and live on campus still came to $19,324 in 2012. Largely as a result, fully 91 percent of Ferrum graduates take on student debt. This is far higher than the average borrowing rate for all private nonprofit four-year institutions in Virginia, which comes to 69 percent. Multiplying these borrowing rates by the amount of debt incurred by each graduate yields a weighted debt average of $26,169 at Ferrum, compared to an average $18,910 for all private nonprofit institutions in the state.

However, there is a bit more to the story.

First, any institution with less than a first-to-second year retention rate less than 60% is in trouble. As regular readers know, that is first of two warning measures for an at-risk institution. Second, Ferrum is an undergraduate-only institution with just over 1,500 students. That is well below my second threshold of 2,000 students. Ferrum has challenges, there is no doubt about that.

In 1992, fall enrollment was 1,231 students. In 1997 it hit 908, struggled to 951 the following year, and then stayed below a 1,000 through 2005.  The last eight years have been years of significant growth. Much of this growth has been in the first-year class, representing 585 of last fall’s 1,512 students. This is not an ideal mix.

I can prowl through the data and keep looking for bright spots. The fact is, relatively motivated students with family support can be successful at Ferrum, if they get through the first year. Of first-time, full-time students, 89%  who finish 60 credits with a “C” or better in the first two years graduate within five years. But this is only about 10% of the entering class. Similarly, 65.5% of transfer students who attempt 25 or more credits in their first year will graduate within five years. (I am using 2008 annual cohorts for these measures.)

When I look at graduation rates at Ferrum by income levels, I see that not even income is a good predictor of success, whether based on multiples of the poverty level (accounting for family size) or real dollar amounts. This is unusual, at least in Virginia. What this tells me is that my son’s biggest complaint about his time there, and the source of all his threats to transfer elsewhere, especially in the second year, was valid. There’s not enough to do.

This is a challenge for a small college in what is essentially the middle of nowhere.

Ferrum is a beautiful campus. The town, not so much. Little is really left of what it once was. The nearest community of any size, the nearest Walmart, is Rocky Mount, a full 12 miles away. Staff and faculty do what I think is an excellent job in providing activities and opportunities to keep students engaged. However, there are limits of funding and other resources. Students leave. Some transfer, many don’t. On my last visit to the area, the motel desk clerk recognized my last name and asked if the “big man Massa” that had been at Ferrum was my son. I said that was most likely the case and asked how he knew him.

He had been at Ferrum for a semester and did not go back. College was not for him. Based on other conversations in the area, I am pretty sure his is not an uncommon case. Ten percent (or more) of entering students don’t make it back for a second term. That’s a rather expensive proposition.

Ferrum leaders know they have a retention problem. It is not a secret. They have tried, and are trying, to address it. At least one non-academic attempt to fell flat a few years ago. In 20 years, the rate has never been good – barely over 60% just four times since 1992.

The accounts I heard from my son of the readings and work he did were not trivial. I don’t think there is an academic problem. I’m not even sure there is a support problem. I think there is simply not enough “there” there. One way or another, Ferrum has to solve this problem and build a community that students become invested in and don’t want to leave until they graduate.

Metrics can’t capture this, the best they can do is hint at something missing. At least, as long as the reader has additional information. I don’t think Ferrum has much to fear from the proposed ratings system as it does from its inability to retain more students. Of course, a rating system tied to aid eligibility without a chance for remediation would close Ferrum and dozens or hundreds of colleges across the country.

I wonder though what might happen to some of these campuses, out away from population centers should they close? I guess that is a post for another time.