What I said to the GAO

Should the Federal Government Rate Colleges (and Universities)?

President Obama, Secretary Duncan, Deputy Under-Secretary Studley, and others, have called “foul” on those of us opposing the proposed ratings system since it does not exist. Their position is that our response should be constructive and supportive until there is something to criticize.

They don’t understand why many of us, the data experts, are opposed.

It’s not that a ratings system can’t be developed. It can. At issue is the quality and appropriateness of the available data. The existing data are simply inadequate for the proposed task. IPEDS was not designed for this and organizations like US News & World Report have taken IPEDS as far as it can go and added additional data for their rankings. Also at issue is the appropriateness of the federal government providing an overall rating for an institution over aspects for which it has no authority.

I think it is fully a right course of action for the Department and Administration to rate colleges as to their performance with Title IV financial aid funding and required reporting. Ratings based on graduation rates, net price, and student outcomes of students with Title IV aid would be very useful and appropriate. Placing additional value factors in ratings relevant to compliance and accuracy of required reporting under Title IV would add meaning to a system designed to determine continued eligibility for Title IV programs.

After all, if continued eligibility and amount of available aid under Title IV are the ultimate goals, aren’t these the things to measure?

This is decidedly less politically exciting than saying Institution A has a higher overall rating than Institution B, but it makes a clear relationship between what is being measured and what matters. It also has the advantage of being able to use existing student-level data on Title IV recipients in the same manner as is being done for Gainful Employment. From where I sit, PIRS is simply Gainful Employment at an institution-level as opposed to program-level.

And that is appropriate.

Using ratings to develop performance expectations to participate in the federal largesse that is Title IV would be a good thing. Regional accreditation and state approval to operate is clearly no longer adequate for gate-keeping, if, indeed, it ever was.

The difficulty is determining what those expectations should be. It is quite reasonable to subdivide institutional sectors in some manner, calculate a graduation rate quartiles or quintiles for each group of students in Title IV programs and require institutions in the bottom to submit a five-year improvement plan with annual benchmarks. Any institution failing to meet annual benchmarks two years running could then be eliminated from Title IV. Using multiple measures of success, including wage outcomes from records matched to IRS or SSA, we can reduce any tendency towards lowering standards to survive.

In an ideal world, with quote complete end quote data, a ratings system would be focused on intra-institutional improvement. In fact, this is the language that Secretary Duncan is beginning to use, as he did in a recent interview:

“Are you increasing your six-year graduation rate, or are you not?” he said. “Are you taking more Pell Grant recipients [than you used to] or are you not?” Both of those metrics, if they were to end up as part of the rating system, would hold institutions responsible for improving their performance, not for meeting some minimum standard that would require the government to compare institutions that admit very different types of students.

The problem is that simple year-to-year improvement measures tend to not be very simple to implement. We have substantial experience with this in Virginia, especially this week, as we work through institutional review of performance measures in preparation for next week’s meeting of the State Council. On any measure, annual variance should be expected. This is especially true for measures that have multiple year horizons for evaluation. It is even truer when institutions are taking action to improve performance as sometimes such actions fail.

A better approach is focus on student sub-groups within an institution. For example, is there any reason to accept that Pell-eligible students should have a lower graduation rate than students from families with incomes greater than $150,000? We generally understand why that is currently the case, but there is no reason to accept that it must be so. I would argue, vociferously, that if the Department’s goal is to improve to access and success, that this is where the focus belongs. Rate colleges on their efforts and success at ensuring all students admitted to the same institution have the same, or very similar, opportunity for success. Provide additional levers to increase the access of certain students groups to college. To do this would require IPEDS Unit Record – a national student record system – perhaps as envisioned in the Wyden-Rubio bill, The Student Right-to-Know Before You Go Act.

This means over-turning the current ban on a student record system. It also means taking a step that brings USED into a place where most of the states are. From my perspective, it is hard to accept an overall rating system of my colleges from the federal government when I have far, far more data about those colleges and choose to not to rate them. Instead we focus on improvement through transparency and goal attainment.

I think few reasonable people will disagree with the idea of rating institutions on performance within the goals and participation agreement of Title IV. It is when the federal government chooses winners and losers beyond Title IV that disagreement settles in.

We will face disagreement over what standards to put in place, if we go down this path. That is part of the rough and tumble of policy, politics, and negotiated rulemaking. You know – the fun stuff.

Let’s take a quick look at four very different institutions. These images come from our publicly available institution profiles at http://research.schev.edu/iprofile.asp


Germanna Community College does not have high graduation rates (note these are not IPEDS GRS rates as they include students starting in the spring as well as part-time students). All of these are toward the lower end of the range of Virginia public two-year colleges.  There are a range of differences among the subcohorts, particularly between students from the poorest and the wealthiest families.


Even at the highest performing institution on graduation rates, one of the highest in the nation, there is still a range of difference. A full 10 percentage point difference between the poorest and wealthiest students.


In the last two decades, CNU has more than doubled its graduation rates through transforming the institution and its admission plans. The differences between subcohorts are much smaller, but this has come at a price of denying access to students that sought an open-enrollment institution.


Ferrum College has relatively low graduation rates and high cohort default rates. Using federal data, it does not look to be an effective institution. However, I will point out that it has the highest success rate with students requiring developmental coursework in the first two years. It apparently can serve some students well, and others better than other institutions.


My point with these four examples is this. We need to drive improvements in student outcomes by focusing on differences within institutions, specifically subcohorts of students that are recipients of Title IV aid.



NPR once went beyond headlines

Yeah, this song is on my mind as I consider the silliness of belief and the cost of higher ed.

I’m an innocent victim of a blinded alley
And I’m tired of all these soldiers here
No one speaks English, and everything’s broken, and my Stacys are soaking wet
To go waltzing Mathilda, waltzing Mathilda,
You’ll go waltzing Mathilda with me

–Tom Traubert’s Blues, Tom Waits

In the article, How Private Colleges Are Like Cheap Sushi, Anya Kamenetz writes about how private colleges discount tuition, through institutional grants in the form of student aid. Using economic theory of anchors and signaling to explain this behavior, she points to private nonprofit colleges making these choices to compete in a crowded marketplace. Nowhere does she mention that public institutions engage in the same behavior, albeit to a lesser degree. At least with in-state students. More and more we see tuition discounting for out-of-state students using some of the dollars from the significantly higher tuition paid by non-residents.

Years ago, I was part of the Technical Review Panel that established the net price calculations for IPEDS. They were never targeted at just the privates. We spent more time talking about how these calculations would apply to public institutions than any other single topic at the meeting.

In failing to mention public institutions and the subsidies they receive, it is hard to take the article as little more than a puff piece. As I have argued before, when we look at actual spending for comparable institutions, much of the difference between public and private disappears. (By the way, many of the people who argue with me about this are the same that used to argue that all graduate programs were heavily subsidized by undergraduate programs – which is not really the case they realize now. Professional graduate programs are for-profit centers in public and private institutions.) Public institutions tend to rely more heavily on adjunct faculty…as long as they are in a geographic region where they are available.

I’m not interested in giving the private colleges a pass. I am interested though in seeing people recognize the that two sectors learn from each other and frequently borrow strategies. There is a lot of cross-pollination that takes place.

No institution has a right to survive. Especially not the those that are expensive and have low graduation rates with low measures of student success beyond completion. That should just as readily apply to institutions in all sectors. Survival is not a right.

Let’s also keep in mind that a primary culprit here is USED. Through the use and control of the Free Application for Federal Student Aid (FAFSA), the Department inserts itself into any discussion about price and cost through the resulting Estimated Family Contribution (EFC) – regardless of its accuracy. I have heard enough financial aid policy people, both in and out of the Department and the White House, refer to FAFSA as a “rationing tool” far more often than as an “accurate depiction of what a family can afford.”

Actually, I have never heard the latter.

We have a situation where a couple of generations of Americans were fortunate to have access to a free, or almost free, college graduation. This was possible because they were willfully ignorant of how tax dollars made this possible.

An awful lot of people went to college from groups that never had the opportunity previously. This was a good thing. Many students across all groups made good use of the experience. Many did not.

Now that the tax support for higher education has dramatically been reduced per student, college is seen to be much more expensive and not worth the cost – often based on the experiences and outcomes of those that did not make the best of their college experience.

I think the realities are this:

1) College (or any education) is never worth more or less what one puts into it. Yes, the credential has a value, but sooner or later, one can’t hide behind the credential and must demonstrate knowledge and ability to keep learning.

2) A large part of the costs of college have little to do with the specifics of education. Living expenses, for one, followed by athletics and building costs. Athletics is the only one that can possibly be eliminated and that seems unlikely.

3) Even if we trimmed, chopped, all the non-education costs, that would probably just allow a return to a more accurate cost of education. How? Colleges would stop relying so much on adjuncts. Adjuncts are nice additions to the faculty when they are local professionals with lots of experience to add value to a program. When they are used to provide a large portion of the education as contingent, part-time employees, it is just another form of an abusive system that does not serve the institution or its students well.

I realize nobody wants to pay anything for education. Unfortunately, that is just kind of stupid. Good education is always going to be fairly expensive because it involves lots of human interactions with humans with advanced education that was expensive to achieve. We can continue to work to find less expensive ways to deliver education content and to assess mastery, but engaging, coaching, and mentoring will probably not get cheaper. Unless it is something left in the hands of androids and robots, but then I start thinking about government or school contracts for these androids and it is doesn’t seem that will actually cost less.

We’ll be fighting over the cost of education, especially higher education, for decades, I expect. Until we reach the point where we recognize that we have reduced the costs and trimmed the unnecessary expenses to the point we can’t go any further, institutions will continue to be creative in pricing and aid strategies. Just as every other sector of the economy does.

I’ve rambled enough through this, and probably shouldn’t publish it, but what the hell. It is essentially an allergic reaction to something stupid.

And you can ask any sailor, and the keys from the jailor,
And the old men in wheelchairs know
And Mathilda’s the defendant, she killed about a hundred,
And she follows wherever you may go
Waltzing Mathilda, waltzing Mathilda,
You’ll go waltzing Mathilda with me

And it’s a battered old suitcase to a hotel someplace,
And a wound that will never heal
No prima donna, the perfume is on an

Old shirt that is stained with b lood and whiskey
And goodnight to the street sweepers, the night watchmen flame keepers
And goodnight to Mathilda, too

About the Undeclared Major as a Dead Horse

…I have not finished beating it.

Quick recap: FSA is requiring all Title IV awardees submitted by institutions to have a valid CIP Code (Classification of Instructional Programs), including students in undeclared or undecided status. Virginia institutions have been running into difficulty because they have reporting undeclared students to us for over 20 years using code 90.0000, which is, alas, a Virginia-specific code. It is thus not found in the official CIP 2010 tables and students submitted with these codes have been rejected. USED is advising institutions to use 24.0102 – General Studies.

Apart from my argument that students who are not, by definition, enrolled in a program and should not have a valid CIP code – unless it’s definition is undeclared. This is not the case of 24.0102. However,while  I grant the Illustrative Examples of “Undeclared” and “Undecided” might seem to make it a logical choice, I think a step was missed.

What step? How about checking to see if degrees are awarded under 24.0102? After all, a BA awarded in General Studies would seem to be a pretty declarative statement, wouldn’t it? Wouldn’t it also seem that someone had decided that was an appropriate major?

In 2012, there were 84,118 degrees at various levels awarded nationwide based on the IPEDS Completions report.


In 1992, there were 24,357 degrees at various levels awarded with CIP 24.0102.

Oops. It is a growing problem.

In the grand scope of things, this is a relatively minor problem. Unfortunately it demonstrates how little thought is given to future data collections and future uses of data. Any analysis of these data will be inconclusive, misleading, or wrong, because researchers won’t be able to separate out students who were actually enrolled in a General Studies program v. those who were undeclared.

It would be so very easy to add 00.0000 or 99.9999 or some other code to the valid value table that is based on CIP Codes to avoid this problem. If the logic is that all degree-seeking (and thus potentially aid-eligible) students are enrolled in a program, this is just wrong. Some schools, including one of my previous employers, did not, possibly still does not, allow students to select a major until after the third semester. This allows them to experience more of the liberal arts and sciences before settling down to focus on just one.

I am curious to know if this was a conscious, intentional decision by the department, or a development choice by the contractor that was subsequently (and thoughtlessly) approved by the person running the contract.


Things that go Irk in both the day and night

Some things are just irksome. Tiny, piddly, little things. But they are often things that matter in a bigger way than is readily apparent.

I just finished posting on my work blog to provide guidance to institutions struggling with a very minor change in the data system USED uses to handle Title IV loan originations/disbursements. As of this spring, all students are required to have a valid CIP (Classification of Instructional Programs) Code for their program. Makes sense. However, what to do about students with an undeclared major? Certainly they should not be in a program, should they?

USED says yes. Further, they advise institutions to use 24.0102 General Studies (An undifferentiated program that includes instruction in the general arts, general science, or unstructured studies) because it comes equipped with Illustrative Examples of “Undeclared Major” and “Undecided.”

Fine. I have problems with the logic and whether or not this is accurate for all students to which the code might be applied, but okay, I can see how they got there.


How will the Department differentiate these students who are undeclared from students who are in actual General Studies degree programs? Virginia has 23 such programs at public and nonprofit colleges.

Um, they won’t and they can’t.  (By the way, this conflict arises in Virginia, and apparently only in Virginia, in that we use 90.0000 to report undeclared students to us. Everyone keeps telling me that only Virginia institutions are having trouble with this.)

How good will the data be when some yahoo tries to build a Gainful Employment equivalent for all programs when the enrollment and default rates and who knows what other data all get confused for the actual programs and the undeclared students (who are not in a program, by definition).

At the base of the academic structure is the academic program. USED does not seem to understand to this. Nor do they appear to be thinking ahead about their Title IV data collection and how is might will be used.



It doesn’t get much better

Yesterday, my son and I hauled the kayaks over to Kiptopeke State Park on Virginia’s Eastern Shore of the Chesapeake Bay. It is one of my favorite spots. Not because of the fishing – rarely have we done well – but just because it is a great little oasis. Just three miles from the tip of the peninsula, the park is located at the old ferry terminus that went across the mouth of the bay.

When we pulled in, the temperature was just about 80F with a north wind blowing at about 11 knots. We surveyed the conditions and crowd on the pier to see if folks were pulling fish up, and saw none. We decided not to head out to the grounded fleet of concrete ships right away with the chop and the wind after watching one kayaker working pretty hard to fish the shiDSCN0509ps. Instead, we launched and fished the shallower areas south of the pier where it was a bit calmer. This was about three hours before high tide.

Once we got into position, we were getting hits right away. Not much in the way of connections, until Zach landed a 20 inch flounder. But that remained the only fish for a while. There were enough hits though to keep things interesting. Both of us were working two rods – one with minnows, one with clam or shrimp. The latter options were generating the action.

We beached about an hour after high tide to stretch and fish from the sand. Sometimes we get a bit obsessive and will stay in the yaks for five or six hours without a break. Our record is seven. I was feeling the sun pretty well at this point and starting notice the stiffness in my shoulders that would be settling in later that night. The wind had remained pretty stiff and it was work maintaining position and despite that work, we were a good half-mile from the end of the pier.

We talked over our strategy and decided to check out the conditions around the ships, With luck, the might break a little of the wind.

DSCN0511We noticed about the half the way over the wind was slackening. This was great! We headed to the far north end of the concrete fleet dropped lines and started fishing, drifting up close to the hulls each ship. Zach pulled up a nice spot on the first cast. We spent the next couple of hours in near perfect conditions as the wind dropped completely. The bite was inconsistent, mainly a lot of oyster toads, including some that were quite large. Conditions had relaxed so much that I would occasionally look back and see Zach laying back and dozing, responding only to the occasional bite.  When we finished the northern group of ships, we drifted to the second group and fished until we realized the tide had fully turned and was now pulling us on to the ocean faster than the wind had. So, a few more casts.

Unfortunately I was rigged only for bottom fishing because I looked east toward the ramp, and I saw a heavy body, with a red-tinge, roll about 60 yards away. A big red drum had been cruising the area and that was the only look I got.

Overall, it was day that couldn’t really be improved. Sure, we could have caught more fish, but just getting out with my son was good enough. Easy drive over, relatively easy drive back. No problems, no issues. Just a great day. And the continuing of a “tradition” that started many years ago and continued through years of scouting – I drive back and he sleeps.

It was also pleasant to be totally disconnected from the world outside of what I could see and here. Phone locked away in the car and spending thought capacity on what I was doing at the moment. Not worrying about higher education. Not worried about the current projects, or the next projects. Not worried about Twitter.

It was good. As good as good can be.

Duncan doesn’t understand the opposition to PIRS

Duncan doesn’t get it. Apparently the president does not get it either if this is all coming from the top.

Duncan did ask the ratings system’s many critics to reserve judgment until they actually knew what it would look like.

“This system that people are reacting against doesn’t exist yet,” he said. “Tell us what you like; help us do this.… Let’s not be against something that has not been born yet.”

Read more: http://www.insidehighered.com/news/2014/07/03/arne-duncan-talks-about-ratings-and-student-debt-expansive-interview#ixzz36VOw7Wlh 
Inside Higher Ed 

One of the many things that the Administration does not get, is that some of us are objecting to the ratings system because the currently available data are inadequate. Completely inadequate. Bloody embarrassingly inadequate. These data, for the most part, are the same data that brought us to the current state of affairs.

Asked by Leonhardt to respond to the criticism that the government can’t rate colleges intelligently and effectively, Duncan reiterated that department officials know they have a hard job ahead. “We’re going into this with a huge sense of humility,” he said, and recognize that “intellectually it is difficult.”

Bullshit. Humility is not based on ignoring the advice you receive and doing it anyway.

“I say to the Department this, “We need better data. Let me rephrase that. YOU need better data. This should be the Department’s first priority.”

-Tod Massa (me), PIRS Technical Symposium, Feb 6, 2014.

The Department called together 19 experts in this arena to advise on the proposal. Perhaps it was just a show to support the pretense of listening and engagement. That’s fine, we all do that (every time we nod when our spouse is talking to feign involvement, for example). However, since the lack of humility irritates me, I am choosing to assume they wanted advice, but are unwilling to accept it. (It is too much like a smug, snotty adolescent saying “Uh-huh, I know” when you are trying to convey important information to him. I already have one of those in the house and another enroute.) By the way, push-back from those you intend to rate does not prove the rightness of your cause.

At least the interview demonstrates that the talking points, and perhaps the structure of the ratings system itself, are changing.

“Are you increasing your six-year graduation rate, or are you not?” he said. “Are you taking more Pell Grant recipients [than you used to] or are you not?” Both of those metrics, if they were to end up as part of the rating system, would hold institutions responsible for improving their performance, not for meeting some minimum standard that would require the government to compare institutions that admit very different types of students.

This is a step in the right direction, perhaps the second best approach. Encouraging a constant cycle of improvement is a good thing. Perhaps they will look to states that have experience in accountability measures of this type, you know, like Virginia. One of the things we know on this topic, is that there are annual variations that are little more than statistical noise, especially with smaller institutions. When such accountability measures are first introduced, it takes a number of years before institutional policy and operational decisions catch up. Thus, improvement on all measures every year is unlikely – unless the measures are weak to begin with. Remember, the key decisions about the graduation rates of the entering cohort of 2014 have already been made – who, how much aid, the institutional budget, and the student support programs that will be in place. The decisions impacting the next available graduation rates of the 2008-09 have long been audited and stashed away in a dusty mausoleum.

Another issue that arises is “the pool of possibles.” Is it possible to expect every institution to increase its proportion of Pell-eligible students? Probably not, at least not after some critical point is achieved of overall enrollment is achieved, without dramatically expanding the definition of Pell-eligibilty. Over time a lot of students will wind up being shifted among institutions. Probably not before the next re-authorization of the Higher Ed Act, but I don’t know. It will depend on institutional reaction to the ratings and Congressional action to tie them to student financial aid. Anyhow, at some point one ends up creating a floor value to say, “If an institution is at or above this point, improvement on this measure is not required.”

When we start talking about improvement on multiple measures, we get this argument, “If we enroll more [under-represented students, students with Pell] our graduation rates will go down.” This is a complete and utter bullshit argument that I have heard over and over again. We go back to 2008 on this where I pushed the board to make graduation rates of Pell students compared to students without Pell, but other aid, compared to students without any aid, a performance measure.

There is absolutely no necessity for institutional graduation rates to decrease. “But Tod,” I heard, “we know students that are Pell-eligible are less academically qualified most of the time.” 

“What do you about that?”

“Well, nothing.”

“Therein lies the problem.”

That was a fun conference call. This experience, and more like it, are why I believe a ratings system is best built around comparisons of internal rates at the each institution. There is no legitimate, nor moral reason, why family income should be a primary predictor of student success in college. We know that disadvantaged students have different needs than wealthy students, so let’s meet those needs.

I am glad to see an evolution in thinking on the ratings system. As long as the Department plans to release a draft based on existing data this fall (which is a question-begging time frame to begin with – fall semester or seasonal fall? and I have seen both “in the fall” and “by fall” – implying before), I will continue to oppose and agitate against the current plan. I continue to be willing to help and engage with the Department, but my phone lines and multiple email accounts remain strangely silent.



The $8000 Bachelor Degree

We have finally cracked the code in Richmond for the sub-$10K BA.  Here is how we did it:

No administrative burden!

  1. Award is by a non-accredited institution, so no unnecessary burden in working with sister institutions who provide judgmental visitors that dis what we are doing.
  2. No Title IV federal financial aid. We don’t submit IPEDS or any other federal support.
  3. No buildings or expensive infrastructure to maintain.

No constant pursuit of an unattainable faculty salary goal.

  1. We’ve got NO faculty!
  2. We crowd-source instruction.

Here’s how it works:

  1. Every 16 Weeks we will give you a reading list.
  2. You buy the books and read them.
  3. You create a free WordPress blog.
  4. For each book, post an 8000-word essay about the contents, responding to the writing prompt, “How does this affect me and describe my position in a culturally diverse world where I will never know enough about what is going on to make a difference?”
  5. Readers from around the world will comment on your essays and you will be observed in your responses to their comments.
  6. We will repeat this sequence seven times for a total of eight semesters.
  7. At the end of semester 8 you will write and post a capstone project that responds to the prompt, “This is why I deserve this degree and now I have plan to change the world.”
  8. We will then provide you an .EPS file for you to manipulate and create your diploma.

You never have to send us a check. If you can borrow the books, or go the freaking library, you can do the degree for less.