I am a huge fan of utility. I am also ultimately a pragmatist. Some have called me an unreasoning pragmatist. Or something like that.

However,  when Kentucky Community and Technical College System’s approach to measuring the social good of degree programs that do result in large earnings by creating a “social-utility index.” My response was, “Well, okay.”

There is nothing wrong with the methodology or the conceptual underpinnings. Christina Whitfield, the author, does very good work and so I wasn’t finding fault, it just generated kind of null feelings. Not empty, no content, just null. And I didn’t understand why.

Now I do. When I reviewed the Storify that Dave Mazella had assembled of our conversations last week about assessment, I was sitting in the trailer positioning the new cabinets I had built. As I began fastening them into place,  I realized that my problem is that I don’t think that everything needs to be measured, or assigned a value.

There I said it.  I don’t think that everything needs to be measured, or assigned a value.

I really think we need to make peace with a couple of concepts.

  1. A little inefficiency, like a little nonsense now and then, is a good thing.
  2. It’s okay for something to not have an immediate economic return, or a large one.

I suppose I could explain or justify these, but I don’t think I will. This is what I think, and these are principles that often guide my analysis and recommendations. And they are not new to me.

I don’t think we really need to justify that a well-educated child-care provider is probably a good thing. If you disagree, let me choose your next daycare.


The burning hand, and Frost

In last night’s discussion about assessment, I said this:

We know that the burned hand teaches that the stove is hot. (Actually, it teaches us that a very unpleasant feeling is experienced by touching the stove in a way that we later learn is inappropriate, the same way that unpleasant experience is called “burning.”) The experience of the burnt hand can be observed and measured according to it’s severity and the quickness of the individual’s response. But can learning be measured?

If the “student” can articulate what happened and also articulate that she does not want to experience this unpleasant feeling again, then it seems there is pretty clear evidence of learning. That seems like some kind of proxy measure to me, but a very good one.

The real test of learning, it seems to me, is to observe over time whether or not the experience is repeated.

If not, it seems safe that the lesson is learned.

Of course, all this assumes a physically and mentally normal subject. Someone with nerve damage or Hansen’s disease, may not be able to detect the heat and burning.

All of this occurred to me while driving into work this morning. It also led me, again, to consider measurement. I think about measurement a lot, I really do.

Last night I also said this:

My two years spent running the frame shop at the university museum at SIUE, a number of Habitat for Humanity builds, and years of Scouting, have left me pretty cynical about people’s ability to measure even simple things consistently. Even given that criticism, I was reminded of Roger Zelazny’s novella, “For a Breath I Tarry.” It is the story of a computer named “Frost” in a far-distant future, where mankind has become extinct, that develops a quest, first to understand Man and then ultimately to become Man.

   “Regard this piece of ice, mighty Frost. You can tell me its composition, dimensions, weight, temperature. A Man could not look at it and do that. A Man could make tools which would tell Him these things, but He still would not know measurement as you know it. What He would know of it, though, is a thing that you cannot know.”
   “What is that?”
   “That it is cold.”

I think this is the heart of issue. We can know learning, or we can know measurement. We may not be able to know both.

What the Pell?!!

I am irritated. Very irritated. All of these folks sharing the recent Hechinger report on Pell graduation rates and getting so upset that there is not nationally available data. Fine, we knew this. And those of us paying the least attention since the 2008 HEOA knew that institutions either weren’t publishing Pell & Stafford graduation rates or making them so difficult to find on their websites that it simply doesn’t matter.

Not one of these people are pointing to Virginia and saying, “See how easy it is?” UNLESS I REMIND THEM.

For Pell graduation rates (and Stafford, and 200+ other groups) go here.

See that we publish Pell graduation rates and others on each Institutional Profile by clicking on the Grad. Rates tab.

And we also have a Student Success, Persistence, and Completion Scorecard.

If your data are not as complete, and good, as Virginia’s, you are slacking.

Learning to Count

I don’t know much about assessment anymore. I am also too lazy at the moment to Google “assessing student learning” and am just going to assume that either this hasn’t been modeled before, or that it has and the 17 readers of this blog have not seen the original, or that they have and will tell me so.

For simplicity’s sake, let’s break learning into four categories:

  1. Learning to know.
  2. Learning to do.
  3. Learning to learn.
  4. Learning to be.

So, as long as a student can demonstrate one of these accomplishments (they know what was taught, can do what was taught, can learn to do more, and incorporates all of these) we can measure it and say learning has occurred. Right?


Dave Mazella takes off from When is a waffle not?

There are so many possible ideas to parse out of this Twitter essay.

I think I am going to limit myself, at this time to a defense.

I know that I/we count is so abstract as to have little to no relationship to what actually takes place on any of Virginia college or university campus. 

On the other hand, this is no less true of any well-researched history of an event. Only so much can be conveyed through any narrative, whether in prose or video. With prose you are limited to the words (which also have an agreed up definition or collection of such), arranged in a way that to convey the author’s message, and intent. Video differs only that more visual information is conveyed than can be expressed through the written word, but there is still a point of view and intent.

This is all necessary abstraction.

And it is no different than what we do.

Dave makes two points especially worthy of note.  The first, “since concept of curriculum must assume equivalence of “same” class at diff times, places,” is the idea that we do standardize (Jeff’s translation regime again comes into play here) and standardization is both an abstraction and assumption of equivalence. I suspect most of us in the profession think about it as “approximately equal” as “equal” in any quantitative sense really does allow for variance. (Unless specified as such.)

This takes us to items 8 & 9 – assuming we are counting things and not processes; and that standardization encourages to think about education as discrete and thing-like. Exactly. Counting is about things. Even in the midst of a process, it still comes down to counting things. For example, the speedometer on my car measures the fluid process of movement by reporting an estimate of my speed. This is done by counting the revolutions of the driveshaft within the transmission and converting that to velocity based on the gearing and known size of the tires and wheels. Of course, if the tire/wheel is of an unexpected (non-stock or non-programmed size) the estimate is in error. This value is reported to the driver continuously and looks to be a continuous measure when it is only a collection of values for tiny snapshots in time.

If everything in education moved at 88 feet per second, we could fake a process measurement quite well.

Now for the purists that call me out saying the automotive speed is not a process, that’s fine. Show me any other process management that is not a collection of tiny discrete measures.

To return to  our four categories of learning,  the first two are pretty easy. We can test knowledge gain within reasonable timespans. We can wait years and test knowledge retention. We can do the same with doing. If someone can learn to do something and demonstrate that something, they have clearly learned. If they can still do it two years later, then we should probably consider ourselves successful.

Likewise, if over time, someone demonstrates the ability to keep learning in a specific area, that seems to be a successful outcome.

If someone can do all these things within a specific domain, that also seems a success.

Looks and feels like things that can be measured, but to do so seems that it requires a pretty narrow domain of knowledge to me – not what would be expected in a complete college degree (from the associate’s on up). Perhaps less than even what is in a single course.

Further, it seems that knowing and doing prerequisites to further learning, and by our definition, all three prerequisite for being.

We haven’t even discussed the individual. Or the concept of attribution and who was responsible for what. And so it seems to me that despite decades of work and research, we are not really ready to count learning.

When is a waffle not?

It started with this.

And led to this.

And so the discussion kind of turned to one of what is a waffle? Or rather, how is an Eggo not a waffle?

Some people feel I have a one-track mind, or a little bit of an obsessive-compulsive order. Sometimes I just can’t let things go, like Eggos. I understand where Jeff is coming from in his comment. It is about quality. It implies that something mass-produced and sold frozen is not the same as the thing as made from scratch.

It may not be as good, but it is the same thing. It says so on the package.

They look like waffles. There are shaped life waffles. They have square indentations to hold, syrup, melted butter, sugar, and frosting. Or sausage gravy. They taste like waffles and are almost as good as those served at Waffle House. (That may just be a function of cleanliness.)

Ergo, they are waffles.

Why this matters is again all about counting to one. Language tells us that these are waffles. They have all the obvious characteristics of waffles. It is taste, preference, some other subjective criteria that interfere with some calling an Eggo a waffle.

This is the inherent bias of problems of data systems, and the threat of big data. Automatic application of bias based on unarticulated subjective criteria. I don’t accuse Jeff of anything, I know full well this was light-hearted Twitter conversation about food that I inserted myself into. It is a great example though of how we need to think about data decisions and bias. Especially as matters of belief creep in.

At some point in the future, we will have synthetic/artificial persons, a la Heinlein’s Friday or Dick’s Do Androids Dream of Electric Sheep. We will have to fight to ensure fair treatment and honest counting. Belief can be a poisonous thing in counting.

Even waffles.

A simple devolutionary what if

The last two posts have been about when (not) to count something and how to count to one. I feel the need to go a little further and ask you to consider the possibility that most questions about counting to one have been addressed in the arts and literature. Way back when, we were taught in school four major conflicts in literature that every story could be reduced to:

  1. Man vs. man.
  2. Man vs. society.
  3. Man vs. nature.
  4. Man vs. self.

Let’s revise these and replace “man” with “person.” This is not to be politically correct. (although I was accused of that when explaining the Comonwealth’s version of Zaphod Breeblebox why we used the term “First-time in College” instead of “Freshman”. FTIC is simply more accurate in at least three dimensions.) So then we have:

  1. Person vs. person.
  2. Person vs. society.
  3. Person vs. nature.
  4. Person vs. self.

Of course, this is perhaps a bit specific for those of us that are fans of anthropomorphic literature and and science fiction/fantasy in which even “personhood” may be questionable.

So, perhaps:

  1. Entity vs. entity.
  2. Entity vs. society.
  3. Entity vs. nature.
  4. Entity vs. self.

And now we have a way to talk about conflict and interaction for every data pursuit, big or little. We can think about the interactions of a data entity on another, or others. We can think about how externalities impact an entity, as well as the internalities of the entity itself. (This is especially useful if we are talking about entities within object-oriented programming models and methods, properties, and so on.)

By considering language and meaning in defining One and how literary and artistic history affects our under5940114b57a48c126522c65b6fb0936a900871a0fa482eafabb9e9af07412764standing of the language, we can develop new insights to the data and apply existing solutions to our problems of comprehension. The deep, rich history of cautionary tales throughout ancient and contemporary arts literature provide myriad guideposts.

The arts also help us understand the nature of One.

So few people have read A Tenure Line that I am not surprised this has gone without comment. At the end of the synopsis I left out any mention of the closing number, One. In  “A Chorus Line,” after all the personal stories and the audience has become invested in seeing each character as an individual, all that goes away. Individuals are placed in identical, uniform costumes, singing the same words, dancing the same choreography. Striving for uniform range of motion. Sameness. Eight individuals are now one chorus.

(My old boss at the SIUE University Museum was fond of pointing out, “The sign outside says ‘university’ – ‘uni’ means one!”)

Almost everything you need to know about learning to count to one and the translation regime can be learned through the study of “A Chorus Line.”

After all, the most glorious words in the English language are “musical comedy.”

On Counting to One

In the last post, I mentioned that I describe my job as teaching people how to count to one. This was well-received by some, though I suspect more don’t quite get its significance.

Counting, just counting, is easy. (Kinda.)

number_line We are familiar with the basic number line. Going from 0 to 1 is easy, right? We just move one space. But what does that mean?

“Tod, how many students do we have here?”


“Really? That seems low. I thought I heard the president give a larger number.”

“Well, we have 323 students in study abroad programs in Spain.”

“Okay, that’s better, but Iwas thinking the total was around 15,000.”

“Yes, if you include the students at the off-campus sites, the total enrollment is 15,892.”

“Right! Why didn’t you tell me that before?’

“You asked how many students we have here.”

“Oh. So, are these headcount or FTE?”

And so it goes.

But even more importantly, before we could even get to the first number, we were operating under the assumption that we both have the same definition of “student.” That was never verified in the exchange above, but let’s assume that it is true. That the inquisitor and I knew we both meant that a student was an individual with a specific kind of relationship to the institution. In fact, we knew that “student” is really a general term for a class of groups of individuals with similar, but differing relationships to the institution. These differences may be the level of the degree sought, whether or not they are even seeking a degree, if tuition is paid, if so, under what policies, and so on.

Getting to one requires definition. And as Jeff points out, making choices about who, what, and how someone or something is counted is fraught with peril.

How much fruit do we have?

About seven pounds.

No, no, no, how many pieces?

Gee, that depends on how you cut it.

Dammit, Tod! You know what I mean!

Okay,eight plus 100-150.


Look, there is a big bag of grapes, I’m not going to count them. There are also two bananas, three apples, an orange, and two tomatoes.

The distance between 1 and 2 is easy, as it is between 2 and 3, and so on. But it is only easy because we have done the hard work of defining the distance between 0 and 1.

A Response to The Other Jeff

Jeffrey Alan Johnson of Utah Valley University is the most thoughtful and brilliant practitioner of institutional research I have read in years. This essay touches on issues of data and information that few people really think about. His discussion of the translation regime process of data systems is spot-on.

In this current essay he says:

Data scientists do not often think about our practices as political in nature. But all of the work required to represent some reality in a data system makes data inherently political. Especially as data-driven decisions become norms—even mandates—data scientists are creating the abstract world in which decisions taken place before being implemented in the real world.

The immediate sense in which data translation is political is that the choices made in translating data allocate political power in the real world, not just in the data set. Those who create the translation regime determine which groups do and do not exist, what concepts are available to pursue claims on institutions, which needs can be legitimized and which can be dismissed. The ability to make one’s self, one’s group, and one’s interests legible to the state, organizations, or other individuals is increasingly determined by where one stands in the data.

This is all absolutely true.

But there is another truth. The politics of information are such that those of us in government who oversee large data systems, as I do, often have to be mindful of the potential for misuse and abuse. We also try to focus on what the state (in my case) needs to know.

Over a decade ago, We made two decisions – allow “unknown/unreported/unspecified” as an option for gender reporting to us, and a second decision to go no further than that. We discussed a variety of other options at length and came to the conclusion that the state did not need to know. We felt, given some of the pressures we experienced in 2002 and even 2003, and the political climate, the time was not right to collect something by name and SSN that was not needed. We also kind of felt that it was really none of our business.

Virginia has very strong privacy laws, stronger than FERPA.  But laws can be changed. They do so all the time. I’ve written enough law to be oh-so-very aware of this. I can’t reveal what I don’t know, what I don’t collect, which means I can’t cause certain groups of people to be targeted.

For now, it is kind of a trade-off between the power of being noticed and the comfort of not being able to be targeted.

The power to define is tremendous. For about 20 years in the institutional research profession I have defined my role as “teaching people how to count to one.” This is the essence of the translation regime because the distance between zero and one philosophically orders of magnitude greater than the distance between one and two. Defining what you count, who you count, is everything.

Knowing when to change the definition of one is understanding the politics of information and expressing a readiness to deal with consequences. Every decision I make about defining how to count to one ripples down to nearly 80 institutions and often causes them to change policies. Each decision also begs the question, “Why? Why does the state need to know this and what are you going to do with the information?” Bernard Fryshman is a master of asking these questions of USED.

In my normal mode of obstructing bureaucratic wastes of my time, I challenged the state IT agency over and over again about some of the information they were requesting for the IT strategic plan.

“Why do you need this?”
“We need to know scope of things so we can manage it.”
“Are you going to use it to recommend more IT funding for my agency?”
“No, we don’t do that.”
“What does manage mean – are you going to cut my budget?”
“Why no, we just need to know so we can manage it, to control it.”
“If you aren’t going to add to my budget, or maintain the funding I need to provide the services you mandate, you can only cut my budget, right?”

The politics of information matter. A sensitivity to upstream demand and control is often needed. The decisions about who and what to count are a balancing act.

Why I don’t Quite Fit

I remember standing on top of four feet (or more) of densely packed snow one night in northern Alaska. We were in a large circle under the stars and Northern Lights listening to our company commander explaining how to conduct ourselves during the next day’s inspection by the US Army Chief of Staff.

“Before we leave on the march out tomorrow, General Wickham will walk down the line. He may speak to you, ask you questions. These will not be hard questions. He is not going to ask you the square root of three….”

“1.732 , sir,” I said.

Needless to say, he was not amused. He rarely was ever amused by me. This seemed to be a pattern among Army officers. Well, all officers that I encountered.

In the generally misspent days of my youth, I often thought certain pieces of information would be of use later in life. The square root of the first five ordinal numbers, for example. The densities of certain materials such as gold and lead (1204 lbs and 708lbs per cubic foot, respectively).

This becomes an issue when I read a book, or watch a movie (and movies are generally worse), and gold gets involved. It’s bad enough that at the end of The Hobbit, Bilbo settles on a small chest of gold and another of silver, being as a much as pony would carry.  Small is relative term and ponies can carry a respectable load, of say a 150lbs, and so I can see that in my mind as reasonable. But in the third movie that is based on a story that is similar to that in The Hobbit, Bilbo is carrying what is allegedly a  chest of gold under his arm. Without much effort – and thus without much gold. And don’t even get me started on the gold-dipped dragon, there just ain’t that much magic in the world.

And yes, I start trying to estimate the surface area of Smaug and how much gold might be involved. I don’t get too far fortunately because it is just so silly.

Kelly’s Heroes is one of my favorite World War II movies. I still get amused at the idea of stealing 14,000 gold bars. Since the standard gold bar weighs 12.4 kg or 27.3 lbs, the total haul weighs in at about 190 tons. Any idea how many Deuce-and-a-halfs (2.5 tons) trucks would have been needed to haul all that gold away?

Just one, if it makes 77 trips.

And by the way, if stored perfectly efficiently as cubic foot blocks of gold with no space between blocks, we are looking at a stack of 317 blocks, say a stack of 10 on each side, three complete layers high with a fourth started. When stored in wooden crates at a weight that a single man could lift – six bars to a box – we need over 2300 boxes that take up a lot more room.

This is why I prefer to watch movies at home – I bother fewer people when I share my loss of suspension of disbelief.

What’s worse is visiting an art museum with me. Unless you like laughing inappropriately in public.


A Tenure Line

A musical. In seven one-year acts.

The show opens in the middle of a university hiring committee review of applications. The formidable designee of the Provost, a tin-pot departmental dictator (chair) with delusions of grandeur of Napoleonic proportions, Zach and his assistant chair Larry put the applicants through their paces. Every new PhD is desperate for work (“I Hope I Get It“) {God, I hope I get it, God, I need this job]. After the three rounds of cuts, 17 applicants remain. Zach tells them he is looking for a strong research chorus of eight researchers that occasionally teach. He wants to learn more about them, and asks each prospect to introduce themselves. With reluctance, these new graduates reveal their pasts. The stories generally progress chronologically from early life experiences through adulthood to the end of a career.

The first candidate, Mike, explains that he is the youngest of 12 children. He recalls his first experience with research, watching his sister’s book report when he was a pre-schooler (“I Can Do That“). Mike took her place one day when she refused to go to class—and he stayed. Bobby tries to hide the unhappiness of his childhood by making jokes. As he speaks, the other candidate have misgivings about this strange audition process and debate what they should reveal to Zach (“And …”), but since they all need the job, the session continues.

Zach is angered when he feels that the streetwise Sheila is not taking the audition seriously. Opening up, she reveals that her mother married at a young age and her father neither loved nor cared for them. When she was six, she realized that academics provided relief from her unhappy family life (“At the Library“), as did Bebe and Maggie. The scatter-brained Kristine is tone-deaf to nuance, and her lament that she could never “Get it!” is interrupted by her husband Al finishing her phrases in syncopated rhythm.

Mark, the youngest of the candidates, a sociologist, relates his first experiences with pictures of the female anatomy and his first wet dream, while the other dancers share memories of adolescence (“Hello Twelve, Hello Thirteen, Hello Love“). The 4’10” Connie laments the problems of being short, and Diana recollects her horrible high school chemistry class (“Nothing“). Don remembers his first job at a nightclub and Judy reflects on her problematic childhood while some of the applicant stalk about their opinion of their parents (“Mother“). Then, Greg speaks about his discovery of his homosexuality and Richie recounts how he nearly became a kindergarten teacher (“Gimme the Ball“). Finally, the newly buxom Val explains that talent alone doesn’t count for everything with thesis directors, and silicone and plastic surgery can really help (“Research: Ten; Looks: Three [Tits and Ass]“).

The candidates go downstairs to develop three-minute presentations for the next section of the audition, but Cassie stays onstage to talk to Zach. She is a veteran adjunct instructor who has had some notable successes as a students. They have a history together: Zach had hired her as an adjunct previously, and they had lived together for several years. Zach tells Cassie that she is too good for the the positions he has and shouldn’t be with this group. But she hasn’t been able to find a tenure-track job and is willing to “come home” to the department where she can at least express her passion for research (“The Music and the Coffee“). Zach sends her downstairs to prepare a presentation.

Zach calls Paul on stage, and he emotionally relives his childhood and high school experience, his early career in a drag act, coming to terms with his manhood and his homosexuality, and his parents’ ultimate reaction to finding out about his lifestyle. Paul breaks down and is comforted by Zach who usually has no heart for anything except the latest grant recipient.

During a tap sequence, Paul falls off-stage and injures his knee that recently underwent surgery. After Paul is carried off to the hospital, all the candidates stand in disbelief, realizing that their careers can also end in an instant and spent as a roaming adjunct. Zach asks the remaining dancers what they will do when they can no longer do research. Led by Diana, they reply that whatever happens, they will be free of regret (“What I Did for Love“). The final eight assistant professors are selected: Mike, Cassie, Bobby, Judy, Richie, Val, Mark, and Diana.

At the end of the show, three of the positions are cut from the budget, two offers are withdrawn because of outrage generated on social media for comments made as graduate students. The three remaining new hires had nice little careers and always encouraged the adjuncts they met to work harder and the dream would happen for them, too.

If anyone would like to produce this for stage, please know that most of the music I hear in my head for this driven by open-back banjo and an electric bass.