Thou shalt not worship data, because much of it is no good

Here is what happened: we (higher ed/researchers) oversold the public on the idea of data-informed (or evidence-based) improvement decisions, and are now paying for it. While I see lots and lots of data being collected, the decisions made with the collected data as critical piece of evidence are rare. When we have problems, for example, in teacher preparation, we know about them before data is available to corroborate. When data contradicts the anecdotal evidence, we tend to distrust it. In those rare occasions where data is reliable, timely, and complete, it is more often than not correlational, and thus tells us little about causality. For example, you may find that class size correlates with failure rate. So what? It is very likely that there is a confounding variable that explains both, and you could not or did not think of measuring. If, for example, you find that there is no correlation between student evaluations and grades – yes, this busts the myth that grade inflation is fueled by student evaluations. While correlation does not imply causation, the lack of correlation usually mean the absence of causation (or a measurement error). Similarly, if you find no correlation between the scores on your math placement test and student performance on subsequent courses, your placement test is no good. However, if you do find a correlation, it does not mean the test is good. So, before collecting anything, the simple check is – what are you going to do with it, exactly? Don’t collect just because it is there, and hope it will bring some useful knowledge – it won’t. And if you’re an accrediting agency – don’t ask for data that will not result in any decision.

My basic claim is this: an organization has to evaluate the usefulness of any data like this: i=R/U, where R is resources expended to gather, analyze, and keep the data, and U is the potential usefulness of it for real decisions. One very important stipulation is this: R should include time expenditures. Sadly, it is often the case that all the time is spent on collecting and crunching, so no time or strength is left for using it. My best guess is that in the overwhelming majority of cases i is greater than 1. In other words, we’re wasting a lot of valuable time.

Of course, much of data is collected because various accrediting bodies tell us to do so. However, they also have no idea how exactly the data is going to be used, and if it is any good. For example, NCATE asked us to measure impact on student learning that teacher candidates make. So we all figured out some sort of action research thing for our candidates, with pre and post-test, figuring out the effect size, etc. we complied with the requirement to measure impact on student learning, but that is simply bad science. We could never teach teacher candidates even how to build valid measurements. An instrument simply cannot be validated after a one-time use. Or else, NCATE made us measure candidate performance as we observed in the field. But those observation rubrics often produce flat, uninteresting data, because they are not reliable, and don’t measure what they intend to measure. Even more rigorously designed instruments like Danielson framework, show only modest correlation with teacher quality as measure by student achievement. But in the field, with dozens of supervisors who change constantly, who has time or money for interrater reliability training?

Everyone looks at colored charts, happy, pretending those numbers mean something. And we pretend that oh, yes, we looked at this, and made this specific decision. I don’t want to accuse everyone, but in most cases it is not true. Notice, I am not saying it is never true – the good examples are too rare to justify the enormous time and effort.

Many of us got a case of what I call “the compliance disease.” It feels good to be proficient t something, and we find clever ways of collecting data, aligning it to standards, and presenting. The process itself takes a lot of skill and creativity, so we forget that it is less than useful in the end. This is a common phenomenon – people get better and better at figuring out how to comply, and stop questioning what they have agreed to comply with.

There is a class of data that has direct significance: how many students do we have, what are average class sizes, which groups succeed more and which tend to do drop out, where are the bottlenecks, etc. It is just the measures of quality, derived from performance standards that remain elusive. And it is after at least 30 years of trying. Measuring quality of higher education is still an aspiration rather than reality. We can measure quality of K-12, but very narrowly. It is like looking at vast landscape through a keyhole of standardized testing. But in higher ed, we cannot see much at all.

The data technology is still primitive. What we have now is really quite basic hand tools that require a lot of human labor and subjective judgments. All I am saying is that brains are more needed to improve things we already know need fixing, than on collecting mountains of data we have no time to do anything with. We should only do things that move us forward.

I am not suggesting we give up on the idea of data-informed decision-making. The alternative is pure guessing, or gut instincts – all notoriously unreliable means of decision-making. The alternative is going back to the dark ages. Many people, including me, are hoping that the next generation of data tech, based on naturally occurring digital traces, in combination with the neural networks and predictive analytics will change everything. In the meanwhile, modesty is virtue.

Do we have a choice about our visions?

The answer is – probably less than one may think. It would be completely foolish to dream big dreams without considering the two existential threats to educator preparation at state universities. One is the continuous downward budgetary pressures. The second is the increasing competition from for-profit, online, alternative programs. In California, we should also add various district-based and county-based preparation programs. While we are in a good shape now, the long-term trends look very worrying. You do not need the SWOT analysis exercise to see that. The defunding of public higher education is a national trend, driven not as much by politics, as by the economics of mass higher education. Keep in mind, we remain competitive only because of the public subsidies, and some limited brand loyalty. That is, we compete mainly on price. We tend to lose on convenience, the user-friendliness, and on marketing, and very often - on responsiveness to employers’ needs.

These two threats imply a certain strategy, and I don’t see how one has much of a choice about it.

1. We must learn how to make money, which means developing additional revenue streams. 
2. We must become more flexible, less bureaucratic, and friendlier to students. 
3. We have to become sophisticated marketers. 
4. Finally, we must participate in regulatory politics. If we allow significant deregulation or meaningless accreditation to happen, it may open the flood gates for low-quality competition. Because of the famous Akerlof’s “Lemon Law,” this creates the race to the bottom phenomenon, typical for non-experiential good markets. 

The last thing is too big for each institution; it is a cause for larger professional groups. The first three, however, are the responsibility of each college of education. No one is going to do it for us. Most visions I have seen deal with some sort of growth in reputation, like we will become a premiere institution, or we will be known nationwide, etc. I was thinking along the same lines. However, perhaps we should try something more pragmatic. For example, we can say that we will become financially secure, and have some money to invest in development. I think we can become known for being not just personally, but institutionally friendly to students. We should not proclaim the abstract goals of endorsing diversity and equity – everyone does that. Instead, we can say that our programs and logistics will be tailored for the needs and expectations of minority and first generation, as well as working adults. Finally, we can have a vision of developing a robust marketing machine comparable to some of our best-known competitors.

Perhaps I am missing something, but just want to offer this kind of more pragmatic, less pompous way of envisioning our common future here. I believe we should meet out key challenges head first, with all we've got, and that becomes the shared vision.

Complexity and Justice

At a typical American university, we create insanely complex rules, and then waste enormous resources explaining them to students. Just consider, for example Sac State’s Gen Ed requirements. First, it is 48 credits plus the language requirement. The national norm is closer to 40, or about 1/3 of a BA degree. Plus there are layers of overlapping rules, which should be applied simultaneously. Those are rules of residency (so many should be taken at CSU), the rules about upper and lower division, and the rules of distribution (areas of A, B, C, D, and E). On top of it, there are graduation requirements, such as American History, American Institutions, Intensive writing, English composition, Race and ethnicity, and foreign language. In other words, every course choice should be checked against the five or six sets of rules. Those decisions have to be made by 18-22 year olds, a lot of whom just transferred from community colleges, and have a whole set of issues with equivalent courses, some of which are articulated, while others are not. Only few faculty on campus actually comprehend the rules and even fewer can explain them to students. So we have to have several professional advisers, and train older students to help. It takes years to actually master the working knowledge of curriculum. And with this level of logical complexity errors are absolutely inevitable. The complexity of rules requires the maintenance of three separate technological platforms: the course planner, the registration system, and then the data analysis system. The latter is needed, because we have no idea how many sections of which course will be needed in the next year. I am not just picking at the gen Ed; the same can be said about majors, teacher credentials, and everything else.

Now, if you have a mom and dad with a college degree, you can call them up, and ask to interpret the catalog for you. If you are a first generation in college, and your parents speak another language at home, you may not know what is the upper division and the lower division. So, the complexity affects different people differently. We always proclaim the values of diversity and inclusion, and yet our own indifference to student experiences is partly responsible for drop-outs and forces students to stay longer than they would like, to incur extra debt.

All of this is done with the best intentions, to make sure we educate the kids well. OK, let me take this back – the system evolved because of trivial turf wars, where every department is concerned about its status, its enrollments, and its workloads. The wars are fought with the rhetoric of best intentions, usually, so the combatants re confused where is the real motive, and where is the high-minded rhetoric. A significant part of the problem comes from various mindless bureaucratic decisions done outside campus, too – in various accrediting, and state government bodies. It really does not matter who we are to blame. The bottom line is that chasing complexity with various technologies is an arms race we cannot win. It is not a solvable problem. Yes, we should provide students with good advising and technological tools. But the problem is not solvable without some movement in at least partial simplification of curriculum.

The radical solution is well known: it is cohorting. It works in many cases, for working adults, for professional degrees, for non-traditional students. Some students are willing to give up choice in exchange for stability and the guaranteed timeline. In fact, this is how most of Russian and Chinese students go through college. I would not support the radical solution across the board; there is enormous value in the ability to choose one’s learning path, and in the flexibility of the non-cohorted environment. The rigidity of a cohort system also has exclusionary qualities; it does not accommodate for all life circumstances. Instead, we must simplify the admission and graduation requirements, and other processes by the order of magnitude to actually walk the walk of justice. There is no evidence whatsoever that more complex rules add anything to the quality of education. For example, Brown has abandoned the general education requirement altogether, which did not have a demonstrable negative effect on its graduates. There is no evidence that, say a 72 credit major is more fruitful than a more typical 40-credit major. Only a few of prerequisites actually have pedagogical sense; most are there to force students into a more manageable path.

The main funding of the entire fiel of behavioral economics is this: if you want to encourage people to do something, make it easy. So if we want more diverse population in our student body, if we want to more teachers of color, we have to make it easy, less intimidating; not less rigorous or less demanding; just easier in the process. Let’s move the rigor out of our processes into our classrooms.

The Long Email Combat Ritual

Among the tribes of the Academia, a small minority engages in a strange combat ritual. The weapon of choice is the long email. It is usually reinforced with a tail of previous long emails, and with multiple CC recipients. The warriors may adorn their weapons with brightly highlighted lines, intended to point out how obviously wrong or incompetent the other person it. Some have their desk drawers full of printed out emails as trophies of previous glorious battles, and in anticipation of the Judgment Day.

A battle often begins innocent enough; the exchanges look like simple business-like conversations. However, with time, they become longer, more detailed, and include more and more elements pointing out at the other party’s faults and omissions. At that time, they usually acquire more recipients, including me. The anthropologist in me is fascinated by the elaborate ornaments. The manager in me wonders how much time they spend writing these things.

OK, now seriously: Email is a terrible medium for resolving any problem, much less a conflict. It is cold, emotionless, and always sounds harsher than intended. Moreover, once you are past two exchanges, it is not even productive – it is time to meet or at least talk on the phone. I have learned this rule of thumb from my friend and mentor Eugene Sheehan. We write emails in order to save time for planning a meeting, right? But after four e-mails, you have reached the point of diminishing returns. The sad thing is that the medium itself lends to being weaponized. It does not enhance social cohesion, but may actually corrode it.

So, the grown up thing to do is to get up and meet someone you disagree with, or whose points you do not understand, in a face-to-face situation, or at least give one a phone call. When we do that, we activate psychological deterrence mechanisms that are hundreds of thousands years old. It is more difficult to say something nasty to someone’s face. The psychic cost is much higher, so we normally avoid doing it. It is because we evolved as species attuned to interpersonal communications. Email is too new for us to adapt.

Another great trick I learned from someone is this: if you receive an irritated email, a challenge, really a provocation to fight, answer it with the deadliest of all weapons – silence. A non-reply is a great answer in some circumstances, and it is amazing how many people feel compelled answer every e-mail. Again, the CC recipients create this urge – if you do not answer a publically wielded falsehood, you may look guilty by the virtue of non-responding. But give your audience more credit – they will more likely interpret your silence correctly. A nasty email is like trolling on social media. The wisdom of teenagers – do not feed the troll. Any response is a gift to the troll. The more you sound like you’re hurt, the more successful is the trolling attack. We teach young kids to walk away from the impending fight, but so often don’t know how to do this in our own world.