Campbell’s Law

Donald T Campbell, a well-respected social scientist, came up with this pessimistic law:

• “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”

His own examples are numerous and compelling. Measuring effectiveness of police forces by the proportion of crimes solved leads to “Failure to record all citizens’ complaints, or to postpone recording them unless solved.” More seriously, it encourages criminals to confess to crimes they did not commit in exchange for plea bargains, because police wants to count those crimes as solved and are desperate for plea bargains. It is important to know that not only the data will become inaccurate, but the practice it is supposed to measure actually gets worse. He also examined an early version of high stakes testing: “when test scores become the goal of the teaching process, they both lose their value as indicators of educational status and distort the educational process in undesirable ways.” This was in 1976! In a much darker example, he explains how Nixon administration switched from estimates of enemy casualties (which were utterly unrealistic) to enemy body counts. This bureaucratic trick lead to the My Lai massacre of 1968, and who knows how many unnecessary casualties.

Here is one example from our field. In the early zeroes, I was working in Ohio, and Praxis II passing rates were just beginning to be used for accreditation of teacher preparation programs. We learned that in the neighboring Michigan, all programs made the Praxis II passing a program completion requirement, hence achieving a 100% passage rate. Voila! Problem solved. Almost every institution in the country has done the same thing since.

Another more recent example: the new RI graduation requirement is to pass NECAP test. However, if one failed the first time, but has shown growth in the second attempt, one still graduates. I told about the rule to my non-educator neighbor, and it took him about 5 seconds to come up with a corrupt scheme: fail the exam on purpose for the first time, and then try hard on the second. I have no doubt high school seniors will come up with that idea sooner or later. No matter what your score is, you will improve. Of course, this will also lower the average passing rates for the State, which is the effect opposite to what RIDE intended.

What do we do then, abandon all measurements? This does not seem to be a viable option. Campbell himself entertained such solution as the use of external and independent evaluators, and finding measures immune to corruption. He also believed that the use of multiple measures, each of them imperfect, will reduce the corruption pressures. Basically, Campbell called for always paying attention to the corruption pressures, and carefully constructing methods that address them. It has been almost forty year, and many of same themes were discussed well before him. I am not sure how much we’ve actually learned. Numbers have a certain seductive simplicity to them. This is why sophisticated measurement experts are suspicious of numbers.

The Bad Apple theory

Over many years, I have seen this particular theory in action many times, in different kinds of communities. It usually afflicts strong ones, with a high degree of cohesion and of the collective self-regard. People locate all their problems in one or two particular members. If only this person, - the theory goes – would have left, we would be so much stronger, and things would become so much better. In some cases, it is the official leader of the community that gets the bad apple designation, but in most cases, it is just one of them. The one thing I am sure of – it does not work. Whenever one bad apple leaves, someone else immediately takes his or her role, and a new schism ensues. In the worst case scenario, a community gets worse with every round of the bad apple removals, and eventually tears itself completely apart. It does happen, and I have seen it happen. (This does not mean I have not – in the past – embraced it; I did, and am not proud of it).

Where does it come from? Again, it afflicts groups who have a strong collective sense of themselves. Their collective expectations are perhaps a little inflated, a little unrealistic. Real people have a hard time meeting those. Next, we all have a tendency to personalize evil. In other words, when things go wrong, it is incredibly liberating to us to see that it is someone’s fault. This is why we have such a hard time dealing with natural disasters, and why we cope brilliantly with acts of terrorism. An enemy is a gift of a sort, a place where we could deposit our rage. On a smaller scale, this is how the bad apple phenomenon works. Once we start picking apart one of us, - and sometimes with good reasons! – the negative bias becomes self-perpetuating. Keep looking for bad things in one particular person, and reinforce your findings with your friends and colleagues – and viola, you will find a lot more bad things. Moreover, this one person’s real or imagined failings become the cause of the general malaise. We as a community are not what we can be because of the one bad apple. This thought is as comforting as it is misleading.

Next, a lot of time and energy is diverted from positive things into routing out the bad apple. As a results, fewer good things are done, the community falls behind of its own expectations even further, and there are even more reasons to blame the bad apple.

Of course, the bad apple is a position, a role rather than one person’s intrinsic qualities. Not one of us can sustain the prolonged negative attention of an entire group without eventually starting to behave badly in response. So the bad apple is created, albeit unintentionally. And bad apples tend to fight back, making the whole cycle even more vicious and eventually self-defeating.

Great communities find a different way of dealing with the bad apples. They still have their outliers; by definition, if a community has a set of expectations, someone has to be less conforming than others. It is just a statistical fact. But great communities embrace their bad apples, even treasure them in a weird way which I cannot quite pin-point. Rather than removing the bad apple, everyone is concerned about finding the right place for it.

A doze of humility is an essential component of a great community; it includes the general realization that humans are very flawed creatures to begin with, and that none of us the “good” apples is that great either. There is also a healthy tolerance for imperfection: we as a community is not perfect, and things do not always go as they should. And finally, great communities are focused on the outside, and are not obsessed by their own internal relational dynamics. They simply don’t have time to worry who is a better apple; they have a lot to do.

Big data and big-enough data

Big data is information that is too big to process by any kind of manual data analysis tools. For example Walmart has databases equivalent of 167 times the information in all the books in the Library of Congress. But besides purely technological, there are also significant institutional limitations to process data. For example, in our old assessment system, one data manager was able to process most of the reports within six months or so. It was always too late to make any decisions for the following academic year. It just took too much work to bring our puny (in comparison to Walmart’s) amount of data into a usable form.

Now we are working on a new system, where many different people will be entering data more or less in real time, into one integrated and publicly accessible warehouse. But even with those efficiencies, the question remains – is it the right size and the right quality of data we can actually digest? Is it too big, or too small? Is the quality of data good enough to make it actionable? And finally, are the time and resources used to collect and aggregate it justifiable? Can it actually improve the quality of our decisions? Those are all questions that can only be answered through experiences.

Various accrediting bodies, including NCATE and most states’ departments of education tried to impose the culture of data onto its member institutions. That first attempt more or less failed, because no one knew what the right size of data is appropriate for a particular kind of institution. As a result, most of the teacher preparation institutions contracted the compliance disease. I venture to guess that the quality of data collected actually got worse because of these miscalculated policies. They are now trying to correct their own error by encouraging institutions to think deeper about what data is needed, and how it can be improved and used. Data that is generated for compliance reasons only is always too big. Therefore the ownership of the process turned out actually a lot more important than the size of the data.

In a sense, we all are starting from square one again; and this would be true not just for the teacher preparation programs. The questions to ask in the square one are not what we can collect and what does RIDE or CAEP or a SPA want. Those are entirely wrong questions. The questions should be like this: What do we not know, but would like to know? How can we be surprised by data? Is it interesting to look at? What can we feasibly collect and store? What and how can we process quickly, in time for some decisions to be made? What tools and resources do we have to make all of this possible?

The unchecked truth

At the heart of many conflicts is a belief about the nature of truth. Two or more people will construct their version of “what happened.” In some cases, they have been the direct participants of the past events, and therefore they claim the right to know the truth of events as they happened. In other cases, they heard it from people they trust. However, almost inevitably, their versions of events in the past are very different. In the absence of conflict, such disparate versions of reality happily exist in different heads, and their owners either do not know or do not care that someone else remembers the same events very differently. However, when they are asked to recall it in a confrontational situation, those differences come up against each other, clash, and tend to deepen the conflict rather than resolve it. When someone else is retelling the story with which you grew comfortable, it creates a whole set of questions about the other person’s motivation, truthfulness, the attack on your integrity, etc.

Why is this happening? First, because we’re the story-telling creatures. We have the deep need to construct a coherent narrative. Therefore we fill in the gaps in our factual knowledge with guesses, especially about other people’s motives (which we cannot read). We have to; otherwise the story does not hang together, and bothers us. It remains in what some psychologists call the cognitive dissonance. Only a few people have developed a habit of constantly challenging their own version of events. Self-doubt is a relatively rare skill; most of us do not have it. But the need for story-telling makes us create a total, holistic story. And how does a story become complete? - By providing an explanation to the actors’ motives. Aha, I know WHY he did this; I can now rest easy, and archive the event. To commit an even to our long-term memory, we need a label, a value judgment: “a bad person story,” “an incompetent person story,” “a nice person story.” This is why we actually have the social memories – to keep track of our friends, enemies, and who we owe a favor, and who owes favor to us. The evolutionary function of the brain is largely to provide for this kind of accounting. It worked well in small bands of early humans with exhaustive face-to-face interactions, and repetitive events. It does not work that well in complex organizations. Our brains are seriously deficient for the world we have created.

In complex organizations, where many transactions are by e-mail, through a third person, or spread over time, we simply do not have access to the whole set of facts. This is especially true in interactions that include many people. But no matter how little we know, our instinct is to create a full story.

In addition, human memory changes the story every time we recall it. The more often we recall a certain event, the more details we add to it. Most people know about the phenomenon of false memories, or confabulations. Unfortunately, we all suffer from the very mild version of it. Take any of your youth or childhood memories, preferably one that you have recalled many times, and try to fact-check with other family members. You will see that certain details of it you have made up. But because in our society there is such a premium on honesty, and on good memory, we tend to balk at a suggestion that we misrepresent a factual event. The mechanisms of self-justification kick in powerfully when we are doubted.

The solutions to this dilemma have been around for a long time. One is to agree that no one has the whole truth. No matter how righteous you feel, and how noble your motives, you do not possess the whole truth, nor are other people obligated to buy your version of events. The second solution comes from the ancient Greek and Roman laws. It is still the cornerstone of our judicial system. There are rules for public contest between the conflicting versions of the truth. Basically, the public, or its representatives have the right to construct their own version of the truth by challenging and considering the individual versions.

And finally, my favorite philosopher Mikhail Bakhtin made a more general claim. He said that error is not the only source of multiple voices. If two people disagree, it does not mean that one of them wrong. It may be the case for a narrow set of empirically verifiable disputes, but does not apply to the vast majority of human interactions. The ability to internalize dialog, to be able to challenge and check your own truth is becoming an essential skill for living in the contemporary society. The truth is irreducible to one story, it is always a set of mutually addressed, but distinct stories.