Search This Blog

May 31, 2013

Campbell’s Law

Donald T Campbell, a well-respected social scientist, came up with this pessimistic law:

  • “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”
His own examples are numerous and compelling. Measuring effectiveness of police forces by the proportion of crimes solved leads to “Failure to record all citizens’ complaints, or to postpone recording them unless solved.” More seriously, it encourages criminals to confess to crimes they did not commit in exchange for plea bargains, because police wants to count those crimes as solved and are desperate for plea bargains. It is important to know that not only the data will become inaccurate, but the practice it is supposed to measure actually gets worse. He also examined an early version of high stakes testing: “when test scores become the goal of the teaching process, they both lose their value as indicators of educational status and distort the educational process in undesirable ways.” This was in 1976! In a much darker example, he explains how Nixon administration switched from estimates of enemy casualties (which were utterly unrealistic) to enemy body counts. This bureaucratic trick lead to the My Lai massacre of 1968, and who knows how many unnecessary casualties.

Here is one example from our field. In the early zeroes, I was working in Ohio, and Praxis II passing rates were just beginning to be used for accreditation of teacher preparation programs. We learned that in the neighboring Michigan, all programs made the Praxis II passing a program completion requirement, hence achieving a 100% passage rate. Voila! Problem solved. Almost every institution in the country has done the same thing since.

Another more recent example: the new RI graduation requirement is to pass NECAP test. However, if one failed the first time, but has shown growth in the second attempt, one still graduates. I told about the rule to my non-educator neighbor, and it took him about 5 seconds to come up with a corrupt scheme: fail the exam on purpose for the first time, and then try hard on the second. I have no doubt high school seniors will come up with that idea sooner or later. No matter what your score is, you will improve. Of course, this will also lower the average passing rates for the State, which is the effect opposite to what RIDE intended.

What do we do then, abandon all measurements? This does not seem to be a viable option. Campbell himself entertained such solution as the use of external and independent evaluators, and finding measures immune to corruption. He also believed that the use of multiple measures, each of them imperfect, will reduce the corruption pressures. Basically, Campbell called for always paying attention to the corruption pressures, and carefully constructing methods that address them. It has been almost forty year, and many of same themes were discussed well before him. I am not sure how much we’ve actually learned. Numbers have a certain seductive simplicity to them. This is why sophisticated measurement experts are suspicious of numbers.

No comments:

Post a Comment