Donald T Campbell, a well-respected social scientist, came up with this pessimistic law:
Here is one example from our field. In the early zeroes, I was working in Ohio, and Praxis II passing rates were just beginning to be used for accreditation of teacher preparation programs. We learned that in the neighboring Michigan, all programs made the Praxis II passing a program completion requirement, hence achieving a 100% passage rate. Voila! Problem solved. Almost every institution in the country has done the same thing since.
Another more recent example: the new RI graduation requirement is to pass NECAP test. However, if one failed the first time, but has shown growth in the second attempt, one still graduates. I told about the rule to my non-educator neighbor, and it took him about 5 seconds to come up with a corrupt scheme: fail the exam on purpose for the first time, and then try hard on the second. I have no doubt high school seniors will come up with that idea sooner or later. No matter what your score is, you will improve. Of course, this will also lower the average passing rates for the State, which is the effect opposite to what RIDE intended.
What do we do then, abandon all measurements? This does not seem to be a viable option. Campbell himself entertained such solution as the use of external and independent evaluators, and finding measures immune to corruption. He also believed that the use of multiple measures, each of them imperfect, will reduce the corruption pressures. Basically, Campbell called for always paying attention to the corruption pressures, and carefully constructing methods that address them. It has been almost forty year, and many of same themes were discussed well before him. I am not sure how much we’ve actually learned. Numbers have a certain seductive simplicity to them. This is why sophisticated measurement experts are suspicious of numbers.
- “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”
Here is one example from our field. In the early zeroes, I was working in Ohio, and Praxis II passing rates were just beginning to be used for accreditation of teacher preparation programs. We learned that in the neighboring Michigan, all programs made the Praxis II passing a program completion requirement, hence achieving a 100% passage rate. Voila! Problem solved. Almost every institution in the country has done the same thing since.
Another more recent example: the new RI graduation requirement is to pass NECAP test. However, if one failed the first time, but has shown growth in the second attempt, one still graduates. I told about the rule to my non-educator neighbor, and it took him about 5 seconds to come up with a corrupt scheme: fail the exam on purpose for the first time, and then try hard on the second. I have no doubt high school seniors will come up with that idea sooner or later. No matter what your score is, you will improve. Of course, this will also lower the average passing rates for the State, which is the effect opposite to what RIDE intended.
What do we do then, abandon all measurements? This does not seem to be a viable option. Campbell himself entertained such solution as the use of external and independent evaluators, and finding measures immune to corruption. He also believed that the use of multiple measures, each of them imperfect, will reduce the corruption pressures. Basically, Campbell called for always paying attention to the corruption pressures, and carefully constructing methods that address them. It has been almost forty year, and many of same themes were discussed well before him. I am not sure how much we’ve actually learned. Numbers have a certain seductive simplicity to them. This is why sophisticated measurement experts are suspicious of numbers.
No comments:
Post a Comment