Institutional data has come a long way. I still remember the hard copies of IR annual books; they were the only way to get any kind of stats about your programs. They were also mostly wrong, by the way. Now we have many sophisticated real-time data portals, showing you everything you need to know about your organization. I used to think that certain number of errors in the reporting was just a temporary thing, something we will eventually overcome. However, I do not believe this anymore. The errors are built in, and they are not caused by information technology. The errors come from human-data interactions.
There are at least two different causes of the intrinsic errors. One has to do with Campbell’s Law, an under-appreciated phenomenon; I wrote about it a few years ago. If consequences of a quantitative indicator are high, the measurement tends to corrupt the very practice it intends to measure. For example, if in my College FTEF/FTES ratio looks high, I will be thinking of ways to record both indicators to make us look better. Trust me there are ways.
However, there is something Campbell did not know, because in 1979, there were no user-input databases at every organization. Let us call this the Sidorkin’s Law: Categorical scarcity leads to workarounds that will corrupt data input. When designing a database, one cannot use a very large number of fields/categories. That would make the database unwieldy and unmanageable. It would be impossible to produce a usable report. The whole point of database is simplification, standardization of information. If you want to compare, say, two colleges, you have to make a call: let’s call student teaching something similar to nurses’ clinical practice, for example “supervised field experiences.” One can see how they are similar, but they are also vastly different. The need to measure and compare brings about emphasizing the similarities and ignoring the differences. However, life is always just a bit more complicated than any set of categories, and people learn to operate within the limits presented by the database designers. Considerations of convenience force us enter data in the fields and categories that were not initially intended.
For example, one of our program areas has struck a deal several years ago that when they teach larger classes, they can get an extra workload credit. However, the workload database did not have a specific category to reflect such an arrangement. Moreover, there was initially a rule that synchronized student credit hours with faculty workload, a very sensible one, by the way. Therefore, someone came up with a work-around: We record these units as Discipline-Based Research. No one remembers what it was intended for, but it was a convenient way to solve the data input problem. As a result, our overall instructional workload units looks smaller than they actually are. This is just one small example; I can name a dozen or so workaround fixes like this. Cumulatively, they can significantly affect the reports that come out in the end. This is why I always prefer to see the raw data, understand the sources and the formulas used in a report, and never trust prepackaged reports.
On top of these two data corrupting influences, there is a whole thing about what the report data means. It deserves a special blog one day. For example, the 4-year graduation rate can be dramatically different among various programs. Does it mean that one is doing a better job than the other is? Obviously not: programs that are more selective will have higher graduation rates, no matter how hard you work on student success.
There are many threats to data reliability, and even more to their pragmatic validity. We cannot afford to ignore the data, because it is better to see your world through some distorted funhouse mirrors than see nothing at all. It just helps to remember the mirrors are not that accurate, not because of imperfection, but by their very nature.
There are at least two different causes of the intrinsic errors. One has to do with Campbell’s Law, an under-appreciated phenomenon; I wrote about it a few years ago. If consequences of a quantitative indicator are high, the measurement tends to corrupt the very practice it intends to measure. For example, if in my College FTEF/FTES ratio looks high, I will be thinking of ways to record both indicators to make us look better. Trust me there are ways.
However, there is something Campbell did not know, because in 1979, there were no user-input databases at every organization. Let us call this the Sidorkin’s Law: Categorical scarcity leads to workarounds that will corrupt data input. When designing a database, one cannot use a very large number of fields/categories. That would make the database unwieldy and unmanageable. It would be impossible to produce a usable report. The whole point of database is simplification, standardization of information. If you want to compare, say, two colleges, you have to make a call: let’s call student teaching something similar to nurses’ clinical practice, for example “supervised field experiences.” One can see how they are similar, but they are also vastly different. The need to measure and compare brings about emphasizing the similarities and ignoring the differences. However, life is always just a bit more complicated than any set of categories, and people learn to operate within the limits presented by the database designers. Considerations of convenience force us enter data in the fields and categories that were not initially intended.
For example, one of our program areas has struck a deal several years ago that when they teach larger classes, they can get an extra workload credit. However, the workload database did not have a specific category to reflect such an arrangement. Moreover, there was initially a rule that synchronized student credit hours with faculty workload, a very sensible one, by the way. Therefore, someone came up with a work-around: We record these units as Discipline-Based Research. No one remembers what it was intended for, but it was a convenient way to solve the data input problem. As a result, our overall instructional workload units looks smaller than they actually are. This is just one small example; I can name a dozen or so workaround fixes like this. Cumulatively, they can significantly affect the reports that come out in the end. This is why I always prefer to see the raw data, understand the sources and the formulas used in a report, and never trust prepackaged reports.
On top of these two data corrupting influences, there is a whole thing about what the report data means. It deserves a special blog one day. For example, the 4-year graduation rate can be dramatically different among various programs. Does it mean that one is doing a better job than the other is? Obviously not: programs that are more selective will have higher graduation rates, no matter how hard you work on student success.
There are many threats to data reliability, and even more to their pragmatic validity. We cannot afford to ignore the data, because it is better to see your world through some distorted funhouse mirrors than see nothing at all. It just helps to remember the mirrors are not that accurate, not because of imperfection, but by their very nature.
No comments:
Post a Comment