Arne Duncan should check his facts

Arne Duncan, the Secretary of Education, has just spoken to release his plan for teacher education reform. Once again the plan cites the McKinsey report : “only 23% of all teachers, and only 14% of teachers in high-poverty schools, come from the top third of college graduates.” OK, let’s go to the report. It in turn, cites these numbers and the source is “Derived from the US Department of Education, NCES, 2001 Baccalaureate and Beyond Longitudinal Survey.” Well, this does not shed much light on the methodology, so I asked them. But in the meanwhile, let’s look at the raw data  to see how they found it out.

How do you actually identify the top third of college graduates? Note, they don’t cite the ETS report on SAT scores, also erroneous. We certainly do not rank students in our graduating class, unlike some med schools do. The survey has only a few categories related to performance:

• Overall Undergraduate GPA
• GPA in Undergraduate major
• Graduated with academic honors
• Received incomplete grade
• Repeated class to earn higher grade
• Withdrew from course due to failure

What do we see? Graduates who work in Education (which is mainly school teachers) have the third highest average Cumulative GPA, the highest GPA in the major, the second highest number of people graduated with honors. They are in comfortable thirds and second quartiles on the three other quality measures. This is consistent with our data for just one institution. I don’t know if my data table will re-run for you, but you can easily rebuild it, or see my exported table.

Anyway, as Russian say “Either I am stupid, or these skis don’t slide” (Don’t ask). But I really would like to know how is it that the secretary of education finds is acceptable to cite a non-refereed source in a major policy speech, and how that source can publish data without at least explaining its methodology.

Do master’s degrees matter?

In the last few months, I happened to have many exchanges on this particular question. Does it make sense to require teachers to continue their professional development in a formal academic setting? Do master’s degrees or their equivalents matter? All these talks made me realize how differently people view research depending on how they perceive and understand it. In any public debate it is important to assume that your opponent is neither evil or stupid, and that differences may arise from different, unspoken assumptions, which in turn, created by different life experiences.

Those of us who struggled through a doctoral dissertation or another research project, walk away with a deep but ambivalent feeling. On one hand, we know just how messy the research process is, and how deep is our collective ignorance about the world. On the other hand, we learn to tell the few things we really do know from the mountains of common sense, nonsense, and political hype. Having a doctoral title does not make anyone smarter, but it does add both a specialized expertise and a particular ethos of looking at research claims.

For example, here is the summary of current findings, which I borrow from Dan Goldhaber and Michael Hansen’s recent article:

First, teacher quality, measured by value-added models (VAMs), is the most important school-based factor when it comes to improving student achievement. For example, Rivkin et al. (2005) and Rockoff (2004) estimate that a one standard deviation increase in teacher quality raises student achievement in reading and math by about 10 percent of a standard deviation – an achievement effect that is on the same order of magnitude as lowering class size by 10 to 13 students (Rivkin et al. 2005). Second, teacher quality appears to be a highly variable commodity. Studies typically find that less than 10 percent of the variation in teacher effectiveness can be attributed to readily observable credentials like degree and experience levels (e.g. Aaronson et al. 2007; Goldhaber et al. 1999; Third, while the evidentiary base is thin, it appears that only a strikingly small percentage of teachers are ever dismissed (or non-renewed) as a consequence of documented poor performance.

I am not going to dispute any of these. But when I read it, I think immediately: “the most important school-based factor” is teaching, yes, but the school itself is probably 15-20% of influence over student outcomes. The rest of it is the social circumstances, so why are we focusing so much attention and money on the 15%, while almost ignoring the rest: early childhood and expanded learning programming, and taking care of children’s nutrition, health, and young parent education? Even if a miracle happens and we put an outstanding teacher in every troubling school, it is not going to solve the problem.

I also reflect on how pathetically little we know about the reasons the good teachers are good. The definition of a good teacher Rivkin uses (I believe it is actually Sanders and Rivers definition from their 1996 paper) is tautological. You’re good teacher when your students progress well, this is how we know you’re good. This is more of a mystery than a finding useable in public policy decision-making. We don’t know how to predict who is going to be good, nor do we know how to improve people’s performance. We don’t even know if their success is owed to some intrinsic features of their mind and character, or to their training, or to the kind of work environment and support they receive when they start. It is very likely that a combination of all three is at work, but no one could back this up with a large-scale study. And finally, on the point that academic credentials do not predict teacher performance. I also happen to know that none of these studies so far disaggregated between degrees in education and all others. Of course, if you a teacher of Chemistry and are doing a degree in fashion design, it is likely to detract you from doing your job. But if you are an elementary classroom teacher, and you are in, say an M.Ed. in Reading or TESL, we can expect impact on student learning – no one just got around measuring them.

And that’s the problem – most people do not realize how little research is done in education. Same scholars who are now so excited about the teacher factor (economists, mostly), were commenting for decades that teaching does not seem to matter at all. It does not mean the quality of teaching was unimportant for 30 years, between 1966 (The Coleman Report) and 1996 (Sanders and Riversmpaper). The reality did not change, but our thinking has changed somewhat, and will change again. Just because no one has done a good study on measuring teachers’ academic credentials on their performance, does not mean it does not exist. It is also frustrating that much of great research that is outside of the simple linking factors to student test scores, gets no public recognition, and is routinely ignored by policymakers (just browse through recent papers of P. Grossman, D. Boyd, M. Cochran-Smith, L. Darling-Hammond, B. Floden, and many others).

But someone else looking at the same paragraph, has a very different reaction. Of course, they think, we need to find out how much student growth each teacher is responsible for, and get rid of bad teachers. Of course, they say, we need to stop requiring master’s degree of practicing teachers, because they do not matter. A similar inconclusive set of research findings exist about teacher certification, so let’s get rid of teacher training while we are at it, and replace it with short alternative training programs.

Research is a funny thing, it is double-edged. It exists to correct our common sense assumptions. However, in unskilled hands, tidbits of good research can be used to make huge policy mistakes. With the research information becoming readily available to anyone, many smart, well-intended, but unprepared people are trying to interpret it. It happens not just in education – we have millions of amateurs reading medical research, and forming passionate if not militant interest groups. Thank god, very rarely do they make a visible impact on policy.

Within the educational research community, most people are trying to behave honorably, and always disclose the limits of what we know. But their caveats and disclaimers don’t make it into the media, and ignored by overeager reformers at all levels. For example, Dine Ravitch spelled out in May of 2010 that none of the key provisions of the Race to the Top program can be backed up by research. To my knowledge, no one has disputed her claims. And yet thousands of people around this great country believe as if they are acting on a program of reforms backed by research.

We must work on reform, but only when we know our plans are going to make things better, not worse. How do we know? Research is the best option; it often cannot answer questions we are asking. The next best thing is professional consensus: it is not always reliable and subject to professional biases, but at least it provides some ways of sorting nonsense from good ideas. But we must rule out one way of reforming our education; it is when a few passionate, smart, but unprepared people misinterpret research findings and convince themselves that they know all the answers. Many of them tend to believe that things cannot get worse. In that, they are sadly mistaken. 

Mushroom picking in the Rehoboth forest

An early autumn day can make air so transparent, you half-expect to see the past. I almost expect to see my father, mother, my brother and me when I was five in the next clearing. The air is not thick with fragrance like in the Summer; it is colder, lighter; it breathes light covetously while it lasts. The light is different; colder and yet more penetrating, as if coming from a different direction. And the sound too has changed slightly; it is crisper and less crowded. Svetlana and I went to the Rehoboth forest; the name seems to be picked from Tolkien’s books (although it is Biblical, oh well, the same thing; the name is inviting of giants and creepy things). We went to enjoy the early fall’s air and pick mushrooms.

Picking mushrooms is a multisensory game, which very few Americans seem to enjoy. Good, more mushrooms to us! It is biologically programmed in us: walking in the woods, searching, recognizing patterns, shapes, and colors, reading tracks of looking for food. Just like watching water or fire, one never gets tired of it. And then, of course, there is the inevitable talk about which species are edible, and which are not, with touching, smelling, breaking the fungus in question. Ah, we’re two days late, and worms have feasted on mushrooms that could be all ours. We compare our respective families’ folk traditions, remember how we learned this in our childhood, and who taught us. Our fingers turn black and sticky from some mushroom juices. The dog is happy sleuthing without a leash, engrossed in knows what private doggie fantasies. 

People generally overestimate the risk of eating wild mushrooms. Only a couple of species are deadly, and those are easy to recognize. I remember colorful pictures of common poisonous mushrooms – mulhomor and poganka shown in my preschool. My mother points them out to me every time we go to the forest. Many inedible mushrooms will give you a diarrhea, or will taste bad, and not much more. Driving to the forest is a lot more dangerous than eating the mushrooms we pick, with our average Russian knowledge of the forest. I think about risks we take and do not take. I think about my own life – did I take the right risks? Too many or too few? Who knows what the right amount is? Perhaps the dog, but he is preoccupied with his own thoughts, and won’t tell.

I know the future

Like everyone else, I remember the morning of 9/11/01 very well. I was teaching two Foundations classes back to back; it was at BGSU in Ohio. The second plane hit the building at the end of the first class, and during the break, the scale of the event has began to sink in. I told students that if they want they can stay in class and watch the news with me, or go home. I remember telling the students that many people died today, and please think about them and their families.

Two recessions and two wars later, I keep thinking about both fragility and resilience of the human civilization, and this fine country in particular. That nine guys with box cutters can rattle it to this degree is scary and disconcerting. The fact that that the Lower Manhattan now has more businesses - small and large - than before is also quite astonishing. I just came back from the PARCC institute, where people from 24 states enthusiastically and systematically work on new common curriculum standards for children. This somehow impresses me even more, maybe because it was going on exactly ten years after 9/11.

Educators are optimistic not just by inclination, but also by job description. We do things that usually take years and decades to materialize, and we never quite know how exactly our work is going to turn out. We cannot believe that the world is about to end – according to Mayans, or if the Rapture is just around the corner. No one knows the future, except for us. The future has many names and faces; it brings us homework and asks us questions. It wants a better grade, and it cannot quite get things right away, but we can help. But it is quite real; you can look it in the eye.

Teacher Education Innovation Lab

Over the summer, I started to think more and more about innovation (see this June blog). One reason for that is that we failed to get ourselves into the news in any meaningful way. And it just occurred to me that we do not have any news, in the sense that the media would recognize. I also was listening/reading a lot of Harvard Business Review, the Economist, and the Financial Times, paying attention to the discourse of innovation in business. One simple lesson  learned is this: you need to actually spend time and effort on innovation, support and nurture promising ideas, kill the dead branches, and generally have a strategy. It does not happen on its own, or in occasional spurs. And we don’t have a strategy of innovation.

So, here is my plan for this year. We will have a group which I called TEIL. I only have a rough outline of the beginning – first two or three meetings; so we will spend some time talking about our own process. We start with thinking about everything but teacher education: about the world of business, non-profit entrepreneurship, politics, products, services, the internet, the social media, etc. There may be some homework here, where the group’s members will investigate a favorite company, or an organization to see how it innovates. Then we will take a very thorough look at ourselves – what do we spend our time on, how new ideas are born, introduced, how they are implemented or dropped, why and why not.  I want to talk about the quality of experience as the key criteria for improvement – our students’, our faculty and staff’s. Then we should try several structured brainstorming sessions to try to find several new ideas, especially if they fit together.

Just one example: Let’s think about the ritual. For us to make a stronger impact on our students, we need to employ what all cultures in the world do – ritual. We have a few; none is specific to teacher preparation. Why don’t we have admission to Feinstein to be a memorable event? Why don’t we ask them to take a teacher’s oath?

Perhaps at some point we will split into smaller project-oriented teams, and each team will develop a proposal, while other teams will provide feedback. Perhaps by Spring we will have a clearer idea on what innovation support structure we should have in place, and what resources we could muster to support it. By that time, we should have the Advisory board operating, and perhaps it can help by bringing an outside critical perspective. We also do need to look at a few truly innovative ideas that exist in our field today.

That’s what I have so far. Not much, I know, but this is going to be a collaborative and evolving process. How can I sell it to faculty and staff? How can I convince at least some people to come and spend a few Friday afternoons doing something in addition to their regular heavy workload? Well, only this: teacher preparation as a field has not shown a lot of innovation. We placed all our bets on the continuous improvement process spearheaded by AACTE and NCATE. While it is useful and in the long run is going to be effective, it is just simply not enough.

Another argument: I came to work to higher education, because I like to talk to people about ideas. If you do, too, sign up. You only need to make the majority of meetings, not every one of them.