Wednesday, March 7, 2012

Teacher Evaluation Formula Fails to Evaluate Teachers

Posted by on Wed, Mar 7, 2012 at 3:45 PM

The big K-12 education "reform" legislation passed this session was SB 5895, establishing a uniform system of evaluating teacher performance... because who doesn't want to separate the good teachers from the bad, right? I mean with uniform teacher evaluations, principals can work with the underperforming teachers to help them improve—or more likely, fire them, or harass them into quitting. But either way, it's hard to argue against at least knowing how our teachers are doing. Knowledge is power!

Except, how exactly does one objectively measure a teacher's performance when it's the students who are actually taking the tests?

One metric is based on the "value-added model" (VAM), which, according to the Gates Foundation (which absolutely loves VAM) attempts to statistically measure the impact of a teacher on student achievement by adjusting for each student's starting point coming into the class, and then comparing the student's improvement to similar students elsewhere. If a teacher's students outperform their peers, that constitutes positive student growth or "value-added."

Given the Gates Foundation's advocacy, it's not surprising that SB 5895 appears to adopt this approach, mandating that "student growth data must be a substantial factor in evaluating the summative performance of certificated classroom teachers..."

So how reliable are the VAM formulas already in use elsewhere? According to a huge dump of teacher evaluation data recently released by the New York public schools, not so much. Over at, a blog serving Teach for America alumni, Gary Rubinstein has been analyzing NYC's value-added data, and finding it surprisingly useless. For example, take the following scatter plot he generated of 665 teachers who taught the same subject at two different grade levels during the same year (2010). The x-axis represents the teacher's VAM score for one class, and the y-axis represents the same teacher's VAM score for the other:

  • Gary Rubinstein |

If the NYC value-added formula accurately measured teacher performance, you would expect there to be some correlation between a teacher's ability to teach, say, 7th grade math and that same teacher's ability to teach 8th grade math. That would show up on this graph as a cluster of data points on a diagonal line rising from bottom-left to top-right. Instead, the VAM scores are almost completely random.

Rubinstein found a minuscule correlation coefficient of only 0.24. The average difference between the two scores was nearly 30 points, and 10 percent of teachers had differences of more than 60 points. And he got similar near-random results when plotting the value-added scores of elementary school teachers teaching different subjects to the same students in the same classroom in the same year.

"I hope that these two experiments ... bring to life the realities of these horrible formulas," Rubinstein writes, adding that "the absurdity of these results should help everyone understand that we need to spread the word since calculations like these will soon be used in nearly every state."

Including, alas, Washington, thanks in large part to "reformers" who find it cheaper and/or easier to blame teachers (and their unions) for our schools' problems than to advocate for the tax revenue necessary to pay for reforms like universal pre-school and full-day kindergarten that actually work.


Comments (15) RSS

Newest First Unregistered On Registered On Add a comment
Here we go, back to the late nineteenth and early twentieth centuries of the robber baron days when those wonderfully nice people of wealth donated their riches to us poor people in the form of scholarships and music halls. All of which is unattainable for us poor, but which bears their name in the form of 'generosity'. This completely ignores the REAL social issues we face today in our society, again. Instead, it places the teachers in the scapegoat position, and we all know that SOMEONE has to take the fall for the inept leadership our legislative body has provided us over the last 40+ years. Systematically defunding social and educational programs or spending money on frivolous programs that fail. When will we learn? What can we do to fight back? Go to
Posted by bluesky on March 9, 2012 at 10:16 AM · Report this
Goldy 18
@15: There's a kernel of truth in what you write, unfortunately it's wrapped up in some elitist bullshit.

There are a lot of great public school districts, and many of them are in affluent suburbs. What do this districts have in common? Lots of money, and well-educated professional class parents who are both steeped in a culture of education, and who possess the tools to steep their children in the same.

So yes: a student's family life plays a huge factor in predicting a student's academic success.

But you seem to frame it as some kind of moral failing on the part of the parents, as opposed to the result of a cycle of poverty and unequal opportunity. And even if we could blame some sort of cultural "degradation" as you imply, how can we blame the children?

The fact is, great schools cost a lot of money, and educationally disadvantaged children require even more money to try to make up for what they're lacking at home. You can't make up for all of it, but intensive, round the clock social services can help some children excel academically and rise out of poverty.

The alternative is to just give up on the vast majority of children whose families are unable to support them academically.
Posted by Goldy on March 8, 2012 at 11:40 AM · Report this
Of course, it's all bullshot!

They're going after the teachers, because they are union and they want to privatize (as in monopolize and securitize) public education, tracking it back to the "good ole days" when only the super-rich received a decent education --- although it's almost there now, excepting for the douchey crime schools known as Harvard, Princeton and Yale! (all should be preemptively nuked!)

The final attack, once everything's been privatized, are the coppers.…

They've started wholesale over in the UK, while in America it's only been in the smaller towns in California, Georgia, Indiana, Michigan, etc. (Gov. Wanker of Wisconsin, when he was county exec, privatized the court guards to G4S Wackenhut, since shortened to G4S, which is the international privatizer of choice, financed on Wall Street via Goldman Sachs. Wanker's privatization move was later overturned by Wisconsin courts, stating he didn't have the authority to do that as county exec.)

G4S's main lobbyist in D.C. is Jeffrey Starr (note similarity to name of founder of AIG, Cornelius Starr, whose nephew and trust fund baby, Kenneth Starr wasted endless taxpayer funds on something having to do with a Clinton impeachment for BJs????), who was a top dog with D.I.A. and then with Goldman Sachs' Business Intelligence Group, before moving to Wackenhut, where GS did their merger/takeover by G4S, where Jeffrey ended up.

Always nepotism, always keeping it in the family!
Posted by sgt_doom on March 8, 2012 at 11:26 AM · Report this
I would have loved if the last paragraph had ended like this:

"...easier to blame teachers (and their unions) for our schools' problems than to advocate for the parent involvement, student responsibility, and teacher respect that are necessary to solve the larger problem of American society's gross undervaluing of education."

Universal pre-school and full-day kindergarten are lovely, but they are not solutions for the day in day out degradation of education happening for all ages and at all levels of society. There is a massive societal component that must be addressed here, period. No amount of spending or testing or teacher training will solve anything if parents, students, and society in general continue to treat education like a trip to Nordstroms. Teachers are not here to serve your whims, we are not here to babysit your children, and you don't get a free replacement education when you spill wine all over it.

Respect teachers, respect yourself, LEARN (whether you think you're teacher is an idiot or not) and take part in your own or your children's education.

And the real beauty is; that's all free.
Posted by ace9415 on March 8, 2012 at 12:55 AM · Report this
If only Bill Gates had just bought dozens of yachts like other billionaires.
Posted by sarah70 on March 7, 2012 at 11:05 PM · Report this

Maybe you're asking too much.

Do you have an "exceptional" service tech at Jiffy Lube?

Does it make a difference?

Warren Buffet said that the exceptional business can have mediocre management.

Maybe a school is such a good system, any reasonable person can fill the slot. There is no "exceptional teacher".
Posted by Supreme Ruler Of The Universe http://_ on March 7, 2012 at 10:49 PM · Report this

Or, it's spectacularly accurate, and what it is saying is the effect of a particular teacher on a class is nil...given that they simply perform the basics of the task.

Posted by Supreme Ruler Of The Universe http://_ on March 7, 2012 at 10:48 PM · Report this
I'm a math teacher. When I saw that graph, I laughed pretty hard. Then again, I laugh pretty hard at pretty much everything our legislature has done to teachers recently. I can't wait to pay more for less health care, have my salary and teaching status based on ineffective assessments of students (I mean, data), and basically have my profession continually attacked for the sake of blame.

I guess that even when students graduate, they still don't respect their teachers.
Posted by thedonproject on March 7, 2012 at 10:45 PM · Report this
Goldy 10
@9: No, what this data shows is that NYC's value-added model does not accurately measure teacher performance.
Posted by Goldy on March 7, 2012 at 10:27 PM · Report this

According to the data, basically two illusions are shattered.

The first is that there are no "better" ways to teach a class. But the second is that given the first, it would seem any reasonably trained person could in fact run a class.

So it is not that charter schools are superior -- it is that they are no worse, and so then the issue is cost.
Posted by Supreme Ruler Of The Universe http://_ on March 7, 2012 at 9:26 PM · Report this
Reverse Polarity 8
Great. We just adopted a new "reform" teacher evaluation system that is exactly as accurate as a random number generator. Good job, legislature!

I'm sure that will be much more effective than, you know, actually fully funding education.

Posted by Reverse Polarity on March 7, 2012 at 8:41 PM · Report this
Maybe, and maybe only by definition, only the very best and very worst teachers get consistent result, and that otherwise---as mentioned above---the variability of the students dominates the outcome.

In further news, C.E.O.s seem not to really matter all that much to companies, with the exception of the very best and the very worst, and feeding random noise to a bad classifier will show you all _sorts_ of unreal relationships.

I think it better to admit ignorance than to cling to a bad source of "knowledge", but people seem to prefer a foolish certainty---primate band politics hardwired-in.
Posted by Gerald Fnord on March 7, 2012 at 7:42 PM · Report this
Absolutely brilliant.
Posted by madcap on March 7, 2012 at 7:40 PM · Report this
Any high school teacher can tell you any given lesson plan, strategy, approach can inspire a wonderful reaction in first period and a disasterous reaction with another group of kids in sixth period. Units of study that are dazzling one year, lose their luster in other years. Teaching is part science and part art. Not always with consistent or predictable results. But reformers who have never been in the classroom always think they know best and that coprorate models should be applied.
Posted by 1971 on March 7, 2012 at 4:43 PM · Report this
Urgutha Forka 3
Frankly, I'm surprised he even found a .24 correlation from that mess.

Conservatives always argue that you never get results by just dumping money into a problem. But isn't that how their hero Reagan "won" the cold war? By just dumping money into the military?
Posted by Urgutha Forka on March 7, 2012 at 4:16 PM · Report this

Add a comment

Commenting on this item is available only to registered commenters.