Slog News & Arts

Line Out

Music & Nightlife

« Gonzo Porn Douchebag Wanted on... | What's Next, Summary Execution... »

Tuesday, September 18, 2007

Scientifically Analyzing an Unscientific Poll

posted by on September 18 at 10:42 AM

Eli Asks:

Or, is the Slog tracking poll’s margin of error much, much greater than +/- 6 percentage points? (Paging all math and statistics brainiacs.)

Slog responds:

Computing the margin of error on this type of poll would be like computing the margin of error on reading chicken entrails.

THAT’S WHAT UNSCIENTIFIC MEANS, YOU MORON.

Posted by elenchos | September 18, 2007 9:50 AM

and

Are people over Gore? Yes, about 8 years ago.

Is this due to growing satisfaction with current candidates? No.

Is the margin of error much greater than +/-6%? Certainly yes. First, sample size is significantly different (998 in August vs 620 in September). That alone will skew your results.

Who responded in the first poll because seeing Gore listed was funny? It wasn’t so funny the second time, so they didn’t bother to respond. In August, the “with Gore” poll had 86% the number of respondents as the “without Gore” poll. In September, that dropped to 81%, a difference of 5%!

As for the 39% who still picked Gore in the last poll? I’m guessing they also watch Seinfeld reruns and think they are every bit as funny as when they first aired.

Posted by Mahtli69 | September 18, 2007 10:10 AM

My turn.

First of all, the “margin of error” in any poll is a pretty meaningless value when deciding if a poll result is valid.

When the media print sentences such as “the margin of error is plus or minus three percentage points,” they strongly suggest that the results are accurate to within the percentage stated. That is completely untrue and grossly misleading. The media’s intentions are honorable. They want to warn people about sampling error. But they might be better off assuming — as most of the readers surely do — that all surveys, all opinion polls (and, indeed, all censuses) are estimates, which may be wrong.

As I’m fresh out of chicken entrails, let’s start with the reasonable assumption a slog poll is a decent sample of the slog-poll-responding population.

The standard deviation of any poll result is equal to 1 / sqrt (n), where n is the number of poll respondents.

The n for the first and second polls were 998 and 620 respectively. Therefore, the standard deviations of the two polls are 0.03 and 0.04. If we stretch out to three standard deviations, we’re still at a fairly narrow amount of error.

So, if we ignore the problems of double voting (with different web browsers or computers), unequal sample sizes, non-random selection, and accept this is a sample of slog poll respondents only, we can probably say the difference is statistically significant, and completely meaningless.

Consider this basic argument true for just about any poll you read in the lay press: statistically significant and utterly meaningless.

RSS icon Comments

1

"I am sure you have had the experience of making up your own mind on a question and then discovering, on the evening news of the same day, that only 23.6 percent of people agree with you. Ought you to be depressed or disconcerted by this alarmingly exact dissection of the collective brain? Only if you believe that a squadron of undertalented by overpaid pseudo-scientists have truly and verifiably arrived at this conclusion. And perhaps--indeed I would argue, in any case--not even then. [...] To the consumer the 'poll'... may seem like a mirror of existing opinion. But to the one who produces it, the poll is a swift photograph of the raw material to be worked upon."
-- Christopher Hitchens, Letters to A Young Contrarian

Posted by Katelyn | September 18, 2007 11:15 AM
2

"Consider this basic argument true for just about any poll you read in the lay press: statistically significant and utterly meaningless."

To be fair, your average Gallup poll is still more meaningful than this poll, as they are more likely to get a reflective sample of the population with their methods, and reweight their responses based on known demographic disparities in the sample. Good polling is possible; it's just hard to do.

Posted by tsm | September 18, 2007 11:20 AM
3

Dude, that was completely worthless. What you assume below is exactly what margin of error is trying to capture.

"let’s start with the reasonable assumption a slog poll is a decent sample of the slog-poll-responding population."

And then you compute the SD, which leaves no doubt that you're completely asea. Get thee to a stats class stat!

Posted by Bison | September 18, 2007 1:24 PM
4

"let’s start with the reasonable assumption a slog poll is a decent sample of the slog-poll-responding population."

A poll is supposed to be representative of a larger population. In a scientific poll, you sample in such a manner that you can extrapolate the results. The conclusion "this poll is representative of the people we polled" is useless.

Posted by Gabriel | September 18, 2007 2:02 PM
5

What #3 and #4 wrote -- beat me to it.

Posted by twee | September 18, 2007 2:44 PM
6

@4 - I think you could extrapolate it beyond Slog to include the unemployed and those that slack-off at work on the internet.

@3 - Thank you. Assuming you got a decent sampling is not reasonable.
Assuming the people who lost interest after the first poll were overwhelmingly "Gore voters" is actually much more reasonable.

There is no doubt that the August and September poll results are skewed by the fact that fewer people were interested in filling out a poll for a non-existent candidate twice. Therefore, the results were both statistically insignificant and utterly meaningless.

But, I still enjoy talking about it anyway, so nyaa! And, actually, there is a doubt, even though I say there is no doubt, so there is no doubt that poll results can be interpreted in a way that suits the interpreter/spin-doctor. Does that make sense? Good.

Posted by Mahtli69 | September 18, 2007 2:59 PM
7

An easier way is ask yourself:

1. Did the poll have clear questions with RANDOMLY selected population of at least 525 people? If phone, did they randomly dial or use phone books (miss cell phone users).

2. Even if random large sample, 19 out of 20 polls will be way off. Maybe by a bit - but maybe by a whole heck of a lot.

Posted by Will in Seattle | September 18, 2007 3:19 PM
8

This repeats #7's point, but it bears repeating.

"Let’s start with the reasonable assumption a slog poll is a decent sample of the slog-poll-responding population."

Sorry, but from a statistical point of view, this the Slog poll is NOT a decent sample of anything: it's not a RANDOM sample. Slog readers CHOOSE to participate in the poll, or not. All these online polls suffer from participation (self-selection) bias -- those that want to make their point vote (usually multiple times), those that don't have an axe to grind don't.

Margin of error is based on the strong assumptions of a truly random sample. While this makes sense for a Gallup poll where participants are selected (if imperfectly) randomly, it is totally meaningless for the Slog poll and all self-selected online polls.

Posted by David | September 19, 2007 9:58 AM
9

I don't want to be critical or pile on the complaints. But it might be helpful to ask whether or not one would estimate standard error, not standard deviation, with the formula one over the square root of n. I'm not sure what role standard error would play in any of this discussion -- I thought Eli wanted to know what the margin of error was.

That is, what is the random error? And the reason that the question itself is invalid is that a non-random poll can not have a random error. It almost certainly has systemic error, but there is no way to guess which way that systemic error skews the results.

Posted by elenchos | September 19, 2007 10:32 AM
10

Margarine of air or not, I think these polls are set up to take one's own pulse and thereby conclude that one's opinion shapes further opinion. Sew knot.

For all you Al-Gore-Rhythms: he's won an Oscar, now an Emmy. Why should he run for president when he has the clear oppportunity to join a very select group of nine who've won the Big Four? next a Grammy for "Al Gore Speaks" then a Tony for "Internezzo: the Musical". Gore then could become the fourth elected president to win a Nobel Peace Prize. Google aweigh, kids.

Posted by KY. COL. of TRUTH | September 19, 2007 12:16 PM

Comments Closed

In order to combat spam, we are no longer accepting comments on this post (or any post more than 14 days old).