There’s a big, slow-news-weekend story over at Politico today over allegations made by certain Democrats and liberals that the prolific polling firm Rasmussen Reports is biased toward conservative and Republican causes. I have to catch a flight (weather permitting) in a couple of hours, so I don’t have quite as much time to weigh in on this as I’d like, but let me present the Cliff’s Notes version.
The first thing to note is that there are a lot of different ways in which a polling firm might be biased. Rasmussen is most frequently accused of bias because their results are thought to lean toward Republican candidates. Just to pick a random example, for instance, Rasmussen has embattled Democrat Blanche Lincoln down by margins ranging between 4 and 7 points against three potential Republican opponents in her 2010 Senate race, whereas two other polling firms (neither of which, incidentally, are themselves free of partisan ties) have Lincoln ahead against these opponents by margins ranging from 1 to 16 points. Does this mean that Rasmussen is biased?
For me, it doesn’t — not necessarily. It means that Rasmussen has a different model of what the 2010 election is going to look like, one which will feature a more conservative electorate. But that model isn’t necessarily wrong, nor does it necessarily reflect bias. The polling firm Public Policy Polling has also tended to show poor results for Democratic candidates in its 2010 polling, relative to other pollsters like Quinnipiac. But Public Policy Polling is a Democratic polling firm. Are they biased too?
What Rasmussen has had is a “house effect”. So far in the 2010 cycle, their polling has consistently and predictably shown better results for Republican candidates than other polling firms have. But such house effects can emerge from legitimate differences of opinion about how to model the electorate. And ultimately, these differences of opinion will be tested — based on what happens next November. If Rasmussen’s opinion turns out to be wildly inaccurate, that will impeach their credibility, and believe me, we will point that out. Likewise, if they turn out to be right when most other pollsters are wrong, we will point that out too.
Rasmussen’s election polling has tended to be quite accurate in the past. Nor, incidentally, has their election polling has a particularly strong house effect in the past; it is something new to the 2010 cycle. But that’s OK; each election cycle features different dynamics in terms of turnout and motivation, and what might be smart assumptions in one cycle won’t necessarily carry over to the next.
Now, what you do need to be aware of is that Rasmussen’s opinion is one among many. They might turn out to be right — but so might all of the other pollsters who have a different opinion about the electorate. If you’re running a news organization and you tend to cite Rasmussen’s polls disproportionately, it probably means that you are biased — it does not necessarily mean that Rasmussen is biased.
* * *
But there are other respects in which I’m much less sympathetic to Rasmussen’s case. In particular, this has to do with their choices of question wording and subject matter. The Politico question, for instance, points toward an August question in which Rasmussen asked “It’s always better to cut taxes than to increase government spending because taxpayers, not bureaucrats, are the best judges of how to spend their money.” That is not a question designed to elicit the most accurate reflection of public opinion.
Likewise, Rasmussen recently produced a poll in which they purported to describe the Democratic health care plan to their respondents. Several other pollsters have found that support for the plan increases when it is actually described to respondents, but Rasmussen showed no such increase. However, the second sentence in their description reads:
The plans before Congress would prohibit people from choosing insurance plans with lower premiums and higher deductibles.
I don’t particularly know where this comes from; Rasmussen claims that its questions came from a ‘summary of the legislation provided by the New York Times’, but such a depiction of the health care policy appears nowhere in the New York Times article. But there it is in the Rasmussen survey, where it appears to be designed to build a relationship in the respondent’s mind between the Democratic plan and higher premiums.
I’m not saying that Rasmussen’s question wording is always biased. It isn’t. And I’m sure you could find a couple of cases where the wording tend to portray the liberal argument more favorably. But cases like these happen consistently enough with Rasmussen that I’d say it’s a concern. And when they do use unorthodox question wording, nine times out of ten it favors the conservative argument. I would describe this as a form of bias — although it should generally implicate only the poll in question, and not their overall enterprise. In other words, if Rasmussen uses some misleading wording in a health care poll, that means I’m not likely to take that health care poll very seriously — but it doesn’t particularly mean that you should throw out their presidential approval polling, or their Arkansas polling, or their polling on gay marriage, or whatever. Yes, this does mean a bit of extra work — we have to scrutinize each particular poll for potentially misleading wording — but that’s something that we should be doing more of anyway.
I also have some questions about Rasmussen’s choice of subject matter. In particular, they have a knack for issuing polls at times which tend to dovetail with conservative media narratives. Rasmussen, for instance, recently decided to issue a poll about Ben Nelson’s standing in Nebraska in light of his vote for health care, which is unpopular in the state. But did they issue a similar poll for Joe Lieberman, who until recently looked like he might vote against the health care bill — and who opposed the public option, a policy which is very popular in Connecticut? No. They did poll Connecticut in December, but they asked only about Chris Dodd, and not Lieberman. Certainly, there’s nothing wrong with polling on Ben Nelson — or with not polling on Joe Lieberman, who gets polled frequently by home-state pollster Quinnipiac. But if you see this sort of pattern consistently, then it may reflect a certain kind of bias.