Not all polls are created equally. Know how to read them and what to look for to cut through the noise

Key Points:

  • Reading a poll correctly requires knowing something about the way the poll was constructed.
  • The way the poll was conducted matters and will tell you about poll stability and who is more likely to respond.
  • The questions asked matter. Polling on parties versus party leaders can generate different numbers and give you important information about whether leaders are boosting or suppressing votes.

Public horse-race polls tend to have fairly wide dispersions. Most (normal) people pay little attention to politics outside of election day. And swing voters are more likely to be low-information voters. They have lives, jobs, kids and interests outside of politics. They are unlikely to have strongly formed thoughts about who they would vote for outside of election day.

But there are other reasons polls differ. Polls are conducted differently and ask different questions. We rarely talk about these things when a poll comes out, but we should.

How the Poll Was Conducted

Interactive Voice Response (IVR) polls are pre-recorded phone calls where you register your opinion by pushing a button. There is no person on the phone. This is the cheapest way to conduct a poll. Many people hang up. As a result, IVR polls tend to reach more engaged and/or angry people. This makes IVR polls more volatile. They also tend to pick up the latest shifts faster than other types of polls, even as they tend to overstate those trends.

Online polls have the opposite problem. Respondents are picked from among a pre-determined pool of people who have agreed to participate, often because they are given some incentive, like cash, a gift card, or reward points. The polls are conducted using an online survey. These polls are not a true random sample and cannot report errors of +/- an amount 19 times out of 20. As a result, these polls tend to be much more stable over time as you are pulling from the same group of people. Some of the groups may have biases.

Old-school telephone polls are somewhere in the middle. Their downside is cost, because they rely on live trained callers, and people also tend to not complete full surveys. Their hang-up rates are lower than IVR, which means they don't bias towards more engaged and/or angry people. They can also ask open-ended questions – as opposed to questions with three, four, or five predetermined responses - that are more difficult to conduct using IVR polls. They tend to be more stable than IVR polls and more responsive than online polls.

There are also a bevy of firms experimenting with AI-driven polling or data scraping, but they are less common. A topic for another day.

The Question Asked

There are generally two questions asked in horse-race polls: one asks which party you intend to vote for, and the other asks which party you support with the party leader's name.

These two methods produce different results, particularly outside of voting day.

Whenever I have had a say on internal polling methodology for a campaign I've been involved in, I insist on split-sample polling. Half the sample gets the party question, and half gets the party+leader question. Other than the final days of an election, when the party responses tend to converge towards the party+leader responses, the two questions can produce different results.

There are at least three reasons for this:

  1. Some leaders are more popular, or less popular, than their party. The “Conservative Party of Canada” (CPC) polled ahead of “the CPC led by Stephen Harper” until after the global financial crisis when Stephen Harper became more popular than the party. Our party results went down in the final days of the 2004 and 2006 elections. They went up in 2011, in both cases moving towards the party+leader results.
  2. Some people think about provincial parties instead of the federal parties when asked who they will vote for federally. Our internal BC numbers were typically off until the final days of the campaign in BC because many federal Conservative voters were then provincial Liberal voters. Party-only numbers were further off in BC than elsewhere and came up substantially as people got closer to going to the polls. The party+leader numbers were less affected.
  3. Our elections are increasingly presidentialized. People increasingly vote for who they want to be Prime Minister, not who they want to be their local Member of Parliament. In my view, including the leader is a better way of asking a horse race question.

So the next time you see a poll, before celebrating a good result or sulking in despair after a bad one, ask yourself: How was the poll conducted? And what question was asked?

You're welcome.

The link has been copied!