On Monday, the polling company ICM released two EU referendum polls. One was carried out online; it showed Leave ahead by four points. The other was a telephone poll that gave Remain a 10-point lead.
According to The Guardian, the two surveys were conducted at the same time and deployed as "similar vote adjustment methodologies as possible".
So what is actually going on?
The short answer is: ¯\_(ツ)_/¯. The longer answer is a bit more complicated.
Although the difference between these two sets of figures is particularly stark, they follow a similar pattern to other polls. Telephone polls are consistently showing strong Remain leads, while online surveys paint a much closer contest, with several putting Leave ahead.
Experts are divided on which method they believe to be better. According to Matt Singh of Number Cruncher Politics, one of the few analysts to correctly predict last May’s general election, polling carried out over the telephone is more reliable. In a paper published by Singh and James Kanagasooriam, of polling company Populus, the duo claimed phone polls are more effective at compelling respondents to choose between the options that will appear on the ballot paper – remain or leave – and more closely align with the demographic makeup and social attitudes of the electorate. They concluded that internet polling overstates the level of support for those who want to leave the EU.
However, equally convincing arguments have been put forward to suggest that online polling will have the upper hand in the referendum. The CEO of polling company YouGov, Stephan Shakespeare, suggests the anonymity of online polls makes it easier for people to give their real opinions – while the convenience makes them better at reaching social liberals.
Normally the past would be a useful guide to navigating these differences. Unfortunately, in this case, last year's general election is not very helpful. Despite telephone and online polls telling the story of two rather different campaigns, in the end they both reached the same (incorrect) destination. In fact, the post-election polling inquiry into what went wrong found that there were no significant differences between the modes. The average Labour lead in the final polls was 0.2% in phone polls and 0.2% in online ones. In the actual election, the Tories won a small majority, with 38% of the vote against Labour’s 31%. The average error compared to the final result was 1.6% for both phone and online surveys.
The two methods' performance was also relatively similar in 2014’s Scottish independence referendum.
Polling was last tested earlier this month during local elections. On that occasion, the polls, which were mostly carried out online, were broadly right. Although they overestimated support for UKIP, possibly indicating that they might be doing so in referendum surveys, it is safe to assume pollsters will be adjusting for such effects.
And as Martin Boon, ICM director, points out: “The narrative that phone polls are more likely to be right ignores some fundamental flaws in phone methods. Labour supporters are continually oversampled by phone, and that may matter more than those same phone polls missing out on supposedly pro-Remain types, who are disproportionately less likely to turn out to vote.”
The polls could of course converge, like just like they did last May: One thing both pollsters and experts agree on is that a lot could still happen over the next five and a half weeks. With just under 40 days to 23 June, the proportion of people who are still undecided is still in double figures. But as things stand, phone and online polls cannot both be right.
Updated #EURef polls table after the latest ICM/Guardian pair (forecast update to follow...) https://t.co/hUxOTPhRiB
The second factor most agree on is that voter turnout could prove to be the decisive factor. But the issue here is that polls haven’t been particularly strong in anticipating the proportion of the British public that actually show up come election day.
Common knowledge indicates that low turnout would favour Brexit, yet untangling what drives turnout is more complex. Young people lean towards Remain but are less likely to cast a ballot than older people, who vote in great numbers – and have so far tended to lean towards exiting the EU.
However, undecided voters usually split in greater numbers for the status quo. Meanwhile, higher social grades are more favourable to Remain, and they tend to vote in greater proportions than people classed in lower social grades (but there are more of the latter).
Age 43 is the crossover point between Remain and Leave, according to YouGov via @MattChorley's Red Box email
The real state of affairs may well sit somewhere between the two methodologies, but a simple average between the two is not a robust solution if it turns out that one set of numbers is right while the other is wrong.
So once all these indicators are factored in, we are left with only hints of evidence to help us pick which method we should be relying on. The truth is that nobody really knows with a sufficient degree of certainty what is actually going on. As Boon puts it, “If you want to ask me ... the answer you’d get is 'I just don’t know'.”