Trump Is Yet Another "Destabilising" Event The Pollsters Failed To See Coming

The failure to anticipate the Brexit vote, the Tory election win, and Trump is a wake-up call. "Dramatic, destabilising things" are happening to our politics.

Andrew Kelly / Reuters

Political scientists in the UK feel they have woken "like survivors after a nuclear holocaust" after opinion polls failed once again in predicting Donald Trump's election.

Polls would now be trusted less, experts told BuzzFeed News, because researchers "failed to grasp" the tectonic shifts in turnout and the factors behind events like the shock Tory general election victory in 2015 and the EU referendum, which the on-the-day YouGov poll wrongly suggested would result in a Remain win.

Hours before US voters chose Trump as their president, several major pollsters said Hillary Clinton had at least a 90% chance of beating him. One poll from Reuters and Ipsos claimed she was on track to smash the 270 Electoral College votes she needed to win while the Huffington Post forecast that Trump had a mere 2% chance of taking the presidency.

Shock results are “obviously territory with which we in the UK are now very familiar”, said Martin Boon, the director of polling company ICM. He highlighted the dramatic failure to foresee the Conservative win in the 2015 election, "which there is no escape from".

One British political scientist has publicly apologised, saying he and his colleagues had "failed consistently to give you an accurate account of what is out there... we need to do better".

Rob Ford, a professor of political science at Manchester University, tweeted his dismay at the number of recent election results "which elections & public opinion researchers regarded as highly unlikely".

Twitter

Twitter

"Dramatic, destabilising things are happening in our politics. We have not got a good grasp on their depth or magnitude," he wrote, noting that Jeremy Corbyn winning the Labour leadership and Trump's presidential nomination also blindsided polls.

Political scientists were aware of the motivations behind votes for Brexit and Trump, Ford said, but they "utterly" failed to grasp their "scale and intensity".

Twitter

Twitter

Twitter

As the dust settles, some of Britain’s leading polling companies told BuzzFeed News the last two years can teach us plenty about why the polls once again failed to get it right.

It really is like Brexit

Vincent Kessler / Reuters

The backdrop to Trump’s win is eerily similar to that of the UK’s EU referendum in June. “What we have are a number of consistencies: a narrow two-horse race and a lot of polls coalescing around the closeness of the race, which obviously gives statistical get-outs in some ways,” said Boon.

“I was quoted left, right, and centre saying I didn't know what the Brexit vote would be because there are so many difficulties in this race. I think some of the conditions which we have here must have translated over into the States.”

In both cases the race was so close the margin of error meant polls were more likely to be wrong, according to Ben Page, the chief executive of Ipsos Mori: "You’ve had this series of events where the outcome is very close and that's where inaccuracy becomes tricky."

Anthony Wells, the head of research for YouGov, claimed that the margin of error that wrongly predicted a hung parliament in the 2015 UK election wasn't much different from the margin in predictions for Labour's landslide 1997 election win – it just mattered more as the race was so much closer.

If one side is set to win comfortably – as is expected in the next UK general election at present, for example – polls are unlikely to attract attention. "The Conservatives are likely to win by such a margin that if the polls aren’t perfect no one will notice,” said Page, who added that “you only hear about the polls that go wrong”.

"Shy" Trump voters – the cousins of "shy Tories"?

Carlo Allegri / Reuters

Some have also blamed the higher-than-expected Republican vote on a “shy Trump voter” phenomenon – another concept well known to Brits after "shy Tories" were reported to be behind the Conservatives picking up more votes than expected in the 2015 election.

This effect has also been called "social desirability bias" and is based on the idea that people may be ashamed to tell interviewers that they will vote for a certain party or candidate.

“We in the UK have a lot of experience of that,” said Boon. “It's the shy Tory phenomenon in our context, otherwise known as the spiral of silence, in which people are embarrassed to admit that they support a particular party or candidate before elections.”

But others are sceptical about this idea. “It’s incredibly difficult to test,” said Wells. “Either people lie before the election and then tell the truth afterwards, in which case you can’t distinguish it from a late swing where people change their minds. Or, if people lie to you beforehand and then after the election you ring them up again and they still lie, it’s impossible to detect.”

An inquiry from the British Polling Council could find no evidence for "shy Tories" in the 2015 UK election, but a study this month from Morning Consult suggested US voters were more likely to express pro-Trump views in impersonal online polls, compared with when they spoke to an interviewer on the phone – suggesting the "shy Trump voter" notion could have some basis in truth.

Some polls did actually get it right

Mike Segar / Reuters

It's important to remember that some polls did actually get it right for all of these events. In the US, many correctly predicted a small lead for Clinton in the national popular vote – the total votes she received across the country. The scientists behind them “must be feeling sick” to hear accusations that the polls were entirely wrong, said Wells.

One major tracking poll from USC and the LA Times even consistently predicted that Trump would win the presidency, but it was ignored, claimed Boon. “We had a narrative in which the central thrust is that one person is going to win, and outlier polls which challenge that were often, I think, ignored or even ridiculed.”

He claims he was similarly “castigated” when his company, ICM, predicted a nine-point lead for the Tories before the 2015 UK general election – while most other polls at the time predicted a Labour win. “[Our poll] was called a complete rogue, as simple as that, and it was airbrushed out of history," he told BuzzFeed News. Statisticians call this effect “herding” – when pollsters ignore outlier results, creating an artificial consensus.

Boon believes this happened in the case of the Brexit vote, when he says the consensus was that all polls forecast a Remain win, ignoring “many polls [that] were predicting we would leave the European Union”.

“The outlier can be the most accurate, but that gets submerged into a swamp of polls," he said. "If they all suffer from the same methodological negative characteristics, obviously you don’t know about that in advance, and even after a lot of work you might not really understand afterwards.”

The price of "overclaiming"

Bria Webb / Reuters

Brexit taught pollsters that voter turnout is getting harder and harder to predict. Pollsters tend to ask people if they are planning to vote, but voters don't always turn words into actions.

“Normally it’s very accurate but obviously there is a great big pitfall if patterns of turnout change and suddenly lots of working-class people who don’t normally vote suddenly get enthused," said Wells.

Respondents "overclaiming" about how likely they were to vote was one downfall the British polls before the last general election. “What happened then was that many people said they would turn out and vote, and of course they didn’t, and they were all young working-class Labour supporters,” said Page. By being factored into the poll results but never turning up to vote, their "overclaiming" wrongly skewed the data to point to a Labour win.

So when it came to the Brexit vote, pollsters tried to prevent overclaiming by not only asking respondents if they would vote, but whether they had voted in previous elections. If they hadn't, they were discounted from the data.

But they were again thwarted by unexpected behaviour. “Lots of older working-class people who hate the European Union and hadn’t voted in the 2015 or 2010 elections suddenly think, Fuck it, I’m going to take back control, I'm going to vote. Turnout goes up to 72% and we’re wrong again," said Page.

The group that unexpectedly turned out in the EU referendum have a lot in common with the demographic that many think powered Trump's election, he said. “It seems like they got people who haven’t voted in several of the recent elections. It’s disproportionately older working-class people."

In both cases, there is not just one "group" behind a vote win. "As with Brexit, Trump’s support was from the Rust Belt: older, often white industrial areas like Sunderland and Hartlepool and Wales, for example, but also from a wider group of people angry at globalisation, and who feel marginalised and ignored by the elite.

"In Brexit, 40% of Londoners – in the most cosmopolitan city anywhere pretty much – still voted against the EU and globalisation and against the elite."

Pollsters in the US, as in Britain, missed the scale of this anger, he implied. “It shows that people are pissed off, they are alienated, they feel ignored, and somebody has managed to press their buttons effectively.”

Both changes in voting patterns are only easy to see in hindsight, Wells said: “Now, we will be looking for them. But the trouble is if people then revert to their previous behaviour in [future] elections. The unpredictability of it, and the volatility of it, is the challenge.”

The politics of shame

Ammar Awad / Reuters

A bigger problem is that "shy" voters in both the UK and US may not even want to take part in polls in the first place.

“If you’re disinclined to admit that you are a Trump supporter, maybe you refuse to even pick up the telephone or to join online panels in the first place – which are ‘unknown unknowns’, as someone famous once said,” claimed Boon.

As right-of-centre voters are usually the ones affected by "social desirability bias", their views are often not captured in a representative way in polling, he said – something at play in both recent UK experiences. “What elections around the world have there been where the right has not been underrepresented in pre-election polling?

"Polling misses around the world have almost exclusively been about the right wing, in whichever country, being underrepresented. There is a problem around the world of reaching people with right-of-centre views.”

He called for pollsters to develop methods to reach more right-wing people or to "impose more robust post-field work to adjust your data in the direction that you think that it is inaccurate".

“Trump is more proof than ever that if polling is going to be wrong, it will be in having too few right-wing voters in the sample,” he added.

The global polling crisis

Edgard Garrido / Reuters

All these events point to the struggle facing pollsters everywhere – it's increasingly hard to get a representative sample of the population amid our changing media habits, Wells said.

“All across the world, response rates to opinion polls are falling, fewer and fewer people have landlines, fewer and fewer people answer landlines to strangers and then are willing to take part. Equally, people who join internet panels, unless you put a lot of time and effort into getting a good one, can be a bit weird."

This may have scuppered samples for Trump and Clinton's contest and certainly affected Britain's last general election, Wells said: "It meant [panels] didn't have many people who didn’t have many educational qualifications, or didn't pay much attention to politics.

"Those sort of people behave differently. If the US pollsters have got the same problem, I can quite imagine that people who don’t have many educational qualifications and don't care about the Washington bubble are the sort of voters who would probably go for Donald Trump."

Because some voters can't even be reached, their views and behaviour are a black spot for researchers. Pollsters want to weight results to reflect these absent demographics, Wells argues, but people who won’t pick up the phone may be fundamentally different from those who do, in ways that researchers can't assess.

"If, for example, old people won’t pick up the phone, that’s not actually a problem because everyone weights by age and we get a good result. It’s if the people are different in some far more nebulous and subtle way that you haven't thought of.”

UK poll makers are trying out approaches like weighting for those who are interested in politics, or have common feelings about broad issues like immigration, to try to get a handle on who these "unknown unknowns" are. “Basically, because we messed up, we are trying to put this thing right," Wells said.

Better to be boring?

Mike Segar / Reuters

The legacy of the Brexit vote, Trump, and other statistical failures is that polls will be trusted less in future, Boon said.

"What all these things show is that you really need to keep considering if your methods are correct, keep looking for reasons that you might not have something covered. It’s difficult to do accurate polling," he admitted, saying he encourages people to regard polls with a healthy suspicion.

“Media needs news stories for its readers, you don’t want a dull story," said Page, but warned researchers shouldn't feel pressured into giving the media a predicted winner if it's too close to call.

"We’re looking at how we present things graphically, to show you how much noise there is in the data.”

He stressed that it will take lengthy research for anyone to confirm for sure why the polls were wrong about Trump. "Like survivors after a nuclear holocaust, we’ve just woken up and this is our immediate reaction. It was a nuclear bomb, but I can’t tell you if it was a five-ton bomb or megaton one, until I’ve done the analysis."