Is it the end for telephone polling?
- Published
Last week I went along to the first public meeting of the investigation into what went wrong with the polling before the general election.
It is the British Polling Council's attempt to find out why all the pollsters spotted that the Scottish National Party was going to do well and the Liberal Democrats were going to do quite badly, but failed to spot the size of the gap between Labour and the Conservatives and predicted a hung parliament.
Indeed, the way they all got pretty much the same wrong answer made it look as if they had been copying each others' homework.
It was an extraordinary get-together of top polling organisations to say they had messed up and suggest reasons why.
The star of the show was Martin Boon from ICM, who asked the unaskable question: is it still possible to get a representative sample over the telephone?
The way phone polling has been done for a long time is that the polling organisation will work its way down a list of random phone numbers asking questions until it has asked the right number of people for the poll.
The idea is that if you ask enough randomly selected people you will get a sample of responses that represents what the whole population thinks.
They now include 15% or 20% mobile numbers with the rest landlines.
But the problem, according to Mr Boon, is that in order to get responses from 2,000 people, they had been forced to call about 30,000 random numbers.
Too enthusiastic
When the response rate is that low, there is a real danger that there is a reason why people are responding, which is stopping the sample being representative.
Adam Drummond, from Opinium, suggested that the people who agreed to be interviewed were so politically enthusiastic that they would even vote in elections for the European Parliament.
The political engagement of respondents is important because pollsters have to find out not only which party will be supported by an individual, but also how likely they are to vote.
Mr Drummond said the polls had overstated how likely people were to vote, especially younger people who were more likely to vote Labour.
Gideon Skinner from Ipsos Mori said that general election turnout was usually 10% lower than raw polls suggested it would be, but that in this election it had been 16% lower.
This seems like a Groucho Marx problem - people answering the phone and agreeing to answer questions about their political opinions automatically means they are too politically engaged to be representative.
Online polling
This is a criticism that has been made in the past of online polling. Online polls are answered by a database of volunteers who have signed up to be on a panel and who the company knows a lot about. When the company is commissioned to do a poll it can be sure that the people it is asking have the same features as the whole population in - for example, the proportion of men and women, or the age profile or income distribution.
The online polls came out with much the same inaccuracies as the phone ones. Does people's willingness to be on a panel automatically make them unrepresentative?
If it does then the only type of polling available is face-to-face, which is the blue riband of polling, but is much more expensive than the other two methods.
No wonder Mr Boon said there were "genuine reasons to worry about the future of political polling".
It seems unlikely that the final report on 1 March 2016 will conclude that phone polling is unacceptable, but I would be surprised if the problem of overstating people's likelihood to vote is not identified as a big area for further investigation.