Headline numbers: Problems with a huge survey

A lot of the stories I am asked to look at involve organisations trying to make big claims based on surveys of tiny numbers of people with questionable methodology.

But occasionally I also have to take issue with the use of much bigger surveys, such as the country's biggest household one, the Labour Force Survey (LFS).

One of its main uses is in the production of the unemployment figures every month.

It used to be that the headline unemployment figure was the number of people claiming out-of-work benefits, but that meant the figures were vulnerable to changes in who was allowed to claim such benefits, so instead the country switched to the International Labour Organisation's definition of unemployment, which is people out of work and seeking a job, whether or not they are claiming benefits.

As I reported a few weeks ago, the claimant count has lost its national statistic designation, because the introduction of universal credit has made it more difficult to calculate.

But our headline unemployment figure comes from the LFS, in which the Office for National Statistics (ONS) speaks to 60,000 households once a quarter. It's a big survey, but it's still a survey. The ONS is very clear in its monthly releases about its limitations.

Two weeks ago we had the unemployment figures for the three months from March to May, which it turned out had risen by 15,000 compared with the previous three months.

But lower down in the release, readers were warned that the ONS was 95% confident that the actual figure was within plus or minus 80,000 of that mark, so between a rise of 95,000 and a fall of 65,000.

Which may make you question our top line, that unemployment had risen for the first time in two years.

Migration claims

Unemployment is not the only figure taken from the LFS, and there was some interesting analysis done by Migration Watch last week using the survey.

The group said its research showed that groups of migrants with weaker economic characteristics (based on level of employment, rates of pay and likelihood to claim benefits) outnumbered those with stronger economic characteristics by two to one.

The House of Commons Library has been looking at this analysis and questioning whether the LFS is a robust enough source to make such claims.

As an example, it considers the figure given for the gap in likelihood to claim benefits between UK residents born in the UK, and those born in Pakistan or Bangladesh, between the ages of 40 and 44.

It says that despite its huge sample size, the LFS is only likely to have interviewed about 27 people who fall into the category of 40 to 44 year-old benefit claimants born in Pakistan or Bangladesh.

The Library suggests: "a nuanced narrative might be more appropriate than broad statements about migrants' likelihood to claim".

UPDATE 11:35am: Migration Watch has been in touch to say that its research did not draw any conclusions from the 27 people who the House of Commons Library blog was talking about and that the conclusions were in fact based on 140 respondents who claimed housing benefit out of the total sample of 922 people in the LFS identified as being born in Pakistan and Bangladesh.