Policy development and the power of randomness

Scientific rigour has helped inform policy on malaria prevention. Can politicians deploy it more widely?

"Resign!" is the refrain when MPs detect the merest hint that an opponent's policy is proving to be less effective than hoped.

But what if failed policies could be identified more swiftly, and nipped in the bud before it came to that?

Cabinet Office minister Francis Maude tentatively urged the Commons on Tuesday to adopt a more entrepreneurial approach to policy development.

"Not everything new that gets tried will work," he ventured, "but the best organisations learn at least as much from things that are tried and do not work as they do from things that are tried and do."

This is the premise behind <link> <caption>a new report</caption> <url href="http://www.cabinetoffice.gov.uk/resource-library/test-learn-adapt-developing-public-policy-randomised-controlled-trials" platform="highweb"/> </link> produced by the government's Behavioural Insights Team, or "nudge unit" as they are sometimes styled, which would - if widely adopted in Whitehall - result in a radical shift in how polices are evaluated.

Random power

The report highlights some shocking cases of hubris.

Steroid injections were a common treatment for people with severe head injuries for decades, but in 2005 it was revealed that they were making patients more likely to die.

A US programme that tried to frighten delinquents away from a life in crime by introducing them to hardened criminals was initially thought to be so successful that it was exported to several countries, including the UK; it later transpired that the programme was actually increasing crime rates.

In both instances, the truth was unearthed thanks to the power of randomised controlled trials.

Taking the first example, this meant comparing the survival rates of patients in similar predicaments who had been divided up randomly into two groups: one received the steroid injections; the other did not.

'Reckless'

"The study was controversial, and many opposed it, because they thought they already knew that steroids were effective," the report notes, before demonstrating that the received wisdom had been lethal.

Analysis of the young-offender programme was hampered by not comparing like with like.

Offending rates before and after the programme suggested that the intervention had worked.

But a randomised controlled trial showed that offending rates fell further in a group not exposed to the costly intervention than one that was, the report continues, confirming the folly of the initiative.

The number of randomised controlled trials has skyrocketed in the field of medicine since the 1960s, the report-authors point out, and a failure to use the technique is "regarded as bizarre, or even reckless".

It is also now common practice among web designers as a simple way of establishing whether the new layout improves a user's experience, thereby boosting sales.

A number of intriguing examples in the report show how the practice has been used to deliver concrete political benefits in recent years.

'Very exposed'

In one trial, a large number of people who had failed to pay their court fines were randomly divided into two groups. People in one group were sent text messages reminding them to cough up; the others were not.

The text-recipients proved to be much more likely to pay, and the cost of sending texts messages was dwarfed by the savings in bailiffs' fees.

International development practitioners are increasingly deploying randomised controlled trials to learn how to tackle low crop yields, encourage the use of mosquito nets, or foster entrepreneurialism.

So why are the majority of policy-makers so stubbornly sceptical?

"We often have no idea whether the things we do in government actually work or not, and achieve their stated goals," <link> <caption>bemoans journalist Ben Goldacre</caption> <url href="http://www.badscience.net/2012/06/heres-a-cabinet-office-paper-i-co-authored-about-randomised-trials-of-government-policies/" platform="highweb"/> </link> , a co-author of the report.

"This is a disaster."

He calls on policy-makers to "have a little humility, and accept that they don't necessarily know if their great new idea really will achieve its stated objectives".

Mr Goldacre may have found a new champion in Mr Maude, who observes of Whitehall: "No one's career ever seems to suffer if they continue to preside over an inefficient status quo, but if people try something new that does not work, they can feel very exposed."

Whether the force of the arguments wielded by the minister and his allies is enough to change Britain's raucous political culture, in which raw emotion tends to trump cool analysis, is another matter.