Coding wizards offered X Factor-style competition

Code helps power the modern world

Television may have created the impression that talent contests are only open to those who can sing, dance or persuade a dog to juggle but a new phenomenon is springing up online that offers a much more cerebral type of X Factor aimed at coders.

Outside of the world of reality TV, it is algorithms that are the real stars - powering a range of services from the financial markets to the websites we use every day.

And there are plenty of people willing to pay good money to make them work more efficiently.

Movie rental site Netflix recognised the importance of good code back in 2009 when it offered a $1m (£645,000) prize to anyone who could improve the algorithm that offered film recommendations based on previous rentals. The competition was a huge success and the improved algorithms are responsible for 60% of movies rented from the site.

Now Heritage Provider Network, a US healthcare provider, is offering $3m to a developer who can create code that accurately predicts and prevents unnecessary hospitalisations.

The competition runs until spring 2013 and organisers hope that the winning equation can cut spending, freeing up more money for vital disease prevention research.

"A winning solution will change healthcare delivery as we know it from an emphasis on caring for the individual after they get sick to a true health care system," the organisers said.

Worth billions

These contests helped inspire Kaggle, one of a growing number of websites set up to offer a place for coding wizards to show off their talents.

Founder Anthony Goldbloom spotted a gap in the market while he was working as an intern at the Economist newspaper.

He said that the business was providing a vital link between firms wanting their data crunched and the scientists able to do it.

"Machine learning is becoming huge but the answers it gives depends on what questions we ask it. There is a limited group of people able to do this. They are massively valued and have have amazing skills but they tend to be hidden away in Cambridge University or in a hedge fund," he said.

Competitions featured on the site range from the fun - working out who might win the next Eurovision song contest - through to the socially useful - finding better ways to treat HIV - to the highly complex - mapping dark matter.

Each contest runs for an average of three months and the public competitions typically offer cash prices of around $10,000.

The website also hosts private contests where data is hidden from all but those selected to take part and prize money can be much higher.

Because of the sensitive nature of much of the data, each competitor must sign a non-disclosure agreement.

Extremely valuable

Wall Street is now full of data scientists, mathematicians and physicists because algorithms are increasingly important to the financial markets.

One of the competitions aims to map dark matter

One of Kaggle's current competitions is looking for an algorithm designed to predict how the money markets will react to different trades.

So-called algo-trading has not been without controversy.

The Flash Crash on May 6 2010, which caused the Dow Jones index to drop by 10%, illustrated what could happen when the code behind the numbers ran out of control.

One super-fast trading algorithm sold 75,000 stocks with a value of £2.6bn in just 20 minutes, causing others to follow suit. Although the problem was sorted within minutes, no-one has ever got to the bottom of why the code started misbehaving.

But software that does what it is supposed to can be extremely valuable to companies.

Another of Kaggle's current competitions is looking at ways of predicting which models of cars are more likely to result in the owner making a claim for bodily injuries.

Such predictive algorithms are "worth hundreds of million of dollars to the insurance industry," said Mr Goldbloom.

Coffee house

Jason Tigg is an avid coder and has so far has taken part in four competitions.

"I wanted to see how well I would do against other people and it became addictive," he said.

"For me it's a hobby. Some people like Sudoku or chess, and some people like solving these problems."

As one of first physicists on Wall Street, he can testify to how important code is becoming to the financial markets.

"When I started working on Wall Street it was barrow-boy like with not many mathematicians. Now Wall Street is dominated by them," he said.

Algorithms make automated high-frequency trades possible that do not rely on floor brokers

He acknowledges that the complex code that they have created could have played a part in the current global financial crisis.

"Maybe they [the mathematicians] are at fault for making things so complicated and that's why we are in the mess we are now in," he said, only half-jokingly.

But he believes the potential benefits offered by good code outweigh the risks.

"I look forward to the day when cars drive themselves and the traffic is less congested," he said.

Topcoder offers a similar service to Kaggle and boasts a community of over 300,000 "digital professionals".

Making sense of data is just one of many services it claims to provide.

"Whether you're building an entire digital strategy, creating a web site, mobile app, internal application, or trying to make sense of your data — tapping into the Topcoder community will open up opportunities that simply weren't possible before," it boasts on its website.

It compares itself to the 18th Century coffee houses that allowed intellectuals to get together to spread ideas.

"Today, caffeine is still rampant and obviously more accessible than ever, but it's the other, the access to challenges, the ability to collaborate and iterate with thousands of participants via virtual communities that has of course been the phenomenal game-changer," it said in its blog.

Full-time job

Chris Adlard is marketing director for Coverity, a firm which fixes software bugs in complex code such as that used by the world's largest particle accelerator, the Large Hadron Collider. Such websites are a fascinating new phenomenon, he thinks.

"It is a very interesting idea - using web 2.0 and social collaboration to solve complex business and academic challenges through algorithmic analyses," he said.

"The business world today would simply not function without software. Software algorithms are now at the heart of business processes and products across every sector - in the cars we drive, the medical devices in our hospitals, the financial trading applications used across our markets, the stock systems in our supermarkets," he added.

As algorithms increase in importance it seems so will the number of similar competitions available to coding wizards.

Kaggle now has some 20,000 data scientists on its books and aims eventually to allow them to make their living from such competitions.

"We are looking to pay people. It could become a full-time income for people like Jason," Mr Goldbloom said.

While the X Factor contestants sing for their supper, the mathematicians will be busy coding for theirs.