The boffins digging for nuggets of gold in big data

It's in there, somewhere: firms make big money from selling big data

If you're looking for love on an online dating site, did you know there's a particular way of filling in your personal profile that will maximise the chances of someone contacting you?

Or that if you provide computer helpdesk services, what most customers tend to care about is how quickly you send them a reply, not whether you actually solve their problems?

These are some of the nuggets of information discovered by information boffins trawling through vast oceans of data.

And there's a growing number of companies that now offer to collect huge amounts of data and analyse it as a service, with the promise of helping businesses find nuggets of information that give them a competitive advantage in their markets.

These companies draw information from the vast amount of data that's generated on the internet every day - anything from the search terms that people are using, to videos they are uploading, to messages they are posting on social media networks such as Facebook, LinkedIn and Twitter.

This data falls in to a category known as Big Data.

"For something to be Big Data there obviously has to be a large volume of it," explains Yvonne Genovese, an analyst at Gartner.

"But it also has to have variety: it can include pictures, video, and text, not just numbers you could enter in a spreadsheet.

"And there's also velocity - it's generated very rapidly."

Useful nuggets

Big Data has been a problem for businesses in the past because it's been expensive to store and because huge amounts of computing power are needed to analyse it to get anything useful from it.

But cheap storage and computing resources are now available in the cloud, so analysing Big Data is no longer such a financial challenge.

But one problem remains, says Ms Genovese: "The question is, how do you get useful nuggets of information out of Big Data?"

Bloomreach uses sophisticated algorithms to sift through data

The answer, she says, is by developing complex mathematical formulae or algorithms designed to analyse this Big Data to produce answers to specific business questions.

These algorithms need constant fine-tuning as new data are generated.

And it turns out that developing and fine-tuning them is a formidably difficult task - one that's the preserve of Big Data specialists.

"In the old days you had companies that provided mailing lists and other data because collecting them was hard, so they had value," says David Feinleib, managing director of technology consultancy The Big Data Group.

"Now Big Data is freely available on the internet, but developing the algorithms is hard, so there's value in combining this data with proprietary algorithms and offering that as a service."

One example of a company offering this type of Big Data service is California-based Bloomreach.

Put simply, the company analyses Big Data to understand what language consumers use to look for things they want to buy.

It then ensures that its customers' websites use the right words, and are structured in the right way, so that these consumers find what they are after.

The company grabs most of the data it needs from the internet and analyses it using storage and servers in the cloud, rented from Amazon.

'Printed jeans'

You might imagine that you could simply hire a search engine optimisation (SEO) consultant to ensure that your products can easily be found, but that's not nearly as effective as using algorithms, according to Mike Relich, chief information officer at fashion retailer Guess, a Bloomreach customer.

"In fashion, the lexicon used by shoppers in search terms changes very quickly," he explains.

"It may be 'animal prints' one day, 'printed jeans' the next."

That means any changes an SEO consultant makes are probably out of date before they are implemented manually on your website, he adds.

Finding the right information from Big Data means companies can strike gold and earn big profits

But because Bloomreach is able to analyse Big Data in real time - as soon as it is collected - Guess's website can be updated automatically according to its findings at any given moment.

"We certainly don't have the expertise to analyse the data that's available in this way in-house," Mr Relich says.

The ability to analyse vast amounts of amounts of Big Data from social media in real time is particularly suited to answering marketing questions, says Bloomreach's Joelle Kaufman.

"It's where you need to take decisions quickly and repeatedly, informed by massive amounts of data," she says.

This explains why many Big Data analysis services, offered by Bloomreach and other companies such as Bazaarvoice and Salesforce Radian6, are marketing-related.

One thing that's important to consider from a business point of view is the extent to which companies that offer Big Data analysis as a service can provide their customers with a competitive advantage.

After all, Big Data from the internet is now a commodity available to all.

And if competitors can use the same services, which analyse the same Big Data, then that would seem to imply that any useful nuggets of information that arise from it are also a commodity.

But Gartner's Yvonne Genovese maintains that that is absolutely not the case.

That's because there is such a large volume of Big Data, and new data is generated on the internet all the time.

"Don't forget that every company is looking for something different," she says.

"The Big Data pool is just so rich that thousands of companies could access it and find something different that benefits their business each time."