Big data retailing offers tailor-made shopping for all

Retailers know so much about us from big data analytics that they can tailor their services to suit our needs and preferences

Imagine everything being known about you the minute you walk in to a department store - your name, measurements, purchase history in-store and online, even your views on life, the universe and everything.

Would that make you feel like a celebrity or a victim of intrusive surveillance?

How you answer could affect the future of retailing, which is undergoing a radical transformation driven by real-time big data analytics.

Barbecues and beer

Ever since Tesco-owned Dunnhumby helped the superstore set up its Clubcard loyalty card scheme in 1994, we've grown used to retailers collecting large amounts of information about us.

But now, historic sales data is being supplemented by a whole range of new data sets, such as weather - crucial for knowing how many barbecues, beers, or brollies to stock up on, for example - social media content, and location from mobile devices.

And the ability to analyse all this data in real-time is giving retailers and their staff an unprecedented opportunity to offer us tailor-made services, online and offline.

Weather data is proving increasingly important for retailers wanting to know how many barbecues, brollies or beers to stock up on

"If you know what your customers are buying and what you have in stock you can provide the right special offers to suit them, but do this in real-time," says Klaus Boeckle, of software company SAP, whose big data analytics platform, Hana, is used by the likes of eBay and B&Q.

Made-to-measure

Shop assistants, interrogating such a big data service on a portable device, will call up our profiles and know from our recent social media posts that we're planning a holiday, for example, or looking for a new party dress.

We'll then be encouraged to buy relevant products - stuff they know we'll be interested in and the kind of stuff that we, or our friends, have bought before.

Apple's iBeacon technology - in-store Bluetooth location trackers designed to interact with smartphones - will soon enable retailers and app publishers to identify us individually the moment we enter a shop, says Owen Geddes of Appflare, a company specialising in deploying and managing the coin-sized gizmos.

Relevant special offers will then be pinged to our smartphones, and could change depending on where we are in the store.

Lush staff can change the layout of their stores in response to real-time sales data

"But the customer will always be asked for permission first," says Mr Geddes, concerned to head off criticism about consumers' privacy being compromised.

More data encourages better service, says Scott Silverthorn, head of data services for cosmetics retailer Lush.

The company has big data analytics available to its staff on the shop floor as well as in its warehouses, so that they have real-time sales statistics at their fingertips.

"Not only has this helped to tap in to the ambitious spirit of staff - competing over which store can do best in terms of sales and performance - but it also gives them information to improve the customer experience."

For example, if staff notice a particular bath bomb is selling well with a certain shampoo, they can change the store layout so the items are closer together, he says.

'Bigger is better'

But it is online where the big data trend towards personalisation has been most obvious.

Amazon, which has approaching 240 million customers worldwide and annual revenues of nearly $75bn (£46bn), can trace its success back to its ability to analyse customer data and adapt its services accordingly, argues David Selinger, chief executive of San Francisco-based content personalisation specialist, RichRelevance.

"In 2004, Amazon had better data capabilities than most retailers do today," he says.

Most retailer websites can tailor their website and email content to make it relevant to you based on your preferences and previous purchases

Werner Vogels, Amazon's chief technology officer, told the BBC: "You can never have too much data - bigger is definitely better. The more data you can collect the finer-grained the results can be."

The rise of cloud computing and real-time data processing is enabling retailers to target offers at their customers far more accurately, he argues.

"For example, on a particularly cold winter's day in your town a retailer will be able to recommend a coat from a fashion collection you have purchased before. When you then add other data sources, like voice and video to this, the possibilities get very interesting."

Amazon's purchase recommendation engine, which suggests other products shoppers may be interested in based on their previous buying behaviour and ratings, was "not always perfect", Mr Vogels admits. But thanks to "machine learning" - automated self-improvement - it is getting better, he says.

"You may be looking for a kettle but we will recommend the kettle that matches best the other things you've already bought for your kitchen."

Fight back

Traditional retailers are fighting back against the Amazon onslaught, wielding big data weapons of their own, argues Mr Selinger.

His company, RichRelevance, whose retail clients include Marks and Spencer, Boots, John Lewis, Argos, Dixons and Ann Summers, specialises in taking the masses of data that retailers collect on their customers and using it to personalise their shopping experiences.

Retailers know your preference for certain brands and will target their offers and content accordingly

Its software runs all this data through the open source framework, Apache Hadoop, then applies 125 different algorithms that try to predict what products the customer is most likely to buy at that exact moment, based on their previous and current behaviour.

This is all done in 20 milliseconds, he says.

Each of the algorithms is scored on the accuracy and influencing ability of its predictions, and these scores affect what images and offers are presented to customers the next time they visit the retailer's website or another site, such as Pinterest.

Mr Selinger calls this process "ensemble learning".

"By helping consumers find products that are most relevant to them we drive up sales 3% to 10%," he claims.

And it's not just website content that can be tailored to customers' preferences and buying behaviour, says David Brussin, chief executive of e-commerce personalisation company Monetate, whose system is used by 400 brands worldwide.

"Even the content of marketing emails can be changed right up to the point we click on them," he says.

It seems the brave new world of personalised shopping may be with us whether we like it or not.