Will Big Data lead to Big Brother?

Many countries are in the throes of a debate about the amount of surveillance a government should be allowed to carry out on its own people. But in other countries, where there are few, if any, checks on the state's powers, a potential dictatorship of data is already on the horizon.

The grey, drab former headquarters of the Stasi - East Germany's Security Service - is famous for its miles of paper files. Those files recorded the detailed information kept on the citizens of the former Communist state, drawn from a wealth of human informers and bugging devices.

Parts of the former office complex are now a museum open to the public, but in one corridor normally closed to the public there is a jumble of dated-looking equipment - a primitive computer looking more like a spin-dryer for clothes and old magnetic discs the size of a football, which held a fraction of what you can now fit on a USB stick.

This is all that remains of the Stasi's dreams of what computers could do for them. "I think they realised early on that without using technology their ambition of total surveillance could not be achieved, despite all the people they employed," says Stephane Konopatzky - a former dissident, who now tries to untangle these wires of the Stasi's past.

Gordon Corera with a computer used by the Stasi

The Stasi dreamed of computers combining and cross-referencing all the data they had collected to tease out new information they would otherwise have missed. Computers were faster than humans, more precise "and the mass of data you could deal with was larger", Konopatzky says. In the 1970s and 80s, however, the technology was just not there, and with the fall of the Berlin Wall in 1989 the Stasi was no more.

But as we enter a world in which we all produce so much more data and in which computers are able to do more with it - often described as the world of "big data" - what might the Stasis of today or tomorrow be able to do with technology?

Find out more

Listen to Gordon Corera's report The Dictatorship of Data on BBC Radio 4 on Tuesday 17 November at 20:00 GMT. Catch up via the iPlayer after broadcast.

A failed coup in the Soviet Union in August 1991 marked the final collapse of communism in Russia. That same month something equally momentous happened - although few understood its significance at the time - the World Wide Web was launched. The web seemed to herald a new era in which the free flow of information would sweep away totalitarianism. But in some places the dreams of the Stasi lived on.

"We asked people inside the KGB about technical co-operation between the Stasi and the KGB. All of them told me they were inspired by the Stasi," says Andrei Soldatov, co-author of The Red Web, a book that examines the history of Russia's efforts to control the internet.

In 1991, the Russian state was only able to wiretap 300 phone lines simultaneously in Moscow, which was nothing compared with what the Stasi had been able to do. But after initially feeling overwhelmed in the digital world, the Russian authorities have increasingly looked for ways of using technology to their advantage.

This has included pioneering techniques in voice sampling and recognition, in which Russia is a world leader, as well as biometric and photographic databases.

"I was told many times by officers in the security services that the idea to collect all fingerprints, iris scans and voice recognition from all Russian citizens is very popular within the security service," says Irina Borogan, Soldatov's co-author. One initiative described by the authors involves cameras placed at the exit of Moscow metro stations taking close-up photographs of everyone passing through.

Will the movements of all Russian citizens be recorded?

These days people are also voluntarily posting huge amounts of data about themselves on the internet.

"Social media provides a very easy way to monitor these societies," says Taha Yasseri, an Iranian computer scientist at the Oxford Internet Institute. "The transparency and easy use of social media has made it a very good tool for social activists. But this makes it a very good opportunity for authoritarian states to monitor, and eventually even to predict behaviour."

Pro-democracy activists in Hong Kong last year were using social media and messaging platforms to organise. But when some received a message telling them to download a new app to help them, it is believed they actually downloaded a piece of spyware which was able to send the content of their messages and even their physical location back to whoever was behind the app in mainland China.

A pro-democracy supporter takes a photo during the 2014 Hong Kong demonstration

In the past, a state would need to develop its own technology to carry out surveillance. But now much more of the technology has been commercialised.

Companies look for vulnerabilities in computer systems and sell them to states to exploit. Additionally, some companies sell the ability to carry out remote surveillance. The Hacking Team might sound like a group of rogue teenagers but in fact it is a company based in Italy that sells its services to law enforcement and governments around the world.

Eric Rabe, its chief marketing officer says the company supplies its products to about 50 countries. Typically a piece of software will be surreptitiously placed on one of their target's computers or phones, either by getting hold of the device or remotely, so they can be monitored.

"We're providing tools for law enforcement to use in the pursuit of keeping all of us safe," is his answer to accusations levelled against the company.

Hacking Team's spyware enables a target's computer or phone to be monitored

But what is it like being on the receiving end of surveillance technology provided by Western companies? To find out, I visit Tadesse Kersmo, a member of a movement campaigning to remove the Ethiopian government. Kersmo, who was given asylum in the UK after being imprisoned in Ethiopia, says his computer was bugged by FinFisher - software developed by a British-German company similar to that supplied by Hacking Team.

"It was a shock," he tells me. It was "as if there was CCTV in my home. You can imagine what my wife felt. People become traumatised, afraid to communicate."

We tried to contact Gamma Group - which developed the FinFisher spying software - but there was no response. The Ethiopian Embassy in London said it categorically denied any government involvement in surveillance against opposition politicians based inside or outside Ethiopia, suggesting it had no time for what it called such "futile pursuits".

Eric Rabe says Hacking Team once provided services to the Ethiopian government but later suspended its co-operation. He says that because there was no international regulation, the company has had to make its own decisions about who to sell to, and add clauses restricting certain uses.

Moves are now afoot to place the trade in surveillance technology on a similar footing to the trade in weapons. But on the assumption that export controls will never entirely stop it, some people are focusing on ways of training people to protect themselves.

Online behaviour can have a predictable and repetitive pattern

"We see people making simple mistakes," explains Stephanie Hankey, co-founder of a Berlin-based group called Tactical Tech, which trains activists, journalists and civil society campaigners to become more security aware.

"In Syria in the early days of the conflict, people would be pulled in, open their Facebook account and expose everyone they were talking to."

She also advises people to be aware that even if the content of their conversations may be encrypted, the metadata about the conversation can reveal much about connections and patterns of activity, especially when different elements of the digital trail we leave behind are cross-referenced and cross-mapped - how we move around a city, pay taxes, cross borders and use our credit cards, as well how we communicate.

"If we piece all these things together, this tells everybody about my behaviour," says Hankey. "If you have aggregate information about everybody, you can see who's different. This enables you to see something abnormal, an outlier."

The trend in computing is to use this data to find patterns which are in turn predictive.

"Even though we think we are very creative in our behaviour, the data suggests that our behaviour has a very predictable and repetitive pattern," argues Taha Yasseri. "Combining the data we produce with sophisticated mathematical models and computer algorithms - all these three could eventually make governments and companies and any kind of organisation able to predict our behaviour."

Others in this field are hopeful that new technology - particularly encryption and anonymising tools - will ultimately protect activists and liberty more than it will undermine it.

Surveillance software can record audio from a phone's microphone

But at London's IP Expo 2015, where all the talk is about the huge and mostly beneficial power of Big Data, veteran cyber security expert Mikko Hypponen, believes we are at the beginning of an enormous social change that carries with it real danger.

"We are the first generation that can be tracked from birth to our deathbeds, where we are, what we do, who we communicate with, what are our interests. It's easily trackable and saveable for decades. It feels like we're in a massive experiment done on mankind. Only much later will we realise what it means when all of our thoughts and movements not only can be tracked but are being tracked."

So will Big Data lead to Big Brother? Not necessarily - and in some countries we may have the chance to decide. But there are parts of the world where a dictatorship of data - of the type the Stasi could only dream of - may be taking shape.

Listen to Gordon Corera's report The Dictatorship of Data on BBC Radio 4 on Tuesday 17 November at 20:00 GMT. Catch up via the iPlayer after broadcast.

Subscribe to the BBC News Magazine's email newsletter to get articles sent to your inbox.