Just google it: The student project that changed the world

"Dad, what happens when you die?" "I don't know, son. Nobody knows for sure." "Why don't you ask Google?"

Of course, Google isn't clever enough to tell us whether there is life after death, but the word "google" does crop up in conversation more often than either "clever" or "death", according to researchers at the UK's University of Lancaster.

It took just two decades for Google to reach this cultural ubiquity, from its humble beginnings as a student project at Stanford University in California.

It is hard to remember just how bad search technology was before Google. In 1998, for example, if you typed "cars" into Lycos - then a leading search engine - you would get a results page filled with porn websites.

Why? Owners of porn websites inserted many mentions of popular search terms such as "cars" in tiny text or in white on a white background.

The Lycos algorithm saw many mentions of "cars", and concluded the page would be interesting to someone searching for "cars". In the Google era, this seems almost laughably simplistic.

But Google's founders Larry Page and Sergey Brin were not, initially, interested in designing a better way to search.

Sergey Brin (L) and Larry Page (R) were trying to map the credibility of academic papers when they devised Google

Their Stanford project had a more scholarly motivation.

In academia, how often a published paper is cited is a measure of its credibility, and if it is cited by papers that themselves are cited many times, that bestows even more credibility.

Mr Page and Mr Brin realised that if they could find a way to analyse all the links on the nascent world wide web, they could rank the credibility of each web page in any given subject.

To do this, they first had to download the entire internet.

This caused some consternation. It gobbled up nearly half of Stanford's bandwidth. Irate webmasters showered the university with complaints that Google's crawler was overloading their servers.

More from Tim Harford

Why the falling cost of light matters

How Ikea's Billy took over the world

How economics killed the antibiotic dream

How the invention of paper changed the world

But as Mr Page and Mr Brin refined their algorithm, it became clear they had discovered a vastly better way to search the web.

Porn websites with tiny text saying "cars cars cars" don't get many links from other websites that discuss cars. If you searched Google for "cars", its analysis would be likely to yield results about… cars.

Mr Page and Mr Brin quickly attracted investors, and Google went from student project to private company. It is now among the world's biggest, bringing in profits by the tens of billions.

But for the first few years, Mr Page and Mr Brin burned through money without knowing how or if they would make it back. They were not alone.

During the dotcom boom, shares in loss-making internet companies traded at absurd prices, in anticipation that they would eventually figure out viable business models.

Speed and transparency

Google found its model in 2001: pay-per-click advertising. Advertisers pay Google when someone clicks through to their website, having searched for specified terms. Google displays the highest-bidders' ads alongside its "organic" search results.

From an advertiser's perspective, the appeal is clear: you pay only when you reach people who have demonstrated an interest in your offering.

It is much more efficient than paying to advertise in a newspaper.

Newspapers have seen a significant decline in display advertising

Even if its readership matches your target demographic, inevitably most people who see your newspaper advert won't be interested in what you are selling.

No wonder newspaper advertising revenue has fallen off a cliff.

The media's scramble for new business models is one obvious economic impact of Google search.

But the invention of functional search technology has created value in many ways. A few years ago, McKinsey tried to list the most important.

One is timesaving. Studies suggest that googling is about three times as quick as finding information in a library, even discounting the time spent getting there.

Likewise, finding a business online is about three times faster than using a printed directory such as the Yellow Pages.

Traditional directories such as the Yellow Pages have struggled to compete with online search tools

McKinsey put the productivity gains of this into the hundreds of billions.

Another benefit is price transparency - economist jargon for being able to stand in a shop, take out your phone, google a product you're thinking of buying and seeing if it's available more cheaply elsewhere, then using that knowledge to haggle - annoying for the shop, helpful for the customer.

Then there are "long tail" effects. In physical shops, it makes no sense to display aisle after aisle of obscure products that will be bought only rarely - they focus on a limited range of bestsellers instead.

Natural monopoly?

But a decent search facility makes it easy to find a needle in the product haystack, and that has enabled the rise of online shops offering more variety.

Customers with specific desires are more likely to find exactly what they want, rather than settling for the nearest thing available in the local supermarket. And entrepreneurs can launch niche products, more confident they will find a market.

This all sounds like excellent news for consumers and businesses.

Find out more

50 Things That Made the Modern Economy highlights the inventions, ideas and innovations that have helped create the economic world.

It is broadcast on the BBC World Service. You can find more information about the programme's sources and listen online or subscribe to the programme podcast.

But there is a problem.

Google dominates the search market, handling close to 90% of searches worldwide. Many businesses rely on ranking highly in its organic search results.

And Google constantly tweaks the algorithm that decides them.

Google gives general advice about how to do well, but it is not transparent about how it ranks results - not least because that would give away the information necessary to game the system. We would be back to searching for cars and getting porn.

Google explains how its search works in principle but guards the details of its all-important algorithms

You don't have to look far online (thanks, Google) to find business owners and search strategy consultants gnashing their teeth over the company's power to make or break them.

If Google thinks you are employing tactics it considers unacceptable, it will downgrade you.

One blogger complains that Google is "judge, jury and executioner".

"You get penalised on suspicion of breaking the rules, [and] you don't even know what the rules are," they say.

Trying to figure out how to please Google's algorithm is rather like trying to appease an omnipotent, capricious and ultimately unknowable god.

You may say as long as Google's top results are useful to searchers, it's tough luck on those who rank lower - and if those results stop being useful, then some other pair of students at Stanford will spot the gap in the market and dream up a better way. Right?

Maybe - or maybe not. Search was a competitive business in the late 1990s. But now, it may be a natural monopoly - in other words, an industry that is extremely hard for a second entrant to succeed in.

The reason? Among the best ways to improve the usefulness of search results is to analyse which links were ultimately clicked by people who previously performed the same search, as well as what the user has searched for before.

Google has far more of that data than anyone else. That suggests the company may continue to shape our access to knowledge for generations to come.

Tim Harford writes the Financial Times's Undercover Economist column. 50 Things That Made the Modern Economy is broadcast on the BBC World Service. You can find more information about the programme's sources and listen online or subscribe to the programme podcast.