Are markets 'efficient' or irrational?

Two economists with diametrically opposing interpretations of the way markets behave have been jointly awarded the 2013 Nobel memorial prize for economics. Why?

This story begins way back at the start of the 20th Century, in Paris. Louis Bachelier, aspiring physicist, lives a life filled with misfortune.

His parents die suddenly, leaving him holding his baby brother. He's not free to enter university until he's too old to be admitted. He takes night classes instead, producing one of Einstein's famous results before Einstein does - and he's all but laughed out of academia.

But in 1900, Bachelier publishes work that, long after his death, will be recognised as ground-breaking.

His remarkable idea is that the price of government bonds follow a random course - each new movement in the price will be unpredictable.

At first sight it's an odd idea. Why would the performance of a financial asset be random?

But Bachelier understood that if it was obvious that a share or a bond would be worth more next week, then the price should already have risen in anticipation of that. Predictable moves have already happened, because the market is efficient - and all that is left are unpredictable surprises.

At the University of Chicago's Graduate School of Business in the 1960s, student Eugene Fama knew Louis Bachelier's theory well, but wasn't satisfied.

"It dawned on me that that statement wasn't enough," he says. "There's no economical model behind that."

And so Fama went to work, and ended up demonstrating that stock markets are indeed efficient.

Eugene Fama and Robert Shiller

If you are an investor, one important consequence of an efficient market is that there's no point trying to make a killing. The market will always be one step ahead of you. Better to invest broadly rather than believing in some get-rich-quick scheme.

Fama himself has, in fact, found some ways to beat the market - or at least, investment strategies that would have beaten the market in the past. But he concludes nonetheless that for practical investment purposes, people might as well behave as though markets are efficient, and that nothing is therefore underpriced.

"Because if they are inefficient it's very difficult to tell how, when and where, and the evidence suggests that supposed experts aren't very good at it," he says.

"People spend lots of money trying to hire managers that can pick stocks, although the evidence says quite conclusively that that is probably impossible to do."

So the market is always right. Or is it? A decade later, a challenger to the efficient markets hypothesis arrives on the scene - Robert Shiller, now at Yale, then at MIT.

He wrote a paper making a statistical case that market volatility is too high for the efficient markets theory to be true. His idea was that markets have a tendency to overreact to news, or to react to non-news.

"This got me into a huge controversy then," he recalls. "I was wondering why do people get so emotional about this? I'm just saying something that's common sense and obvious - these markets aren't perfect."

And this argument doesn't stay in academic circles. It makes it into the heart of real world economics. In 1996 Robert Shiller is invited to lunch with the chairman of the US Federal Reserve, Alan Greenspan.

Greenspan: How do we know when irrational exuberance has unduly escalated asset values?

"I asked the table, 'When was the last time a Fed chairman had said that he thought that the stock market was overpriced?'"

The answer came that it was about 30 years ago.

"So I said to Greenspan, 'Maybe you ought to consider making a statement about the overpricing of the market?'"

Just a few days later the Federal Reserve Chairman made a speech that has become famous.

"How do we know when irrational exuberance has unduly escalated asset values which then become subject to unexpected and prolonged contractions as they have in Japan over the past decade?" he said.

Having prodded Alan Greenspan into action, Shiller went on to write a best-selling book called Irrational Exuberance, and he's famous for saying in the late 1990s that shares were hugely overvalued - and they did crash not long after - and for saying in 2005 that US house prices were overvalued. And they crashed too.

Was he just lucky? What were his reasons for pointing to a bubble?

Well, stock analysts commonly looked at how company profits, or earnings, compared to their share price. What Shiller and his colleague John Campbell had decided to do was take a longer-term view of this price-earnings ratio. They compared the share price to the average earnings for the past 10 years. This long-term price-earnings ratio showed a sharp peak in 1929, just before the great Wall Street crash. And the peak in the late 1990s was even bigger.

So why don't we see these irrational booms at the time?

"I think it's because a lot of people benefit by seeing the boom continue," Shiller says. "And there's a sense that no-one should shout 'Fire!' in a crowded theatre - everything's going great, right? Earnings are going up, prices are going up."

But Eugene Fama isn't convinced that it's quite so easy to spot trouble in financial markets.

"The notion that for example regulators should somehow go in and burst bubbles is to me a joke," he says.

"I don't think anybody really recognises bubbles - I mean I don't even like the word - apart from in 20/20 hindsight."

In Gene Fama's view, when technology stocks boomed in the late 1990s, and then crashed, that reflected people's views about the prospects of highly successful companies emerging from the melee. Not quite enough of them did, to sustain share prices at that level. But that didn't make the market irrational, it just made it wrong - with hindsight.

We shouldn't overlook Lars Peter Hansen. He was the third Nobel prize winner, for developing statistical tools that are now hugely influential in economics - especially in the study of financial markets.

The other co-prize-winner, Lars Peter Hansen

But we wouldn't be human if we weren't curious about the apparent contradiction in awarding shares of the Nobel prize to Eugene Fama for showing that the market is efficient, and Robert Shiller for showing that it isn't. Is the committee just hedging its bets?

Both will get a mention in the history books as having contributed to the future development of economic thought, in Shiller's view.

"It's a little bit like religion," he says. "I mean there are all these different sects and if you look at them in the whole it doesn't seem to make any sense - they contradict each other so fundamentally. But maybe there's some wisdom about living that comes out of all of them. And I think that the economics profession isn't as fragmented as it may seem."

But was it not a little irritating to share the Nobel memorial prize with someone who seems to contradict you?

"I think there is room for substantial disagreement and I think all points of view - all interpretations of the evidence - should get a full airing," Fama says.

"I mean, all of that makes the world a much more interesting place, and I was thrilled that Bob got it."

Follow @BBCNewsMagazine on Twitter and on Facebook