Is surge pricing a fair way to manage demand?

In the 1950s, the New York subway faced a problem that will be familiar to users of public transport all over the world.

At peak times, it was overcrowded; at other times, the trains were empty.

The mayor commissioned a report, which concluded the problem was subway riders paid a flat fare. No matter where you boarded, how far you travelled, or when you made your trip, it would cost you 10 cents.

Might there be some more sophisticated approach? Perhaps so. The report's foreword singled out a proposal from one of the 17 authors, economist William Vickrey.

"The abandonment of the flat-rate fare in favour of a fare structure which takes into account the length and location of the ride and the hour of the day is obviously a sensible step provided the mechanical problems involved can be solved," he said.

Vickrey's basic idea was simple: when the trains were busy, charge more. When they were quiet, charge less.

The peaks in demand would become less spiky. The subway would be more comfortable and reliable, could carry more people without having to build new lines, and could raise more money, all at once. A great idea.

But how to charge all these different prices? Not with an army of ticket clerks and inspectors; that would take too much time and money, so an automated solution must be found.

What was needed was a coin-operated turnstile that could charge different rates for different journeys at different times. But this was not easy to deliver in 1952.

50 Things That Made the Modern Economy highlights the inventions, ideas and innovations that helped create the economic world.

It is broadcast on the BBC World Service. You can find more information about the programme's sources and listen to all the episodes online or subscribe to the programme podcast.

To give a sense of the scale of the challenge for William Vickrey, consider a dilemma faced by the Coca-Cola company. A Coke had cost a nickel - five cents - for decades.

Coca-Cola would have liked to increase the price by a cent or two but it couldn't.

Why? Its 400,000 vending machines took only nickels and redesigning them to take two different denominations of coin would be a logistical nightmare.

A standard Coca-Cola cost five cents from 1886 until 1959

In 1953, Coca-Cola tried instead to persuade President Eisenhower - in all seriousness - to introduce a 7.5 cent coin. That attempt failed - and Coca-Cola's price remained five cents until 1959.

But Vickrey wasn't daunted and described a contraption that would solve the problem.

"Passengers put a quarter in the entrance turnstile, get a metal check with notches indicating the zone of origin to be inserted in an exit turnstile, which would, through electro-mechanical relays, deliver an appropriate number of nickels according to the origin and time of day," he said.

It sounds clever, so you may be wondering why you haven't heard of it.

One clue comes from the title of the speech in which Vickrey gave that description, My Innovative Failures In Economics. He began that speech by saying: "You are looking at an economist who has repeatedly failed in achieving his objective."

William Vickrey spent his career considering practical economic solutions to urban problems

The variable-price electromechanical Vickrey Turnstile was never built.

So why are you reading an article about a non-existent invention? It's because the idea itself was so important, even if it was initially too complicated to be put into practice.

Vickrey's fellow economists often said he was just too far ahead of his time. He eventually won a Nobel Memorial Prize for his work, in 1996, just three days before his death.

Vickrey was proposing what is often called "peak load pricing" by economists, and "dynamic pricing" by management consultants.

In its simplest form, it's an old idea. "Early bird specials" - offering a cheap deal to restaurant diners at quiet times - date back to the 1920s. It's an easy sell to customers and requires no electromechanical wizardry.

More things that made the modern economy:

• How a razor revolutionised the way we pay for stuff

• How the humble S-bend made modern toilets possible

• The horrific consequences of rubber's toxic past

• How do people learn to cook a poisonous plant safely?

But the idea has appeal in far more complex settings.

Whether you're running a subway system or an airline, trying to fill a concert hall or balance an electricity grid, it can be very costly to add extra capacity just to meet a short-term peak in demand - and it is wasteful to carry unused capacity around at other times. Varying prices make sense.

US Airlines was an early pioneer of the idea, after deregulation meant the company had to compete fiercely in the late 1970s. By 1984, the Wall Street Journal reported Delta Air Lines alone employed 147 staff to incessantly tweak prices.

• Thomas Cook: Why did prices rise for new flights?

• Why your bananas could soon cost more in the afternoon

• Timing is everything: Working out when to buy online

Peak-load pricing no longer requires an army of pricing specialists. A company such as Uber can effortlessly match supply and demand with an algorithm. Uber's "surge pricing" promises to end that painful three-hour wait for a taxi on New Year's Eve; there's always a price for a car right now.

The Uber driver app on the windscreen of a car, showing surge pricing is in effect in Washington DC

But consumer acceptance may be more of a problem. "You're almost at their mercy because you don't want to wait longer for a cab," one disgruntled customer told the Houston Chronicle after being charged $247.50 (£203) for a 21km (13-mile) ride in Houston, Texas. Although, it's worth pointing out he chose to pay so much because he couldn't bear to wait for a cheaper journey.

Consumers can feel exploited by some forms of dynamic pricing - especially when, as with Uber, prices can double or halve in a matter of minutes.

A 1986 study by behavioural scientists Daniel Kahneman, Jack Knetsch and Richard Thaler indicated people found price surges infuriating.

Having once despaired over the lack of a 7.5 cent coin, Coca-Cola unsuccessfully attempted another technological solution in 1999, when it flirted with a vending machine that on sweltering days would raise the price of an ice-cold Coke. It was so unpopular, the company had to backtrack.

But peak-load pricing is likely to play an increasing role in the economy of the future.

Consider a smart electricity grid fed by intermittent power sources such as wind and solar power. When a cloud covers the Sun, your laptop might decide to stop charging, your freezer might switch itself off for a minute, or your electric car might even start pumping energy into the grid rather than sucking it out.

But all that would require those devices to respond to second-by-second price changes.

Drivers who want to use part of Interstate 66 in Arlington, Virginia, during rush hour are charged a variable price according to traffic levels

William Vickrey was also fascinated by the possibilities offered by congestion pricing on roads, designed - just as his turnstile was - to smooth out demand and ensure limited capacity was used well.

That's now becoming a reality. Drivers in Arlington, Virginia, near Washington DC, can switch into a free-flowing lane on Interstate 66 during rush hour if they're willing to pay the variable charge, which can be as much as $40 (£33) for 15km when traffic is particularly bad.

Vickrey had tried to show this idea could work, in the mid-1960s: he built a prototype, using a simple computer and a radio-transmitter to tally every time he used his own driveway.

But as with the turnstile, sometimes good ideas just need to wait for the technology to catch up.

The author writes the Financial Times's Undercover Economist column. 50 Things That Made the Modern Economy is broadcast on the BBC World Service. You can find more information about the programme's sources and listen to all the episodes online or subscribe to the programme podcast.