Why is David Miliband walking out on Labour's top team?

David Miliband has taken four days to announce his intentions

He's popular with the Labour party. He's served in the very front rank of politics, striding the world stage - pictured with Obama, feted by Hillary Clinton. He's still only 45 - relatively young for a politician. And he says he loves his brother.

So, why is David Miliband walking out on Labour's top team just at the moment that they require youth and experience - two qualities which he can display - in order to rebuild after the general election defeat?

Although he was reluctant to say so, during the leadership contest he finally agreed he would serve under his brother Ed if he lost. But he didn't specify in what capacity.

Many of his supporters were emotional in the wake of his narrow defeat and one of the most prominent among them told me that agreeing to work for his brother in theory was fine, but in practice - as his loss sunk in - he felt differently.

The last place David Miliband wanted or needed to be was in the hothouse atmosphere of Labour's conference, under the camera lights, his every move reported.

He took some time out on Sunday afternoon but received phone calls and text messages from his brother.

Ed's supporters told me at that time they were very confident he would take a shadow cabinet position though no formal offer of shadow chancellor had been put on the table.

David was, of course, also having discussions with other members of his family - including his wife, Louise. He had been talked of as a future leader since 2007, when Tony Blair stood down, putting a degree of constant pressure on his life outside of politics.

In the end, by not resigning along with his friend James Purnell in 2009 he had bolstered Gordon Brown.

Now he was being asked to hang on in there, with all those pressures that front-rank politics brings, to bolster another leader he thought was less talented than himself.

Substance

Throughout the leadership contest, he had said if he believed his brother would be a better leader he would have been running his campaign instead.

Nonetheless, just because he had lost the leadership battle, initially he wouldn't rule out standing for the shadow cabinet.

But his speech on Monday, and its aftermath - while not finally making up his mind - certainly created a more downbeat assessment that it would be possible to work with Ed at Labour's top table.

When he simply tried to leave the building after the speech, and told the pack of reporters and camera crews who were pursuing him he was "on the way out", the BBC hit a nerve by asking, "Are you really on the way out?"

But it wasn't this ribbing that worried him, but a question of substance.

Following the speech, he holed up in a backroom of the Livebait restaurant with some of those who had toiled the hardest on his behalf during the leadership bid.

Although it was an emotional meeting, David was said to be hard-headed in his view that he would do what was best for both Ed and for the Labour party - and that wasn't necessarily to take a front-bench job.

A crucial incident took place that day which made him doubt if it could ever work to have not just Blair/Brown rivals at the top of the party - an era they had witnessed at close range as advisers in Number 10 and 11 Downing Street - but two brothers.

Ed Miliband will have a new shadow cabinet

Following his speech to conference he expected to be asked about his brother and about his own career intentions.

Brotherly scrutiny

But the question which threw him was on the US. He was asked about his differences with Ed - who has criticised the last Labour government's closeness to the US on foreign policy.

David regards the substantive - rather than rhetorical - differences as slight. But this all but convinced him that every nuance between him and his brother would be pored over by the media at the expense of scrutinising potentially far more divisive differences at the heart of the coalition government.

But there were two other considerations. Some differences are real, not rhetorical - especially on the speed at which the deficit should come down.

As shadow chancellor that could be tricky. Continuing as shadow foreign secretary would be tricky too, given the differences over Iraq.

But the other problem was not simply about disagreements, but performance.

David's speech had gone so well that London's Evening Standard newspaper was suggesting he could overshadow his brother.

And there was a major reason why this was seen as problematic: legitimacy.

The real leader

Under Labour's three-way electoral college, David had won the most support from party members and MPs. Ed had won only among trade unions and affiliated societies, such as think tanks and interest groups.

If David did well in the House of Commons, then Conservative MPs might bait his brother by calling for Labour's real leader to stand up. That is like giving an already raw nerve sciatica.

But still a section of David's supporters were urging him to stay on - some in the strongest terms, with one suggesting it would be "a dereliction of duty" to stand aside.

Meanwhile, they were working overtime behind the scenes to ensure that if he did go, at least some of his supporters would get on to the shadow cabinet. They got that assurance on Tuesday - along with confirmation that the shadow chief whip Nick Brown, seen as close to Gordon Brown and an opponent of David Miliband - would be removed. But even that didn't persuade David to stay on the frontbench.

Those close to him urged him to leave conference and get some perspective - making the right decision would be better than a rushed decision.

But his opponents in Manchester were now heaping derision upon him, suggesting his inability to declare his intentions proved that he would have been a poor leader all along.

Against this backdrop, frustration clearly got the better of him on Tuesday when he witnessed deputy leader Harriet Harman applaud his brother's denunciation, in his leader's speech, of the Iraq war which he had supported.

So, he took the advice to go home to decide whether to close the door on front bench politics. At least for now.