He wasn't supposed to be the joke candidate, exactly. Call him the courtesy candidate. Jeremy Corbyn's name was put on to the Labour leadership ballot not because anyone thought he would win, but because some well-meaning MPs thought it was important that at least someone from the traditional left had a voice.

Now the unthinkable has happened. Not only is Corbyn proving to be more popular than Liz Kendall, who was always too far to the right, he's also out in front of Andy Burnham and Yvette Cooper. These two are indistinguishable in their policies and have almost equally unexciting personalities. They are white bread, safe pairs of hands. They use sanitized media-speak to say things that'll keep them out of trouble, defined as negative headlines in the Daily Mail. They come without the awkwardness that scuppered Ed Miliband, but that's about it.

Corbyn himself is far from charismatic. He's cantankerous, unclubbable, dour and humourless. But it's that rock-solid independence of mind and indifference to the opinions of others that has helped put him, according to the the latest YouGov poll, no less than 17 points clear of his nearest challenger.

The panic among the Labour commentariat is frankly amusing, as much as anything. John McTernan, a former adviser to Tony Blair, has said: "The moronic MPs who nominated Jeremy Corbyn to 'have a debate' need their heads felt. They should be ashamed of themselves. They're morons." Tony Blair's former press secretary Alistair Campbell tweeted:

Blair himself said that the prospect of Corbyn winning was "like going back to Star Trek or something. Back to the old days." He added: "People who say their heart is with Corbyn, get a transplant."

So what's going on? Why is Labour ignoring the advice of its most electorally successful leader ever and on the verge of a lurch to the left which, on all the evidence, could see it consigned to opposition for decades?

Among the more astute comments on the Corbyn surge is one by Tristram Hunt, himself touted as a leadership candidate at one stage, who compared it to the outpouring of support for anti-austerity parties such as Syriza in Greece and Podemos in Spain. Hunt told Radio 4's Today programme: "The danger is that the Labour party, one of the great governing parties of the 20th and early 21st century that did enormously important things for Britain and Britain in the world, would be on a trajectory to becoming a pressure group – would not that have broad reach into all parts of the United Kingdom."

Now, let's be clear. A Corbyn-led government would not see the streets running red with the blood of plutocrats. His economic plan involves raising taxes on rich people and businesses, reducing corporate tax relief and using the money to support infrastructure projects. He also wants to crack down on tax avoidance (don't we all?) and reverse cuts to HMRC staff to help them collect more taxes.

The economics of this are debatable, shall we say. Raising taxes on the rich in order to increase benefits for the poor and taxing businesses in order to fund state spending is all classic big-state stuff. He is to the left of Ed Miliband on all this, and Miliband didn't convince the country to vote for him so there's no reason to think that Corbyn would either. However, Corbyn is also associated with various other causes that are polling-booth poison. He wants dialogue between Britain and Argentina on the Falklands. He wants to re-nationalise the railways. He supports the Palestine Solidarity Campaign. He supports a united Ireland and invited Gerry Adams and other Sinn Fein members to Parliament in 1984 just after the Brighton bombing. He's referred to Hamas and Hezbollah as "friends". In short he's given away so many hostages to fortune that he can barely field an army. Oh, and he's also anti-Trident, anti-military intervention and generally pro-peace.

There's nothing indefensible about any of these positions, individually. But you can see Labour's problem. Faced with a Tory machine that is the political equivalent of a Great White shark, it's going to need a much, much bigger boat.

So why is it on course to choose Jeremy Corbyn as its next leader? The answer's painfully clear. Corbyn stands for who Labour really, in its heart of hearts, believes that it is. It's anti-authority and anti-imperial. It's the party of the poor. It has a visceral dislike of privilege, and defines that very broadly. It is still the party of Nye Bevan, who famously said: "I know that the right kind of leader for the Labour Party is a kind of desiccated calculating-machine" – and like him, they're not having it.

In short, they like Corbyn because he represents what they'd like to be, though most of them probably aren't. He's the man who's kept the faith.

But here's the thing. The Americans have a saying: you campaign in poetry, but you govern in prose. The flights of rhetoric, the soaring appeals to the imagination, those glimpses of the New Jerusalem on the horizon, are enough to get you elected, but you're immediately plunged into the messy business of pragmatic compromise. High ideals are one thing, but unless they're matched with low cunning, they're completely useless: as WH Auden said, "Poetry makes nothing happen."

The temptation of the Labour Party is to retreat into a high-minded fundamentalism impervious to any concerns or arguments outside its own narrow perimeter. We must be true to our principles, the argument runs, and if the country/the world doesn't agree with us, it's their loss. So we have the ludicrously self-indulgent posturing of a rump Labour "rebellion" against welfare cuts, in which Labour MPs who believed that in voting against welfare cuts rather than abstaining they were being more 'Labour' than those who followed the party's official line. But this was about tactics and the long game, not about betraying their principles. The result is that the party is in chaos – a chaos that is only likely to deepen with a Corbyn victory.

Because what the party has not grasped – and it's rather hard to see why – is that they lost the election. After five years of austerity they lost to a party which promised five years of even more, including £12 billion of welfare cuts. There are various reasons for this – Ed Miliband's public persona being one of them, and the party's baffling refusal to fight back against accusations that it was responsible for the financial crash being another. But fundamentally, that part of the electorate that voted liked the idea of a smaller state and a government which did less. In the face of that shift in the attitude of the people toward their government, Labour had nothing to offer.

The truth is that a party which doesn't listen to the wider electorate as well as to its enthusiastic supporters is headed for irrelevance and extinction. Instead, there has to be a dialogue between Labour principles and economic and social realities. The question is not, "How can we do better at selling what we have?" It's "How can we adapt what we have to sell so that people will want it again?"

What too many Labour activists haven't realised is that there's nothing wrong with compromise. The trouble with fundamentalism – whether it's in the Church or in politics – is that it's ultimately self-defeating. The sense of belonging to a tight-knit group of true believers creates an illusion of strength – it's us against the world! but it never survives contact with wider reality. And because of that, it's impotent.

Tristram Hunt compared Corbynite Labour to Syriza. It's a good example. This was a party founded on as anti-austerity platform, pledged to defend Greece against the might of the European Union. It was as radical as they come and compromise was not in its vocabulary. The result was an excruciatingly long-drawn-out tragedy that has seen Greece nearly bankrupted and poverty on an appalling scale. And yes, Syriza had a good case, and yes, by all means lay the blame at Europe's door – but Syriza's leaders' failure to bend before the inevitable has left their country in a far worse position than it would otherwise have been. 

There is a moral case for compromise. There is a moral case for not voting with your conscience. There is a moral case for admitting that your opponent might have a point. There is certainly a moral case for respecting the people who voted against you. At the moment, Labour is a defeated and impotent party. Electing someone as its leader because he reminds members of their good old days is the best way of ensuring that it remains so. 

Candidly, it's not much of a field – but without someone at the helm who understands the nature of the task ahead, it is the Conservatives who will be setting the country's course for decades to come.