It's business as usual today for the Speaker of the House of Commons as the Westminster wolves seek with increasing desperation to hound him out of office.

More bullied than bullying, John Bercow is quietly visiting a primary school in south London, one of many such unreported trips which thrill students across the country. He will read their work on the Magna Carta and the suffragettes, and inspect their own political manifestos.

For Bercow as well as the pupils, it will be a breath of fresh air away from the troubles in Westminster, where accusations of 'bullying' are reaching a nasty climax. He has faced a range of allegations, some serious – one former aide claims she suffered post traumatic stress disorder (PTSD) after working for him – some much less so. He can at times, with his public manner, be his own worst enemy.

But back in the Commons, as so often, the fearless Ken Clarke said it best yesterday. One of the few genuine big beasts in the current House of Commons, Clarke – who knows a thing or two about being caught out muttering insults – very unfashionably leapt to the defence of the Speaker when he told the House of Commons that if every MP who'd ever done so was kicked out the place 'would be deserted'.

The veteran MP was referring to what the Speaker's critics hope will be the final spark for his departure, the claim that Bercow called the minister Andrea Leadsom a 'stupid woman'.

Leaving aside whether the term is any different to saying 'stupid man', the claim of sexism against this very socially liberal Speaker is surely misplaced. Indeed, there is an irony that one of his several long-time tormentors-in-chief latching on to the 'sexist' charge, James Duddridge, is a keen supporter of Donald Trump while another, Andrew Bridgen, is not known in the corridors of Westminster for his fervent support of feminists.

But this isn't really about sexism, of course, despite the misguided attempts by some liberal journalists to become part of the pack hounding Bercow, as per the usual Westminster game of hunt-the-victim and kick-a-man-when-he's-down. Instead, this goes back to a cabal of rightwing MPs who have sought to bring this reforming Speaker down from the start (I first wrote about this in January 2010 for the New Statesman, in a piece, published just months after he came into office, titled 'Speaker Cornered').

Back then, it was the Tory MPs Christopher Chope and Greg Knight. Now it's Duddridge and Bridgen. What their constituents make of their MPs' obsession with pursuing one man in Westminster, we cannot know.

For them, it is not just about the fact that the Speaker abolished a shooting area in the House of Lords and one of parliament's many bars and created a parliamentary nursery. It's not that he is the most effective Speaker ever when it comes to holding the executive to account, empowering select committees and granting an unprecedented 468 Urgent Questions (UQs). True, the government loathes the fact that Bercow has revived the previously dormant institution of UQs, of which there used just to be two or three a year; now, there cannot be a national crisis – such as Windrush – without Bercow summoning the relevant minister to the floor of the House, and they hate it.

But no, it's because as well as, in 2002, marrying a Labour supporting wife Sally – 'that woman' as some Tory MPs who now claim to be bastions of feminism called her – he has himself completed a journey from right to centre-left demonstrated by his letter to his then constituency party in Buckingham describing his party as 'racist, sexist, homophobic and anti-youth'.

Then, in a move that marked the end of his career within the party, he resigned from the shadow front bench over the refusal by Iain Duncan Smith, then leader, to back adoption for homosexual couples.

In 2005, Bercow further angered the right when he backed Clarke over David Cameron for the party's leadership, and remarked that an Old Etonian and member of the male-only club White's wasn't the best advertisement for a modern Tory party.

This from a man who, before entering parliament, had embedded himself with traditionalist Tories as the secretary of the immigration and repatriation committee of the Monday Club, a right-wing pressure group and who, while campaigning to be selected before the 1997 election, was so keen to win that he used a helicopter to get from one selection meeting to another in time. He was the Thatcherite darling, and the Tory right had high hopes for him.

Now, there is snobbery and perhaps more than a hint of antisemitism among his critics who have referred to him in private as 'oily' and 'not one of us'. Bercow comes from a Jewish, working-class background in Edgware, north London, and is the proud son of a minicab driver.

Right now, John Bercow is in a lonely position. Can he survive?

On June 21, Bercow will host a group of Christians in the Speaker's House under the umbrella of Embrace the Middle East. The topic of discussion will be persecution. The event falls on the eve of the ninth anniversary of Bercow taking office. Criticism is likely to have reached fever pitch by then, as he has indicated that he wanted to do nine years in the role.

My guess is that Bercow will not stop serving immediately but will instead announce his departure date, perhaps the following year. But whatever he says, he will announce it first to the House of Commons which, whatever his faults, he has served with undeniable distinction.