Okay, I admit it. The House of Commons speaker John Bercow, of whom I am a big fan, appears to have become a little more pompous in the chair than when I knew him as he first became speaker in 2009 and before that, when he was a Conservative MP whose journey from right to left of centre so fatefully infuriated some of his hard-line colleagues.

When I interviewed him for a long profile for Prospect magazine in 2011, he was disarmingly unpretentious, microwaving a spaghetti bolognese for lunch after a boisterous session of Prime Minister's Questions and chatting informally over coffee or Diet Coke in his parliamentary apartment with his amiable but controversial wife Sally, their children and their cat larking about in the background.

Now, not for the first time, his personality is under fire, as certain Tory MPs seek to push him out over claims, which Bercow emphatically denies, that he bullied his former private secretary Kate Emms, who worked for him between 2010 and 2011.

Personally, I must say that I find those – so far unproven – claims hard to believe, though they may be his undoing in the end.

But whatever the truth about them, the reality is that Bercow is himself nonetheless being bullied in a distinctly frenzied way by a few Tory MPs who have long held a grudge against him and for a combination of ugly reasons including politics.

One of those is Andrew Bridgen, and you would have had to get to the final paragraph of a recent Guardian news story to find out what is really going on with him: 'Bridgen, who is pro-leave, admitted that his motivation to move against Bercow was partly down to his views on Brexit. "Given his apolitical position as Speaker of the House of Commons, he shouldn't be able to trumpet them from the rooftops as he does."'

Another Conservative MP, James Duddridge who has long had it in for the Speaker, has talked separately about whether Bercow should be referred to parliament's standards committee for bringing the house into disrepute.

The point is that there is nothing new here.

In 2009, Bercow beat nine other candidates to the sacred post of Speaker on a reforming, 'clean break' platform. Since then, he has set about strengthening the legislature against the over-mighty executive, allowing an unprecedented number of urgent questions (UQs), pressing for elected select committee chairs, backing Friday (and September) sittings and, from his first day in the post, speeding up questions in the House.

Yet on the Tory backbenches lurked a rogue band of backwoodsmen determined to undermine him from the start. Back then, it was Christopher Chope, Greg Knight and Nadine Dorries (she later changed her mind about Bercow) who were his frequent tormenters.

They had never forgiven Bercow for a 'conversion' that drove the one-time Monday Club member to describe his party as 'racist, sexist, homophobic and anti-youth'.

Then there is simple snobbery. MPs have overheard Bercow described as an 'oik' and the 'son of a minicab driver' - something of which he is proud.

Sources close to the Speaker's office have even whispered in the past of a strain of Tory anti-Semitism (Ber­cow is Jewish), a stain on the party of Benjamin Disraeli familiar to any reader of Alan Clark's diaries (though to his credit, the then prime minister David Cameron welcomed the 'milestone' of the 'first person of Jewish faith' to be Speaker).

John Bercow has been a good Speaker. 'It's the most rewarding job in the House that I've ever done or could ever do,' he told me in 2011.

Whether he will be doing it for much longer remains to be seen but, as Tony Blair once said of Peter Mandelson, he is a better man than many of his critics.

Follow James Macintyre on Twitter @James_Macintyre