It was easy once to dislike David Cameron intensely. If you grew up observing the genuine reform of the Labour party, under Neil Kinnock, John Smith, Tony Blair and Gordon Brown, it was grating to watch Cameron pretend to have done the same to the Conservative party in opposition.

After all, the Tories throughout his leadership remained unreconstructed on the major issues of the day: fiscal policy, immigration and, of course, Europe, as Cameron found dramatically to his cost. There was no 'Clause IV moment' as there was when Labour abolished its commitment to nationalisation. Instead, Cameron's commitment to gay rights and, eventually, same-sex marriage acted as a fig-leaf awarding him the title 'moderniser' in an often all-too-accommodating media. Cameron's social liberalism masked his party's ongoing, and unchallenged traditionalism and isolationism on areas that are arguably more important.

Yet Cameron, a practising Christian, himself matured as a One Nation prime minister in the Harold Macmillan mould – campaigning passionately for the Union with Scotland and then, less successfully of course, for Britain to stay in the EU – just as he has grown in his faith.

Cameron, who was raised in an old rectory in Berkshire next to a church where his father Ian was a warden, famously said in opposition that his faith came and went like the London radio station "Magic FM in the Chilterns".

Yet his belief is said to have deepened following the deaths of both his son Ivan in 2009, and his father, whom he idolised, in 2010. In office, he could occasionally be spotted quietly attending weekday Eucharist at St Mary Abbots Anglican church on Kensington High Street in London, next to the school attended by his daughter Nancy.

Michael Gove, who came out as a Christian in the Spectator last Easter, used sometimes to join him there. They no longer sit together, apparently. For the betrayal by Gove – even more than that of Boris Johnson – over the EU referendum which sealed the former prime minister's fate and ultimately led to his resignation from the House of Commons yesterday, is still raw. Just as Gove's anti-Europeanism was real, unlike that of Johnson, Gove's one-time friendship with Cameron was real, unlike that of Johnson.

Resigning as an MP, Cameron himself hinted last night that the referendum fall-out – what he called "the circumstances of my resignation [as prime minister]" – led directly to his decision. "I will give the Conservative candidate [in Witney] my full support," he said. "But in my view, with modern politics, with the circumstances of my resignation it isn't really possible to be a proper backbench MP as a former prime minister."

Perhaps predictably, Cameron has received a battering by the right-wing press today. The Daily Mail – which supports Theresa May, who appeared noticeably lukewarm about her predecessor yesterday – accused him effectively of abandoning the country. In its op-ed, Peter Oborne wrote: "David Cameron has gone and done a Tony Blair. In other words, he has deserted British politics and flounced off to make money... What a tragedy. If Cameron had any sense of genuine public service, he would have remained in Parliament, continuing to serve his constituents and the country he claims to love." He continued: "Not only has he damaged his own reputation, he has damaged Parliament and the image of Britain. What a pity. What a betrayal."

Oborne is a highly distinguished commentator and to be fair to him, he does not have an axe to grind against Cameron, as he does, perhaps, against Blair. But is it fair to attack a man whose close allies effectively destroyed him by campaigning against the EU, for leaving the parliamentary hot-house in which those new enemies still hold ambitions? There is also something curious about directing such venom at a man on the basis that you want him to stick around. There may be a hint of Richard Nixon's parting words about Cameron's move: "You won't have Nixon to kick around any more."

There is no evidence that Cameron made his decision to quit as an MP in order to make money (of which he has plenty), certainly not on the scale of Tony Blair. Yes, he has his memoir to write. But Cameron also has a hinterland. He reportedly enjoys "chillaxing" over lingering Sunday roasts and a glass of wine, long-ish walks and yes, occasionally, going to church. There is also some suggestion that Cameron wants to take on an international public policy job.

True, Cameron's fatal misjudgement on holding an EU referendum – as ever, to placate the insatiable Right –damaged much more than Cameron's own reputation: it has deeply damaged the country. But true, too, Cameron has a sense of public duty, and – shock, horror – what if he is telling the truth when he says that he wants his constituency of Witney to be represented by an MP who is "fully engaged" with parliamentary life?

You don't have to believe the fevered Westminster speculation that Cameron couldn't stomach May's reintroduction of grammar schools, to understand that life inside the Commons, the most gossipy club in the UK, did not appeal to a man who has seen his political legacy go within months from hero to zero.

Contrary to conventional wisdom, Cameron failed to change the Conservative party. But for a man who has undergone a change himself, it is understandable that he does not want to spend further years locked inside Parliament. He is not like Ed Miliband, who has stayed on in the Commons post-leadership despite not serving under Jeremy Corbyn, because he loves politics.

Cameron, who even critics accept is a fairly straightforward person, says he wants "to start to build a life outside Westminster". Who can blame him for that?