David Cameron has made a strong defence of Communities Secretary Eric Pickles who has come under fire from Britain's Muslim leaders after calling for mosques and imams to work harder to fight Islamist radicalisation.

Cameron said everyone had a responsibility to fight extremism. He said the letter was the most "reasonable, sensible, moderate" letter that Mr Pickles could possibly have written.

In the wake of the Charlie Hebdo attacks, Pickles and communities minister Lord Ahmad of Wimbledon wrote: "You, as faith leaders, are in a unique position in our society. You have a precious opportunity, and an important responsibility, in explaining and demonstrating how faith in Islam can be part of British identity," the ministers wrote.

"We believe together we have an opportunity to demonstrate the true nature of British Islam today. There is a need to lay out more clearly than ever before what being a British Muslim means today: proud of your faith and proud of your country. We know that acts of extremism are not representative of Islam, but we need to show what is."

The letter led to protests from Muslim leaders. Harun Khan of the Muslim Council of Britain said: "We will be writing to Mr Eric Pickles to ask that he clarifies his request to Muslims to 'explain and demonstrate how faith in Islam can be part of British identity'.

"Is Mr Pickles seriously suggesting, as do members of the far right, that Muslims and Islam are inherently apart from British society?"

Mr Cameron said: "I think it is absolutely right to write this letter, to say that we all have a responsibility to fight extremism."

Answering questions after a speech today on employment in Ipswich, the Prime Minister said: "Anyone who reads this letter, and I've read the letter, will see that what he's saying is that British Muslims make a great contribution to our country, that what is happening, in terms of extremist terror, has nothing to do with the true religion of Islam.

"It's being perverted by a minority who have been radicalised.

"But everyone needs to help dealing with this problem of radicalisation.

"And anyone, frankly, reading this letter who has a problem with it, I think really has a problem. Because I think it's the most reasonable, sensible, moderate letter that Eric could possibly have written. And, frankly, all of us have a responsibility to try and confront this radicalisation and make sure we stop young people being drawn into this poisonous, fanatical death cult that a very small minority of people have created."

In the letter, Pickles and Ahmad wrote: "We have recently seen terrible atrocities committed in Paris. Finding the right response to these events is a challenge for everyone. The hijacking of a great faith to justify such heinous crimes sickens us all. As Muslims around the world have made clear, such actions are an affront to Islam.

"And yet, amid the carnage, came a sign of hope - over three million people of all backgrounds marching to defeat the gunmen and to protect our values: free speech, the rule of law, and democracy.

"We are proud of the reaction of British communities to this attack. Muslims from across the country have spoken out to say: not in our name.

"But there is more work to do. We must show our young people, who may be targeted, that extremists have nothing to offer them. We must show them that there are other ways to express disagreement: that their right to do so is dependent on the very freedoms that extremists seek to destroy.

"We must show them the multitude of statements of condemnation from British Muslims; show them these men of hate have no place in our mosques or any place of worship, and that they do not speak for Muslims in Britain or anywhere in the world.

"Let us assure you that the Government will do all we can to defeat the voices of division, but ultimately the challenges of integration and radicalisation cannot be solved from Whitehall alone. Strong community-based leadership at a local level is needed."