Malaysia has ordered an investigation into a protest by Muslim demonstrators who forced a small church in the capital to take down its cross, the latest sign of growing religious tension in the multiracial Southeast Asian nation.

Prime Minister Najib Razak said his Cabinet had discussed the protest in the mainly Muslim Kuala Lumpur suburb of Taman Medan and that demonstrators could be charged under Malaysia's controversial Sedition Act.

"If they are found to have violated the country's law, action will be taken under the Sedition Act or other existing laws," Najib said in a statement issued late on Tuesday.

Malaysian media reported that a group of about 50 protesters had demonstrated outside the church on Sunday because they believed the cross, one of the central symbols of Christianity, posed a challenge to Islam and could sway the faith of young people.

The cross was taken down by the church a few hours after the protest, media reported widely.

Najib said the Cabinet believed protesters should not have taken any action that could cause unrest in Malaysia, which has a predominantly Malay-Muslim population.

The protest outraged non-Muslims as well as moderate Muslims, many of whom have expressed anxiety about growing signs of religious intolerance amid a drift to a more conservative form of Islam.

Last year, Malaysia's federal court ruled that a Christian newspaper could not use the word "Allah" to refer to God, a case that fanned religious tension and raised questions over minority rights.

An Islamist party is also lobbying lawmakers to support its move to introduce an Islamic penal code in the northern state of Kelantan that stipulates stoning for adultery and amputation for theft.

Malaysia toughened its colonial-era Sedition law earlier this month, which the government had meant to repeal but now says is essential to maintain religious harmony.

Critics accuse the government of using the law to crack down on opposition politicians, journalists, academics and activists.