The Vatican is demanding the ancient Knights of Malta order obey instructions and cooperate with its enquiry into the worldwide Catholic charity.

The extraordinary row centres on the use of condoms in a medical project for the poor. 

The statement on Tuesday comes after the Knights' all male hierarchy defied the Papal-ordered investigation. It represents a new challenge to the reforms of Pope Francis from his conservative critics. 

The battle started after one of the Order's top knights, Grand Chancellor Albrecht Freiherr von Boeselager, was sacked on December 6 in the order's equivalent of a boardroom showdown - essentially because he allowed the use of condoms in a medical project for the poor.

The Vatican weighed in and ordered an investigation with Pope Francis effectively came to von Boeselager's defence. He urged dialogue to resolve the internal dispute and appointed a five-member committee to investigate the circumstances of his ousting.

The Knights of Malta branded the Vatican's investigation illegitimate and told members not to comply with it.

"Considering the legal irrelevance of this group and of its findings relating to the legal structure of the Order of Malta, the Order has decided that it should not cooperate with it," it said in a statement on its website.

It also instructed members not to "directly or indirectly" contradict the official position if they talk to Vatican investigators.

The Vatican responded with its own salvo on Tuesday saying: "The Holy See counts on the complete cooperation of all in this sensitive stage," adding that it "rejects ... any attempt to discredit (commission) members." 

The order, formed in the 11th century to provide protection and medical care for pilgrims to the Holy Land, has the status of a sovereign entity. It maintains diplomatic relations with over 100 states and the European Union and permanent observer status at the United Nations. It has about 120,000 members.

Members of the global Catholic chivalric and charity group are not ordained but take vows of poverty and chastity. 

The Order's chaplain is Cardinal Raymond Leo Burke, an American whom the Pope demoted from a senior Vatican position in 2014 and who has been a leading conservative critic of the pontiff. Burke backed the sacking of von Boeselager.

Von Boeselager said in a statement he shut down two projects in the developing world when he discovered condoms were being distributed but kept a third running for a while because closing it would have abruptly ended all basic medical services to poor people.

The Church bans condoms as a means of birth control and says abstinence and monogamy in heterosexual marriage is the best way to stop the spread of AIDS. 

Francis has said he wants the 1.2 billion-member Church to avoid so-called "culture wars" over moral teachings and show mercy to those who cannot live by all its rules, especially the poor.

Additional reporting by Reuters.