US House speaker Paul Ryan has found himself in a battle with the House chaplain, Patrick Conroy, after it emerged Ryan forced him to resign.

Conroy has now retracted his initial resignation and Ryan has allowed him to stay on.

'I have never been disciplined, nor reprimanded, nor have I ever heard a complaint about my ministry during my time as House Chaplain,' Conroy wrote in a letter first reported by The Washington Post.

'It is my desire to continue to serve as House Chaplain...to the end of my current two-year term, and beyond, unless my services are officially terminated (however that is properly done) or I am not re-elected to the position by the membership of the House,' Conroy wrote.

Conroy is a Jesuit Catholic priest and said he was pushed into resigning by Ryan's chief of staff, Jonathan Burks.

'I inquired as to whether or not it was "for cause," and Mr Burks mentioned dismissively something like "maybe it's time that we had a Chaplain that wasn't Catholic,"' Conroy wrote.

The controversy sparked a backlash from Catholics in both parties as Ryan initially defended the decision to ask Conroy to quit, citing concerns from some about his 'pastoral needs'.

But Conroy denied any criticism over his counselling had ever been raised and instead suggested he had irritated Ryan with prayer on the House floor that was perceived as critical of the GOP tax bill.

Sources initially told The Hill that he was being forced out 'because Republicans thought he was aligned with Democrats'.

In a statement on Thursday afternoon Ryan backed down and said he had 'accepted Father Conroy's letter and decided that he will remain in his position,' promising to meet with him next week.

'My original decision was made in what I believed to be the best interest of this institution,' Ryan said. 'To be clear, that decision was based on my duty to ensure that the House has the kind of pastoral services that it deserves. It is my job as speaker to do what is best for this body, and I know that this body is not well served by a protracted fight over such an important post.'