Justice secretary Michael Gove today said his Christian faith informs his prison policies, telling a critical Tory MP: "I believe in redemption".

Gove rejected criticism from Philip Davies, a Conservative MP who sits on the justice committee, who said that Gove had "gone native" at his department.

Davies, who is on the right-wing of the Conservative party, accused Gove of being more liberal than his predecessors, jibing "come back Ken Clarke, all his forgiven". The MP for Shipley attacked the secretary of state for "hanging off every word" of the Howard League for Penal Reform, which Davies described as the "NUT of the justice system".

"When will the secretary of state get back his mojo and actually put the victims of crime at the heart of what he is doing?" Davies asked.

Gove, however, quickly retaliated.

"I am not sure that MPs on the Opposite [Labour] benches would agree I've become a sandal-wearing muesli-munching vegan vaguester. I think they would probably say I'm the same red in tooth-and-claw blue Tory I have always been," he told Davies.

"It's because I am a Conservative I believe in the rule of law as the foundation stone of our civilisation.

"It's because I'm a Conservative I believe that evil must be punished. But it's also because I'm a Conservative and a Christian that I believe in redemption.

"The purpose of our prison system is to keep people safe by making people better."

Gove was given the role of justice secretary after the 2015 general election and handed the task of repealing the human rights act in favour of a British bill of rights – one of the most controversial aspects of the Conservative manifesto.

However since May, Gove has been less controversial than in his previous role of education secretary. He has reversed a number of unpopular policies of his predecessor Chris Grayling, much to the delight of campaigners.

He has abandoned Grayling's ban on prisoners receiving books and also scrapped the criminal courts charge. He has also ditched a ministry of justice contract that would have meant the UK government was involved in construction of Saudi Arabian prisons.

His latest rebuke to the right of his party will only delight campaigners further.