He replaces Iain Duncan Smith, also a Christian, who resigned on Friday night.
Stephen Crabb leaves his position as Welsh Secretary and will be replaced by Alun Cairns.
David Cameron said he was "puzzled and disappointed" after the Catholic MP Duncan Smith dramatically quit the Cabinet and launched an all-out attack on the "indefensible" Budget.
In a brutal parting shot, he complained that cuts to disabled benefits in George Osborne's financial package were "politically driven" and suggested the Chancellor had abandoned the austerity principle of "all in this together".
"I have for some time and rather reluctantly come to believe that the latest changes to benefits to the disabled and the context in which they've been made are a compromise too far," Mr Duncan Smith wrote in his resignation letter.
"While they are defensible in narrow terms, given the continuing deficit, they are not defensible in the way they were placed within a Budget that benefits higher earning taxpayers. They should have instead been part of a wider process to engage others in finding the best way to better focus resources on those most in need.
"I am unable to watch passively whilst certain policies are enacted in order to meet the fiscal self-imposed restraints that I believe are more and more perceived as distinctly political rather than in the national economic interest."
Mr Duncan Smith has faced mounting calls from senior Catholics to abandon many of his controversial changes to the welfare system since 2010.
In July 2015 more than 70 leading Catholics urged him to rethink, saying they feared the impact of his reforms.
"We understand that your Catholic faith is important to you, and your approach is driven by a desire to improve the quality of individual lives," the letter said.
"However, we believe that [your policies] are in fact doing the reverse. We would urge you to rethink and to abandon further cuts which are likely to cause more damage."
Mr Duncan Smith has been at loggerheads with Mr Cameron and Mr Osborne over whether Britain should stay in the EU, joining a handful of other Cabinet ministers in calling for Brexit. But his letter to the Prime Minister indicated that the row over cuts to the Personal Independence Payment (PIP) had been the last straw.
His announcement came hours after the Treasury signalled a humiliating climb-down over the plans to change PIP assessment criteria, which were expected to slash around £1.3 billion a year off the cost.
Government sources said they wanted to kick the proposals - initially announced by the Department for Work and Pensions last week - "into the long grass" and were not "wedded" to the savings figures featured in the Budget.
In his letter responding to Mr Duncan Smith, Mr Cameron wrote: "I regret that you have chosen to step down from the Government at this moment.
"Together we designed the Personal Independence Payment to support the most vulnerable and to give disabled people more independence. We all agreed that the increased resources being spent on disabled people should be properly managed and focused on those who need it most.
"That is why we collectively agreed - you, No 10 and the Treasury - proposals which you and your department then announced a week ago. Today we agreed not to proceed with the policies in their current form and instead to work together to get these policies right over the coming months.
"In the light of this, I am puzzled and disappointed that you have chosen to resign."