Leaving the cabinet was "in the best interests of my family", he said in a statement.
New Prime Minister Theresa May has replaced him with Damian Green, who served under Ms May at the Home Office for four years as immigration minister and policing minister until his surprise sacking by Mr Cameron in 2014.
Mr Crabb, 43, was only recently elevated to one of the more senior Cabinet jobs and had been tipped as a future Tory leader and potentially prime minister.
He dropped out of the race to replace Mr Cameron lacking support.
On Sunday The Times reported that he had sent sexually explicit WhatsApp messages to a young woman during the EU referendum campaign.
He had built his campaign on his Christian views and portrayed himself to be a family man with strong principles and morals.
In a statement he said: "Over the last two years I have had the huge privilege to serve in the Cabinet.
"After careful reflection I have informed the prime minister today that, in the best interests of my family, I cannot be part of her government at this time.
"I am grateful to my whole team for their hard work and encouragement. I look forward to supporting the government's One Nation vision from the backbenches."