Responding to claims that some of the charity's workers hired prostitutes during a mission to the Caribbean country in 2011, Lord Leslie Griffiths said he repeatedly came across similar accusations during his 40 years' experience in humanitarian work.
The House of Lords member - and Methodist minister - said: "I'm very, very angry and I just hope that the situation now unfolding would lead to a proper investigation, more transparency and a greater sense of moral leadership."
"I think it's the tip of the iceberg; I've been aware of such incidents for the 40 and more years that I've been working in these sorts of areas..."
Meanwhile, International Development Secretary Penny Mordaunt said on Tuesday that a decision on whether to withdraw public funding from Oxfam over the aid worker sex scandal will not be taken "hastily".
The charity has issued an "unreserved apology" to the Government, donors, supporters and the people of Haiti over its handling of the allegations.
Ms Mordaunt said she would take the issue "very seriously" but stressed she would be guided by a Charity Commission inquiry into Oxfam while deciding whether to pull funding.
The charity received £31.7 million from the Government in 2016/17, but the support has been put at risk by the scandal, which led to the resignation of its deputy chief executive Penny Lawrence.
The Charity Commission said Oxfam may not have "fully and frankly disclosed material details" when it first investigated allegations of misconduct in 2011.
The watchdog also said it had concerns about the charity's handling of incidents since, and the impact that these have both had on public trust and confidence.
On Monday, Helen Evans, Oxfam's former global head of safeguarding, told Channel 4 News that she begged senior staff, ministers and the regulator to act on the sexual abuse allegations.
She also detailed three new allegations made against Oxfam staff overseas in a single day.
Shadow international development secretary Kate Osamor said both the watchdog and the Government have questions to answer over Ms Evans's claims they ignored her calls to act.
The Labour frontbencher said: "The Charity Commission and Government departments have serious questions to answer: why did they take no action in response to concerns raised by Helen Evans in June 2015 and August 2015? Are there other whistleblowers that have brought safeguarding concerns to the Charity Commission only to be ignored?
"It is crucial that we now understand how far this appalling scandal reaches, and whether the Charity Commission is operating effectively as an independent regulator."
Four members of Oxfam staff were dismissed and three, including the country director Roland van Hauwermeiren, resigned before the end of the 2011 investigation.
According to The Times, Oxfam knew about concerns over the conduct of Mr van Hauwermeiren and another man when they worked in Chad before they were given senior roles in Haiti.
Reportage by Premier's Alex Williams and the Press Association.
Stay up to date with the latest news stories from a Christian perspective. Sign up to our daily newsletter and receive more stories like this straight to your inbox every morning.