Patel is facing almost certain dismissal after being ordered back to Britain following the disclosure of further unauthorised meetings with Israeli politicians.
She was forced to cut short an official visit to Africa after being summoned by Theresa May to Downing Street to explain herself.
Ms Patel has already apologised to the Prime Minister on Monday after failing to disclose a series of 12 meetings with senior Israeli figures during a family holiday in the country in August.
It has since emerged that she then held two additional meetings, one in the UK and one in the US, following her return from Israel.
Sir Gerald, who's a friend of Ms Patel, told Premier she made an "honest mistake".
He said: "It will be a shame if this media frenzy were to force the Prime Minister's hand.
"I think it's an innocent mistake. No damage has been done, nobody has been hurt, no public money has gone astray."
Sir Gerald added that the media has spun the situation out of control and it's depriving the country of talented people.
He said: "Some journalists are completely unaccountable. They're unaccountable to the people they work for and certainly unaccountable to the public.
"They'll move on to the next story, and no doubt they'll find somebody else and seek to secure their scalp."
Sir Gerald told Premier he was in a similar situation as Ms Patel when he was in office and said the finger pointing is "absolutely dreadful".
Bernard Jenkin MP for Harwich and North Essex also defended Patel suggesting inexperienced people are too quickly put into top jobs without training.
Sir Gerald agreed and told Premier: "It is very difficult... you have no free time, your diary is completely saturated, you are therefore learning on the job and you're dependent on your officials to guide you as far as possible to make sure you don't make mistakes."
Labour peer and former Lord Chancellor Lord Falconer said Patel should "definitely" go and branded her actions "appalling".
He told BBC Radio 4's Today programme: "She is a member of the British Government, she should not be colluding with a foreign government, it doesn't matter whether it's an ally or not ... to not tell anybody, to talk to them without telling the British Government then come back and begin to ask for things that help that government means that she's not part of a collective government trying to do the best for Britain."
Downing Street has denied a report in the Jewish Chronicle that Patel told Mrs May in the run-up to the UN General Assembly in September that her meetings in Israel had included talks with Prime Minister Benjamin Netanyahu.
No 10 also dismissed a claim by the newspaper that the Prime Minister had instructed Ms Patel not to include one of the latest meetings in the list she released on Monday so as not to embarrass the Foreign Office.
Listen to Sir Gerald Howath speaking to Premier's Tola Mbakwe here: