In October, the Church accepted assault allegations against Rt Revd George Bell, who was Bishop of Chichester from 1929 until his death on 3rd October 1958, and paid compensation to the alleged survivor.
In a statement, officials said there was no "reason to doubt the veracity of the claim" and apologised for his actions.
Peter Hitchens
Peter Hitchens told Premier the Church had acted wrongly because Bishop George was unable to defend himself. He said: "The presumption of innocence is one of the most important freedoms that we have, I think it should be applied in this case."
Accusations that Bell abused a young child between the late 1940s and early 1950s were made in 1995 but the Church has now admitted the response "fell a long way short".
The survivor told the then Bishop of Chichester, Eric Kemp, about the abuse in August 1995 but they were only offered pastoral support and the allegations were not referred to the police.
The Church said from the information it has, it seems Bishop Kemp did not investigate the matter further.
It was not until the claim was made to Lambeth Palace in 2013 that the survivor was put in touch with the safeguarding team at the Diocese of Chichester who referred the matter to the police and offered personal support and counselling to the survivor.
The serving Bishop of Chichester, Rt Revd Dr Martin Warner, said he felt "deep sorrow" and acknowledged that "the abuse of children is a criminal act and a devastating betrayal of trust that should never occur in any situation, particularly the church."
Peter Hitchens said statements like this implied guilt to a man who had no trial and could not defend himself.
"It's rather odd for the Church to have gone so public on the reputation of one of the very few of its bishops who could really be called great," he said.
"Of course the charges may be true but we haven't had that demonstrated to us beyond reasonable doubt in a court of law and I doubt we ever will, in which case we shouldn't behave as if they had been.
"It makes me feel that there is the possibility of injustice here and that those of us who are in favour of justice should resist that and say wait, until you've actually proven the case beyond reasonable doubt against someone he must be presumed innocent.
"You can't just say that because an accusation is particularly scandalous and appalling that we have to suspend our normal judgement and abandon the presumption of innocence.
"I don't see why calling for due process should discourage anybody who's been genuinely wronged from seeking to expose their complaint."
The Church of England said it would not be issuing any further statement on the issue.
Dr Jon Bird, NAPAC's Survivor Support Manager, said: "If the Church hadn't fought tooth and nail to protect known abusers within its ranks, this case would have come to light when Bell was still alive."
Listen to Peter Hitchens speaking to Premier's Antony Bushfield:
