They have written a letter asking for Lord George Carey, the former Archbishop of Canterbury, to be criminally investigated.
Published in the Times, it reads: "Following the Gibb report on this case... three issues require urgent attention. First, the police must investigate whether Lord Carey of Clifton deliberately concealed evidence.
"If so, Lord Carey must forfeit his right to sit in the House of Lords. It is unacceptable that someone involved in concealing evidence of criminality should have a role in making laws for others".
Lord Carey resigned from his position of honorary bishop in the Diocese of Oxford on Monday.
In the letter, the survivors also said safeguarding in the Church of England requires independent oversight, saying "the Church cannot be allowed to carry on policing itself."
Their last request was to create a mandatory reporting law "requiring that knowledge or suspicion of abuse be reported to the authorities."
The letter states that Peter Ball would have been caught much earlier if such a law was in place.
