He was jailed for 32 months back in October 2015 after pleading guilty to a string of historical offences, including two counts of indecent assault.
During his trial the judge said he was highly regarded as a godly man who had a special affinity with young people.
"The truth was that he used those 15 years in the position of bishop to identify, groom and exploit sensitive and vulnerable young men who came within his orbit," she said.
"For him, religion was a cloak behind which he hid in order to satisfy his sexual interest in those who trusted him."
The early release of the 84-year-old from jail, understood to have been on Friday, has been branded an "affront to justice" and a "huge blow to his victims".
Richard Scorer, a specialist abuse lawyer at Slater and Gordon, who represent a number of Ball's victims, said: "Ball's early release after serving just half of his sentence is a huge blow to his victims.
"This was a man whose appalling crimes represented a gross and systematic abuse of trust spanning decades.
"The fact he has only served 16 months in jail for crimes that scarred his many vulnerable victims is an affront to justice."
Not all victims share that opinion though. Rev Graham Sawyer's told Premier he has no issue with the early release.
He said: "It's not a problem for me. He's an old man and sending old people like him to prison for long periods of time is not necessarily good for the person who's in prison or society in general.
"It was right he was sent to prison and I'm grateful to the judge for doing that because it sends out a clear sign that Peter Ball's behaviour was completely unacceptable."
Ball has been banned from serving in the Church of England for life but the the National Probation Service has reportedly drawn up "bespoke" licence conditions with the Church so he can continue to exercise his right to worship.
Mr Sawyer told Premier that right is important for everyone.
"No one should be denied access to worship," he said. "It is important that his spiritual needs are met as with anyone else.
"Clearly safeguarding issues do arise and wherever he worships it must be assured that he cannot offend again but everyone has the right to worship."
It is understood the conditions include Ball being forbidden from doing or wearing anything that would give him clerical status, or from leading a service.
A Ministry of Justice spokesman said sex offenders are "robustly risk assessed and subject to a strict set of conditions".
"If they fail to comply, they can be recalled to prison," the spokesman said.
Listen to Rev Graham Sawyer speaking to Premier's Marcus Jones: