Paul Song says he was falsely accused of calling one inmate a "terrorist", claiming the true motivation behind his removal was to clampdown on Christian ministry.
The pastor, who started volunteering at the south London jail in 1998 after moving to the UK from South Korea, said: "For 19 years, I served with an exemplary record.
"I worked alongside the prisoners and other staff members in harmony, recognising our differences and praying that they would come to faith, but equally respecting their decisions and background."
Mr Song said the prison's senior chaplain since 2015, Imam Mohamed Yusef Ahmed, pursued a "discriminatory agenda" which "drastically" changed the culture of HMP Brixton.
The final decision to dismiss Mr Song was taken by the prison's management, not Imam Mohamed.
He claimed he was contacted by the Prison Group Director at the London and Thames Valley Prisoners on 11th January noting that "the exclusion [from HMP Brixton] is permanent", confirming an email in August 2017 from Imam Mohamed to the same affect.
Mr Song claimed he also reluctantly cancelled evangelistic programmes such as the Alpha course following pressure from by Iman Mohamed - a claim refuted by Alpha, which said the course stopped running at HMP Brixton prior to the imam's arrival.
Mr Song alleged that Imam Mohamed wanted the programmes axed despite 70 inmates regularly signing up and the previous senior chaplain, Rev Phillip Chadder, supporting the courses.
He has made contact with the Christian Legal Centre, which is now considering legal action against Her Majesty's Prison Service.
Chief executive, Andrea Williams told Premier: "We've met at Christian Concern many ex-prisoners who've been absolutely delighted - and had their lives transformed - as a result of the work, ministry and witness of Pastor Song.
"This complaint was never substantiated.
"The name of the prisoner the imam said made the complaint was never given and, as a result of that, he was suspended and told he not come back into Brixton."
A spokesperson for Her Majesty's Prison and Probation Service told Premier: "We do not comment on individual members of staff.
"However, we recognise the importance of faith and the positive impact that it can have on the lives of offenders, which is why there are multi-faith chaplaincy teams in every prison."
Click here to listen to Premier's John Pantry speaking with Andrea Williams:
Stay up to date with the latest news stories from a Christian perspective. Sign up to our daily newsletter and receive more stories like this straight to your inbox every morning.