Rev Barry Trayorn who worked as a gardener, but volunteered in the chapel, fell into trouble after delivering a talk to prisoners about homosexuality and sin.
Following a complaint, he was disciplined then later resigned.
Last year an employment tribunal ruled that his employers acted within the law.
A judge has now confirmed that ruling was fair, claiming his words could legitimise mistreatment of homosexual prisoners.
HMP Littlehey
Trayhorn is being supported by the Christian Legal Centre.
Chief Executive Andrea Williams said: "This ruling sets a dangerous precedent not only for prison chaplains but for any minister who preaches the gospel. To say that quoting a verse from the Bible can be offensive, could have serious implications on the freedom of prison ministers to share the good news of the gospel.
"It should not be for the state to decide which parts of the Bible can and cannot be quoted during preaching, nor to dictate that verses that some may find unpalatable should be interpreted to fit with current social norms.
"It was clear that Barry's talk centred on God's forgiveness and love for those who repent of their sin. This is a message that those imprisoned for sexual offences desperately need to hear. Our prisons are in need of the light of the gospel, yet this ruling sets a trajectory towards the Bible being forbidden in these institutions."
Trayhorn will now take his case to the Court of Appeal.