Wayne Murdock, senior investigating officer into sexual abuse allegations against Ball in 1992 and 1993, said he expected Archbishop George Carey to provide information handed over by victims.
Lord Carey, who gave evidence to the enquiry on Tuesday, resigned as honorary assistant bishop in the Diocese of Oxford after an inquiry found he delayed a "proper investigation" into Ball's crimes for two decades by failing to pass the information to police.
Mr Murdock, who was a Gloucestershire Police officer for 37 years, told the Independent Inquiry into Child Sexual Abuse on Wednesday: "Those letters should have been handed over and they could have made a difference.
"They knew what was going on. My expectation would have been that if you've got anything, you hand it over.
"From the Church, you don't expect information to be withheld. You know, that's really common sense."
It was suggested by Fiona Scolding QC that information given to Gloucestershire Police by victims - including details about prayer while not fully clothed and massages - was not materially different to letters received by Lord Carey at the time.
Mr Murdock said: "We should have been the judge of that, not the archbishop. He knew we were carrying out investigation and he withheld that information.
"Had we had names, we could have followed up, gone and interviewed them and perhaps we would have got far more out of those people than had been revealed in those letters.
"I was led to believe these letters were far more damaging. Bottom line was those letters should have been passed on for us to look at and pass judgment on what the evidential value was."
The inquiry heard excerpts from some of the letters on Tuesday, including one given by the mother of victim Neil Todd.
Mr Murdock also denied Ball received special treatment when he was called to Gloucester Police Station for questioning in 1992 but kept away from the cell block.
He told the inquiry he had wanted to stop the media from finding out about Ball's arrest to prevent interference, but said the news broke after being leaked "undoubtedly from somebody at the police station".
Ball resigned in 1993 following a police caution and was jailed in 2015 for sexually abusing 18 males over 30 years.
Stay up to date with the latest news stories from a Christian perspective. Sign up to our daily newsletter and receive more stories like this straight to your inbox every morning.
