Canon Dr Andrew Bunch from St Giles' Church is reported to have threatened homeless people after three tents were set up in the churchyard and amongst gravestones.
Speaking to the BBC he said the church was "sympathetic" to these people, but he added that it was causing problems for people who wanted to use the churchyard.
He wrote a letter saying the church would give a court order if people did not leave, and they moved on.
He stresses that he only did so after asking people to move on and they had said no.
He said: "It had grown over the last three or fours weeks and was becoming a bit of a problem.
"They were causing us trouble because we found that people were feeling threatened as they were walking through, and it was reported to me that needles had been found around that area.
"So we didn't feel that was the right place for that sort of activity, given the fact that we've got other homeless people in the area and also we've got young children."
Canon Bunch said the church actively works with homeless people and hosts the homeless charity Gatehouse in its parish rooms.
