The pope's historic visit included a meeting with the Greek prime minister and the spiritual leader of the world's Orthodox Christians.
Francis took 12 refugees back to the Vatican with him. The group, including six children were all Muslim and will be cared for by the Church.
Two of the families come from Damascus, Syria, and had been forced to leave.
The Vatican called it a "gesture of welcome" to refugees.
Francis' visit was not a political move, the Vatican said, adding it was purely humanitarian and religious.
However it comes as the European Union implements a controversial plan to deport refugees back to Turkey.
For every Syrian sent back, the EU will take another Syrian directly from Turkey for resettlement in Europe. In return, Turkey was granted concessions including billions of euro to deal with the more than 2.7 million Syrian refugees living there, and a speeding up of its stalled accession talks with the EU.
As Francis walked around the Moria detention centre many refugees fell to their knees and wept. Others chanted "Freedom! Freedom!"
One woman told the pontiff that her husband had made it to Germany but European officials refused to let her and her two sons join him.
The pope bent down to console one young girl as she knelt at his feet sobbing uncontrollably.
Another child gave him a drawing. "Bravo, bravo", said the Pope before ordering his staff not to fold it. "I want it on my desk," he said.
Francis urged the refugees to know that they are not alone and shouldn't lose hope. He said he wanted to visit them to hear their stories and to bring the world's attention to their plight.
"We hope that the world will heed these scenes of tragic and indeed desperate need, and respond in a way worthy of our common humanity," he said.
Refugees are not numbers, they are people who have faces, names, stories, and need to be treated as such.
— Pope Francis (@Pontifex) April 16, 2016
The Greek Prime Minister Alexis Tsipras used the opportunity to attack other EU countries who had not done as much as Greece.
He told Francis he was proud of what his country had done "at a time when some of our partners - even in the name of Christian Europe - were erecting walls and fences to prevent defenceless people from seeking a better life."
After meeting the PM the pope had lunch with eight refugees to hear their stories of fleeing war, conflict and poverty and their hopes for a better life in Europe.
He then prayed at a port and tossed a floral wreath into the sea in memory of those who didn't make the journey.
Pope Francis in Lesbos in spring 2016
Chawkat Moucarry from World Vision told Premier the visit was a sign of hope for the refugees.
He said: "The pope, through his visit, will reassure those, whether Christians or not, those who think that the human beings belong to one human family and should show solidarity with each other, the pope will reassure those people."
On the plane to Lesbos Francis told reporters this was the "worst humanitarian disaster since the Second World War".
Listen to Premier's Hannah Tooley speak to Chawkat Moucarry here:
