Housing Justice told Premier the proposal, which is aimed at tackling poverty, drugs and crime, would cause a reduction in affordable housing.
The improvements proposed would see the number of affordable homes cut, the organisation's Alistair Murray said.
He added: "If we do want to regenerate these estates then let's do so, but let's at the same time maintain the housing that is being given to people on low incomes, and that's an increasing number of people.
"Our problem at Housing Justice with many of these schemes is, where are poorer families actually going to live?
"The private rented sector is touted as the answer but in fact the private rented sector is the least secure and the least affordable."
David Cameron pledged that "brutal high-rise towers" and "bleak" housing will be "torn down" in an effort to tackle drug abuse and gang culture.
The £140 million redevelopment programme is to be overseen by Lord Heseltine, who helped to transform the Liverpool and London docks in the 1980s.
His estate regeneration advisory panel has been told to produce a full blueprint by the time of the Chancellor's Autumn Statement.
Details of the scheme will be set out in a keynote speech being delivered by the PM on Monday, in which he is also due to outline plans to double government funding for relation-ship counselling for troubled families and relaunch a coalition proposal to issue vouchers for parenting classes.
Writing in The Sunday Times, Mr Cameron said: "Within these so-called sink estates, behind front doors, families build warm and welcoming homes.
"But step outside in the worst estates and you're confronted by concrete slabs dropped from on high, brutal high-rise towers and dark alleyways that are a gift to criminals and drug dealers.
"Decades of neglect have led to gangs and antisocial behaviour. Poverty has become entrenched, because those who could afford to move have understandably done so."
Alistair Murray said cuts in maintenance in the sixties had let to the problems in the estates.
He stressed: "The answer isn't simply to knock them down and replace them with what the government is calling affordable housing."
Alistair Murray speaking to Premier's Antony Bushfield: