They have become two of the most recognisable estates in the country, after appearing as the backdrop for films such as Harry Brown and numerous TV shows including the The Bill and Spooks.
It is the sheer size of the estates that is a draw to directors.  With long lines, corridors and walkways, they can be a film-maker's dream.
But as a result of complaints, Southwark Council has enforced strict guidelines for anyone wanting to film on the Aylesbury or Heygate housing estates.
"I don't want to see any filming on the Aylesbury that portrays violence and all the things that are not right for the estate," says Jean Bartlett, who speaks on behalf of the Aylesbury Residents and Tenants Association.
On iPlayer
Speaking to BBC Radio 4's Front Row she admits the locals have, in the past, welcomed stars and film crews.
"We all did (welcome them) at first because there was funding coming from the film companies.
"I think that over time time it was the same subjects being used.... violence, drugs and run-down areas, which then portrayed the estate as a hell-hole."
The 2009 film Harry Brown, starring Sir Michael Caine, depicted an elderly man living in fear of violent gangs. Other gritty television programmes filmed on the estates have also included gang drama Top Boy.
Jean is also unhappy at the way the location has been used in the Channel 4 promotional trailer.
"Every time I see the logo it makes me cringe.  They put washing hanging out in areas that doesn't even have washing.
"They put loads of rubbish across the whole of the area. It's absolutely terrible."
In 1997 the then Prime Minister, Tony Blair, chose the Aylesbury Estate to make his first major speech outside of Parliament. At that time, the estate symbolised the problems of urban deprivation and despair.
Mr Blair unveiled his vision for regenerating Britain's inner cities and underlined his government's determination to tackle the issues and declared that there must not be any "no hope areas".
British rapper, Tinie Tempah, grew up on Aylesbury Estate and his memories of living there are in tune with some of the darker sides of films shot there.
"I remember growing up and seeing grey, very narrow walkways which are sort of prone to conflict.
"You can only walk past the same person so many times without someone saying 'Who are you?' or 'What's all this about?'
"There were very grubby lifts where people would urinate. Those high-rises, when you wake up in the morning, they don't inspire you - you don't feel the need to want to get out of there."
Unique memory
Despite millions of pounds being spent on regeneration over the years, the Aylesbury Estate, together with the neighbouring Heygate Estate, are in the process of being demolished and rebuilt.
These two projects will cost Â£3.9bn and involve the construction of hundreds of new homes, shops and open spaces and the rebuilding of several schools.
But the film director Joe Cornish, who made the science fiction hit Attack The Block on the Heygate Estate, says he chose the area because it has a futuristic appeal.
"I came here first and foremost because it's huge, instantly familiar as part of British public housing and it's deserted.
"This architecture reminded me of sci-fi movies in the '60s and '70s like Clockwork Orange or Logan's Run.
"You could go back even to the film Metropolis where architecture like this was seen as something futuristic and aspirational. I suppose it was more viewed as a utopia like it was when it was first built."
Joe Cornish hopes his film will provide a unique memory of the estate.
"Once this place has been demolished it would have been immortalised in a handful of films.
"I guess it would have been portrayed in a very particular way but hopefully Attack the Block portrays it with a bit of escapism, fantasy, colour and upbeat-ness compared to the miserable way it has been portrayed in some other movies."
Southwark Council, who run the estates, say the decision to block the majority of film requests is a reflection of the tenants' views.
"There was a very clear message coming through from the residents that they'd had enough of it," says Councillor Fiona Colley. "They were sick of seeing their estate portrayed as dirty, grimy and dangerous and it had to stop.
"We will occasionally make an allowance. We've recently made an exception for World War Z on the Heygate Estate.
"Rather than portraying the Heygate, it was portraying projects in Philadelphia and it wasn't local kids causing mayhem, it was zombies.
"But in contrast, Harry Brown used the same location and you've got local kids rioting down the same bit of the same estate and people think that is what it's like."
Aylesbury resident Jean Bartlett says film-makers will not be welcome unless "somebody comes along with a decent story that doesn't portray us as 'hell's waiting room' with a negative image."
So, what about if Brad Pitt drops in?
"No, not even if Brad Pitt drops in. No."
Many people would enjoy opening their front door to see a Hollywood star making their new movie, but residents living in the Aylesbury and Heygate housing estates in South London have had enough of film-makers.