The city's mayor, Joao Doria, said he did not want to see an "open-air shopping mall for drugs".
He said he wanted to move the addicts on and provide them with medical services and housing.
Critics say his policies are pushing the problem to other parts of the city.
At the end of May, 500 armed police officers were deployed to clear the main streets in the area of Sao Paulo known as "Cracolandia" or "Crackland".
Nearly 40 people were arrested during the operation for drug-trafficking and many of the addicts fled into neighbouring streets.
The biggest concentration went to Princess Isabel square around 300 metres (330 yds) from the main Crackland area.
During the second operation on Sunday police brought in bulldozers to clear the shacks and tents set up by the addicts in the square.
The mayor said: "We wont be turning back. The idea is not to move them elsewhere,  we intend to provide the addicts with support and try to guarantee their survival."
He has promised to restore and renovate the area providing social housing, schools, kindergartens and hospitals using private and public investments.
Although many people agree in Sao Paulo over the need for intensive policing in the area, Mr Doria has divided opinion over his suggestion that the addicts should receive compulsory rehabilitation treatment.
But workers say many addicts are fearful and hostile to them now and that Mr Doria's policies are destroying years of work building relationships with the addicts.
Social workers also point to the need to provide jobs and housing.
The Doria administration ended a programme introduced by the previous left-wing mayor of Sao Paulo, Fernando Haddad, which had offered hotel rooms in Crackland for a small daily sum to drug addicts who worked in jobs like road-sweeping or gardening.
The mayor has instead recently announced an offer of 100 jobs by a fast food chain to homeless people in general, of which there are around 20,000 in Sao Paulo.
Security forces in Sao Paulo have cleared a central square of crack addicts and homeless people who fled from a similar police operation nearby nearly a month ago.