The centre at Milton, Stoke-on-Trent, is licensed to hold about 500 tonnes of recyclable waste, but last December the Environment Agency estimated there were about 6,000 tonnes.
The agency wants excess waste gone by July and said progress has been made.
Owner Hanbury Plastics Recycling Ltd (HPR) said it was applying to increase its licence to hold 3,000 tonnes.
Resident June Grimes said: "[We] shouldn't have to put up with the smells and the issues as these sites bring when [they're] put on your doorstep.
"What we want to see happen is this site completely cleared and relocated to a suitable industrial site."
Ms Grimes filmed images on her mobile phone of drivers trying to negotiate Milton's narrow lanes and bridges.
She added: "We have got heavy goods [vehicles], left hand drive, coming on our roads where our children walk to go to school and they're that big, they're mounting the pavement as they're going down."
A spokesman for the owners said it was the best way to deal with plastics, otherwise they would end up in landfill.
HPR said it was planning to double the workforce to 60 over the next few years.
An agency spokesman said it issued the company with a notice in December to remove excess waste from the firm's site.
He said: "Since that time, we have carried out regular checks to ensure that the company is working to remove the waste by the end of June 2014.
"To date, the company have made progress and moved waste from the site. We will continue to monitor progress and then decide on the best course of action once the deadline has been reached."
Residents are calling for a recycling collection centre to be closed after it was told to remove excess waste.