RPM's, VM's : time consuming
Have you considered setting up a build server (i.e. Jenkins, TeamCity) with disposable build agents?

If you do it in the cloud (i.e. via ec2 integration), you can literally set up an AMI with generic dependencies, then configure it to spin up a new VM for each build agent, which can be configured whichever way, and then delete the VM once you've done publishing your artifact (i.e. RPM).