Do we really need to leave materials below the completed structure?
In the base game there is no building that interact with the materials, so, in this case, the answer is yes, you can remove them from the game safely.

But in "Glory to Rome Promo Cards" set you have three buildings that do interact with the materials. Castra Praetoria, Solarium and Watch Post. For the first two cards the kind of interaction is not important here, they move materials upon building completion, after that you do that you could still remove from the game the rest of materials. But Watch Post states : you may use a CRAFTSMAN action to tear down one of you completed buildings. Add foundation to your hand and materials to your stockpile. 
So if you are playing with that card the answer is no, you cannot remove them from the game safely.

(I hope that my answer is understandable ... I am not english native speaker)