How to build building that is made of small pieces - how to correctly put rigid body over rigid body?
AFAIK Bullet (the library blender uses for rigid body simulations) is designed more for real-time speed over accuracy, so you might have to use some tricks.


Select all the objects in the tower and enable Enable Deactivation and Start Deactivated in Properties > Physics > Rigid Body Collisions.



Then right click on each setting and select Copy to selected. 

This will make it so the tower will not be simulated until another active rigid body comes near it.
Increase the number of simulation steps in Properties > Scene > Rigid Body World:



The higher this value is, the more CPU time is needed to calculate the simulation. However, higher values will also make the simulation more accurate.


Result: