How to get displacement working on a completed house?
You can create vertex groups in edit mode and attach them to the displacement modifier. 


Select your wall's vertices.
Press space and search for add vertex group
Assign the vertex group to the modifier.