Problem with Displacement in material
Displacement works by using existing geometry on the object.

It looks like your building has very little geometry, so only the small amount of existing vertices are moved causing this bending effect. To fix this, just select all of the edges in edit mode by pressing A, and subdivide a few times to create more geometry. As a little note, displacement often works best when the faces on the object are mostly squares.