How to get better displacement result
I don't think this is a polygon problem, you are just stretching the texture so much it just has nothing to put on this side of the brick, it has no information. It's just not possible to do this this way, you have to model it and texture it manually