Possible to displace whole polygons only, not points
Displace is always calculated on a per vertex basis, regardless of what you do, you can separate individual faces, and displace them, but the displacement is still gonna happen on each vertex individually, deforming each face. 

Also since most displace textures are "continuous" functions, the faces will still look as a continuous uniform surface as if they were still attached, since the displacement wont introduce any gaps or discontinuities.

You can try and mark all edges of your mesh as sharp, add an Edge Split modifier to separate all faces, then apply the displacement to your mesh.

To make the effect more visible you may also add a smooth modifier