Understanding Displacement Maps
The Displacement input of the Material Output node treats 0.0 as no displacement - rather than 0.5 as you expect. If you are using a Gray-scale image for your displacement you will need to adjust the values (0.0 to 1.0) to be centred around 0.0 rather than 0.5 and youcan achieve this by simply subtracting 0.5 using a Math node. You may also want to feed the output of that through a Multiply node so that you can adjust the magnitude of the displacement.