What is the difference between the displace socket and a bump map?
Theoretically they are meant for similar purposes and functionally there is overlap when used in simple cases.

However, when used for extreme bump mapping, the bump map node produces much better results:



Left: Texture plugged into displacement output Right: Bump map node plugged into shaders



The other difference is of course that the displacement output can be used for actual geometry displacement (when when using the experimental feature set):



Left: Bump displacement Right: True displacement

Both spheres use the same material with the texture plugged into the displacement output. The sphere on the left uses the default Bump displacement method, while the sphere on the right uses the True method:



Note that the sphere on the right has a Subsurf modifier on level 4 to give it enough geometry to displace. There is a Use Subdivision option in the displacement settings, but I found this causes some wireframe artefacts.



Finally, using the Bump node allows you to choose not to have a bump map for certain shaders. For example you may want the gloss on the material to appear like a glossy coat or varnish, while still having a bump map for the underlying diffuse shader:



Left: Same displacement for all shaders Right: Different displacement for each shader



For reference, here is the sphere with no bump or displacement