What does the "Height" map mean in Blender?
Height map means that it has values of height from the surface mapped to color values in an image file. This usually means that black is the level of the surface and the lighter it gets the higher off the surface it is, however it could be implemented the other way around in some rendering engine and the maps could still be called the same. 

This kind of a map can be used as a bump map or a displacement map depending on your needs and choice. Displacement maps can also consist of vectors. In that case 3 values are stored in the image to define the offset of the surface in 3 directions. In that case the displacement map would usually not be called a height map. 

There is no real difference between a height map and a displacement map usually, however there is a small difference in the meaning of the names. A height map can be a displacement map as well as a bump map, however a displacement map does not necessarily need to be a height map. A bump map is pretty much always a height map. That is all as far as I know.