What exactly does the "Non-color data" option for image/environment textures do?
This option stops blender from applying color-spaces to the image, this is useful when you want to use the numeric value of an images pixel rather then the perceived color value.
By default Blender converts an 8bit-per-channel image (As you would get with JPG, TGA, PNG, BMP etc.) from sRGB-colorspace to linear-colorspace (also known as rec709), before using it for compositing - where all color is assumed to be in linear-colorspace.
Examples where you don't want this include normal-maps, z-depth-buffer and displacement-maps - Where 0.5 should displace half the distance then 1.0.
Color spaces are fairly involved, so more could be written about the topic - but simple rule of thumb is if you are not viewing the image data as a RGB-Color, then you should set Non-Color Data.

Note that OpenEXR is an exception and color is assumed to already be stored in linear/rec709 colorspace, to display these images Blender converts them into sRGB (unless you choose a different colorspace).