What does the "cancel" icon in the volume prompt mean?
It means that you cannot change the volume for the selected output device. For example, if you connect your mac to a TV using a HDMI cord, you will not be able to control the TV's volume with your mac. You'll have to use the TV's remote instead.

This shouldn't ever happen when you connect something into the headphone jack though. Perhaps you have a faulty set of headphones?