What could cause water to come out of the faucet when the shower is on?
Your diverter needs to be replaced - in your case, this is part of the spout. The diverter is responsible for allowing water to flow out of the tap, or when engaged, it closes the spout and forces the water up through the shower head.

You can try to disassemble it and find a replacement part, but often it is easier to just replace the entire spout.

Usually the spout is held onto a piece of copper pipe with a set screw in the bottom of the spout. Loosen the screw (usually with an allen key) and pull away from the pipe.  Replacement is the opposite - twist onto the pipe, align it properly and tighten the set screw.

You might not find an exact replacement but any spout with a diverter in it should work.

This should be a pretty cheap and easy replacement.  Good luck!