Why are sodium and potassium displaced from soap by calcium and magnesium ions in hard water?
I have no external justification to point to on this, but I've always thought one aspect contributing to this behavior is the fact that the divalent $\ce{Ca^2+}$ and $\ce{Mg^2+}$ bind two soap fatty acid molecules, whereas $\ce{Na+}$ and $\ce{K+}$ as monovalent ions can only strongly bind one molecule.

As a result, the alkaline earth cations have two fatty acid "tails" that can 'glom' around them, shielding them better from the water molecules and making the resulting complex more stable, as well as less soluble.

On the other hand, since the alkali cations only have one fatty acid "tail" they're strongly associated with, that one tail can't wrap around nearly as completely, and so there is significantly greater accessibility of the solvent to the cation.