Why are chlorides, sulfates, and bicarbonates of Ca and Mg solely responsible for the hardness of water?
If I remember correctly, hardness is defined by the Mg and Ca content of water. Temporary hardness is the mineral content which can be removed by boiling and permanent is the total Mg and Ca content - this is what counts when the precipitate with the fatty acid is being formed.

Now typical potable water will only contain sulphide, bicarbonate, halide and perhaps nitrate anions. Other anions either don't occur naturally or they are harmful so they are not considered in the context of drinking water.