Why is the solubility of calcium sulfate so low even though it is a salt?
Solubility of calcium sulphate is quite high, compared to many much less soluble salts, like barium sulphate, calcium phosphate or calcium fluoride.

Most of insoluble minerals are salts.

Insoluble/limited solubility salts have as crystals lower Gibbs energy than dissolved, leading to solution being thermodynamically  unfavourable.

That is related to mutual relation of energy of crystal lattice and of hydration of ions, and also to dissolution entropy change.